This study explored how changes in stress, anxiety about COVID-19, and leisure participation affected individuals’ well-being during the COVID-19 pandemic. The study included two waves of data collection from 454 participants aged 30–49 years in South Korea; data collection occurred during the two peaks of the pandemic in 2020. A series of analyses was used to examine the associations in leisure patterns, demographic characteristics, stress, anxiety, and well-being during the pandemic. The results indicated that the participants’ leisure patterns shifted from participating in more home leisure to outdoor leisure between the two waves during the pandemic. In addition, participating in more outdoor leisure activities was found to contribute to better well-being. An improved understanding of the associations between the COVID-19 pandemic and various aspects of life is important for effective leisure development and to help people overcome life’s difficulties during and after the pandemic.