Aim: Social distancing policies to combat the pandemic of COVID-19 directly impacted the physical activity (PA) index of the population. This study aimed to analyze the factors associated with insufficient leisure-time PA and the changing behavior of residents of the city of Sao Paulo, the epicenter of the pandemic in Brazil. Methods: The questionnaire used in this research was made available online and included questions on demographics and physical activity. A total of 1.096 adult residents of Sao Paulo completed the questionnaire between July 8th and August 8th, 2020. Results: For individuals who were already physically active before the restrictions, a total of 55.8% of the sample kept insufficiently active during the quarantine. Factors such as education, working conditions, and direct relationship with Covid-19 care were associated with decreased leisure-time PA practice, impacting 57.6% of the sample, especially females. People with less formal education, those who possibly changed their routines to working from home, and those who were directly connected to Covid cases were not able to maintain a daily PA routine. Conclusion: The data presented here warn about the impacts of the pandemic on engagement in PA, suggesting the need for incentives from intersectoral public policies on sports, leisure, and health to perform these activities.