Choosing appropriate courses for a semester is a challenging task for undergraduate students. To facilitate the course selection process, different course recommendation systems have been proposed implementing different machine learning algorithms and techniques. At the same time, recently, there has been a rapid development of Large Language Models (LLMs) (e.g., GPT4, Llama3, and Gemini), which have been adopted in numerous applications and have influenced all walks of life. In this paper, we explore their potential to assist stakeholders in higher education with course recommendation. We explore two different ways to directly (and offline, to ensure no sensitive data leakage) generate recommendations from pre-trained or fine-tuned LLM models. We also propose a novel ChatGPT-assisted course recommendation model (GPTaCR) which follows a different methodology. It utilizes the output of ChatGPT to form rules that capture relationships among courses. Based on these rules, we generate a set of courses for every student to take next, given their prior enrolment history. We use a real-world dataset to evaluate the performance of our proposed models compared to other relevant course recommendation approaches. The findings highlight that our proposed manages to best combine the rich knowledge base of ChatGPT with information about past students’ enrollment history. We hope that this work can be a source of motivation for researchers to look into how LLMs might improve recommendation performance and other educational data mining tasks.