Great Valley Team Awarded for AI Research in Data Science Coding
At the 2025 International Conference on Mining Software Repositories, Penn State Great Valley’s software engineering faculty and students earned the Distinguished Paper Award for their study on large language models (LLMs) applied to data science coding challenges.
Assessing AI Tools for Data Science Code Generation
LLMs are AI systems capable of generating code to assist data scientists with tasks like data analysis and visualization. The research team, led by assistant professors Nathalia Nascimento and Everton Guimarães, investigated how effectively four prominent LLMs—Microsoft Copilot, ChatGPT, Claude, and Perplexity Labs—handle diverse coding problems over a controlled two-month study.
The study focused on assessing these models' ability to generate accurate solutions for analytical, algorithmic, and visualization tasks. The team also developed a novel public dataset to benchmark LLM performance on data science coding problems, providing a valuable resource for future evaluations.
Key Findings and Contributions
- All four LLMs surpassed a 50% baseline success rate, showing their coding capabilities exceed random chance.
- ChatGPT and Claude stood out by achieving over 60% success rates, although none reached 70%, indicating room for improvement.
- The performance varied depending on task type and difficulty, highlighting strengths and limitations of each model.
The study delivers a rigorous framework for evaluating LLMs in practical data science coding scenarios. This approach helps researchers and practitioners make informed decisions on which AI assistant to use based on specific coding tasks.
Collaboration and Next Steps
The research was conducted in partnership with two graduate students, Sai Sanjna Chintakunta (data analytics) and Santhosh Anitha Boominathan (software engineering), who contributed to experimental design and analysis. Following the conference, the team is expanding their work with additional models and deeper dataset analysis in a journal extension.
The paper, titled “How Effective are LLMs for Data Science Coding? A Controlled Experiment,” was selected from over 100 submissions for the technical track at MSR 2025, underscoring the quality and relevance of their contribution.
Implications for AI in Software Engineering
This research provides a practical benchmark for AI-assisted coding tools in data science, highlighting current capabilities and limitations. It informs both the development of future models and the strategic adoption of AI coding assistants in research and industry settings.
For professionals interested in AI tools and coding automation, exploring further resources on AI-driven development and data science coding can be valuable. Relevant courses and certifications are available at Complete AI Training.
Your membership also unlocks:
