Finding and Acquiring Data: One of the biggest hurdles students face is finding and acquiring appropriate data to use for their project. Data science projects require real-world datasets, but students may not have connections to access private industry data. They need to search extensively online to find public datasets that meet their project goals and scope. Some datasets require applications for access as well. The process of identifying, requesting, and receiving suitable data can be very time consuming.
Data Cleaning and Preprocessing: Once data is acquired, students typically spend a significant amount of time cleaning and preprocessing it before it can be analyzed. Real-world data is often incomplete, inconsistent, lacking in context or just messy. Tasks like dealing with missing values, correcting or removing erroneous entries, transforming features, and handling outliers are tedious but necessary. The amount of work required to get data into a usable format for modeling is frequently underestimated. Issues with data quality can also undermine project objectives.
Selecting and Justifying Modeling Techniques: Data science encompasses a wide range of analytical and predictive modeling techniques. Students need to research which methods are best suited to address their project goals and hypotheses. They also need to demonstrate an understanding of the underlying algorithmic and statistical concepts. Justifying the selection of techniques based on the problem, data characteristics, assumptions, and performance metrics is an important part of any data science project. With many options to choose from, this can be confusing for students to navigate on their own.
Implementing Models and Getting Them to Work: Even after selecting techniques, students often struggle with the practical implementation and execution of models. Desired libraries may have steep learning curves. Configuring parameters, preventing overfitting, and debugging code all take significant trial-and-error. Capstone projects have expectations for demonstrating workable end-to-end solutions, but getting complex models fully functioning can be a major obstacle, exacerbated by tight timelines. Ensuring reproducibility of results is another challenge.
Effectively Communicating Findings and Results: A capstone project aims to showcase data science skills, but also requires strong communication skills to convey technical concepts to both technical and non-technical audiences. Students need to prepare documentation, presentation slides, written reports, visualizations, and demonstrate their work. Synthesizing technical details, results, limitations, and next steps into a clear narrative can be difficult, as can answering questions from reviewers. Communication skills take experience to develop and represent another hurdle for graduating students.
Time Management Under Constraints: Most capstone projects have strict time constraints spanning a single semester or quarter. Between class work, jobs, and other commitments, maintaining consistent progress on a project within scheduled deadlines can be tricky. Data science work also involves many iterative trial-and-error cycles that are difficult to estimate up front. Setting interim milestones and sticking to a schedule helps, but unexpected hiccups and dead-ends and over-ambitious scoping are perpetual risks that students must navigate under pressure. Learning to focus, prioritize, and manage competing priorities is challenging.
These represent some of the most significant hurdles students typically face when tackling an independent data science capstone project within the structure of an academic program. While intended to showcase mastery of skills, the open-ended nature of capstones and need to take an idea from start to finish also exposes gaps and areas where students require further practice, experience, mentorship, or real-world support. Successfully overcoming the common challenges through hard work and iterative improvements is thus an important part of overall learning and professional preparation. Clear guidance and feedback from mentors can help ease obstacles that students may face along the way as well.
Andrew Stroud