Objective
In 2018, 101 million students were enrolled in massive open online courses or MOOCs such as Coursera, Udacity, and edX1. Practice problems are an essential part of the learning process for online courses. They allow students to gain hands-on, practical knowledge of the theory learned in lessons, as well as provide exposure to multiple case studies of real-world problems. Traditionally, practice problems are generated by those creating the learning material. However this method is usually tedious, time consuming and difficult to scale. Students can benefit from having more diverse problems written by many people or even fellow students. Imagine having an online archive of hundreds of thousands of practice problems that are all user submitted and vary across all domains and levels of difficulty. A student could potentially have an unlimited number of unique practice problems at every step of their learning journey. The challenge then becomes recommending an appopriate problem to a student based on the material currently being covered in their curriculum while also fitting the difficulty level that would challenge the student but not be so difficult that they don’t learn or become discouraged. In this project, I use some meta data available for each practice problem along with a user’s history of practice problems solved and attempted, to recommend appropriate practice problems.
Project Notebooks
Summary
The goal of this project was to build a recommender system for an online agent to recommend problems to students. The critical task of this recommender was to predict how many attempts a user would take to solve a problem the user has never seen before. I compared several models including a random forest model and several types of collaborative filtering models. The out-of-box random forest model performed the best with an f1-score on the cross-validation data of 0.5191. The best collaborative filtering model produced an f1-score of 0.4922. These f1-scores are relatively low for a good recommender system, however this particular dataset was quite sparse with only 0.76% of the attempts data available! This made predicting the missing attempts very challenging.
There are several ways to further improve the collaborative filtering model that I created here. The first is to add bias terms to the cost function. While regularization helps reduce the model’s overfitting by penalizing higher order terms, including bias penalties helps take into account the fact that users will have a different average number of attempts across all problems. For example, a beginner may require on average 8 attempts to complete most problems while someone with a strong background may require only 3 attempts on average.
The actual recommendations were made using a very simple set of rules. I showed several examples of recommendations and they all looked like very logical recommendations. In future work, I propose to run the model in production and perform A/B testing with a control group that receives random problem recommendations and a test group that receives problems recommended by the current model. The click-through-rate (CTR) can be collected for both groups and a simple two-tailed t-test can be performed to determine if the two groups have a statistically significant different CTR.
Leave a Comment