Originally published 2 January 2020
What is Student Success System (S3)?
- Student Success System is an Early Intervention System that empowers institutions with predictive analytics to improve student success, retention, completion, and graduation rates.
- Student Success System can measure student performance starting from the first weeks of the semester.
- Student Success System provides educators with early indicators and predictions of student success and risk levels.
How does S3 work?
- S3 generates final grade predictions based on predictive models that are created by applying machine learning algorithms on historic course data (prior offerings of the same course).
- The predictive models are adaptable and customizable to the instructional approach of each course, as well as engagement and achievement expectations.
- The system provides weekly predictions of student success levels within their courses in the form of a success index.
S3 Visualizations
- S3 provides interactive visualizations to highlight patterns and indicators about the student and their position relative to the course expectations and to other students within the class.
- These visualizations allow instructors access to new and powerful information.
- S3 is designed to let the instructors visualize and compare key factors to prepare for interventions.
On the main page of the student success system, instructors can monitor the status of each student in terms of their predicted success as shown.
Figure: Example image of a student success system classlist.
Success Index
- For each student, a Success Index is displayed for the current week. The success index is expressed as a category, a corresponding score on a scale of 0-10, and trend sign as shown.
- There are three levels of success indicated by the color and shape of an associated symbol: At Risk (red triangle), Potential Risk (yellow diamond), and Successful (green circle). The levels are determined based on thresholds on the predicted grade.
- The defaults are: 0%-60% for At-Risk, 60%-80% for Potential Risk, and 80%-100% for successful.
- These instructor-specified thresholds can be configured for each course during the setup of the predictive model. The value of the success index is determined based on the predicted grade.
- The trend arrow indicates whether the predictions are changing upwards or downwards compared to the previous week.
Emailing Students
From the Instructor Dashboard, an instructor can initiate a group-intervention based on the success category via email. An email dialog will pop up with all student addresses in the Bcc field as shown
Figure: Composing a new message.
- Instructors can choose to select an individual student to view the prediction information needed for this student so that the instructor can design a personalized intervention.
- Student page brings together information about the student, through interactive visualizations.
- The goal is to highlight patterns and indicators about the student and their position relative to the course expectations and to other students within the class.
Student Profile
- On the top-left, basic profile information about the presented student. This includes the student picture, name, and campus Id. If the institution permits access to additional data elements from the Student Information System (SIS), it will be presented.
Course Timeline
- On the top–right side, a timeline chart shows the weekly success index, indicated by its value on the y-axis and color. It provides an at-a-glance view of the student trend-line.
Figure: Student Success System chart data.
Win-Loss Chart
- As shown, a win-loss chart shows how the success index is designed as a combination of predictors.
- The chart shows the student success based on each of these predictors and how they compare to the mid-range expectations.
- The vertical reference line in the middle of the chart corresponds to the middle point within in range of Potential Success (if the range is 60%-80%, the middle point is 70%, which corresponds to the value 7 for the success index).
- The success indicator for the student along each of these indicators is compared to this reference point, leading to either a “win” where the bar falls on the right side of the line or a “loss” where the bar is on the left side of the line.
- The “loss” indicators point to expectation gaps where improvement can be made.
Figure: Student Success Index.
Predictions domains are described as follows:
- Course Access: This domain consists of a set of related measures that describe student engagement in terms of logging into the learning environment and accessing the course page.
- Content Access: This domain consists of a set of related measures that describe student engagement with content material for the course
- Social learning: This domain consists of a set of related measures that describe student engagement in discussion forums within the course.
- Grades: This domain consists of a set of related measures that describe student performance on assessments.
- Preparedness: This domain consists of a set of data elements from the Student Information System, including admission scores, overall college performance, and demographics.
Risk Quadrant
- Risk Quadrant positions the student within the class based on the success index and the current grade as shown.
- Success index provides the overall predicted outcome that is primarily based on engagement (as per the above-mentioned domains).
- The current grade provides an overall measure of performance based on the calculated grade to-date. Each dot on the chart is a student. The current student is highlighted.
- Quadrants are as follows.
- The withdrawal/ dropout risk is where the student seems to be struggling in terms of both engagement and performance. In this case, the student may be at risk of dropping out.
- The academic performance risk and is where the student seems to be engaged but is struggling in terms of performance.
- The under-engagement risk is where the student seems to not be engaged, yet is achieving high grades. In this case, the student may be under-challenged.
- The on-track, not at-risk quadrant is where the student is engaged and achieving high grades.
Figure: An example of data in the Risk Quadrant.
Grades Domain
- By drilling into the Grades domain, a visualization of the student’s grades is displayed as shown.
- The purpose of this chart is to provide a compact visualization of the student’s performance across all assessments while taking into account how the student performs relative to classmates, and the impact of different assessments on the student’s overall performance.
- The chart is a variant of a pie chart with more information being displayed.
- Each slice corresponds to a grade item or a category of grade items (e.g. Assignments, Quizzes).
- The size of each slice corresponds to the weight of the grade item.
- The student grade is displayed as an arc with a diamond symbol within each slice on a scale of 0-100 that is shown on a top radius.
- A band within each slice shows the class range as the bottom and top quartile for the corresponding grade item as shown.
- The chart is also interactive to enable further exploration of the student’s performance. If a particular grade corresponds to a category with more items within it, an outer edge indicates the number of items.
- Clicking on this part of the chart allows the drilling into a display of the grade items within it using the same visual technique. You can also roll back up by clicking in the center to go back to the overall aggregated view.
Figure: Grade distribution data.
Social Learning domain
- By drilling into the Social Learning domain, a visualization of the student’s network is displayed as shown.
- The sociogram is a network diagram where each student is represented as a node (circle) and each connection between two students indicates that they exchanged communication through in-class discussion forums, where one student has replied to the other.
- The relative sizes of the nodes are proportional to the amount of communication that students have sent and received through discussions.
- The layout for the sociogram is based on social connections where students are clustered closer together if they are connected and are farther apart if they are not connected.
- Some students can be more connected than others and so they would take central positions, while others can be less connected and take positions near the boundaries of the network or the clusters within the network.
- Some students can be completely isolated and would appear as unconnected smaller nodes.
Figure: An example of a student's sociogram.
Model Building
- The administrative part of the system where the configuration of the predictive model can be adapted to each course by the S3 administrator as shown.
- It shows the selection of “Domains” (the predictive components of the model), the range for the success levels and the historical offerings of the course that can be used as representative of the current offering of the course for which the predictions are generated.
Figure: Creating a predictive model for a course.
Predictive Modeling Evaluation
- MSE = Mean Squared Error
- Difference between Predicted Grade and Actual Grade
- The Lower the Better
