STUDENT DATA-TO-INSIGHT-TO-ACTION-TO-LEARNING ANALYTICS SYSTEM AND METHOD

• US 20170256172A1
• Filed: 03/06/2017
• Published: 09/07/2017
• Est. Priority Date: 03/04/2016
• Status: Abandoned Application

• Current Assignee: Civitas Learning, Inc.
• Original Assignee: Civitas Learning, Inc.
• Inventors: Kil, David H., Derr, Kyle, Whitfield, Mark, Eads, Grace, Daly, John M., Gallaway, Clayton, Harmse, Jorgen, Wimalasuriya, Daya Chinthana
• Application Number:
  US15/451,147
• Publication Number:
  US 20170256172A1
• Time in Patent Office: Days
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 50/205 Education administration or guidance
• CPC Class Codes: G09B 19/00 Teaching not covered by other main groups of this subclass teaching or practice apparatus for gun-aiming or gun-laying F41G3/26
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 5/02 with visual presentation of the material to be studied, e.g. using film strip

Abstract

Student data-to-insight-to-action-to-learning analytics system and method use an evidence-based action knowledge database to compute student success predictions, student engagement predictions, and student impact predictions to interventions. The evidence-based action knowledge database is updated by executing a multi-tier impact analysis on impact results of applied interventions. The multi-tier impact analysis includes using changes in key performance indicators (KPIs) for pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

Claims

1. A student data-to-insight-to-action-to-learning analytics method comprising:

computing student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;

applying appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and

executing a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

2. The method of claim 1, wherein executing a multi-tier impact analysis includes computing utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window.

3. The method of claim 2, wherein executing a multi-tier impact analysis further includes determining whether the interventions are message nudges, and for the message nudges, performing natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages.

4. The method of claim 1, wherein applying the appropriate interventions to the pilot students further comprises:

monitoring incoming streams of event data to detect when any of the engagement rules are triggered;

if more than one engagement rule is triggered, prioritizing the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and

applying an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

5. The method of claim 3, wherein executing a multi-tier impact analysis includes:

aligning the applied interventions with respect to time;

creating a pool of control students that are similar to each pilot student exposed to one of the interventions;

creating groups of pilot and control students that have similar metrics; and

performing difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

6. The method of claim 5, wherein executing a multi-tier impact analysis further includes correlating the success metrics with the utility scores.

7. The method of claim 6, wherein executing a multi-tier impact analysis includes:

segmenting the students using data footprint;

selecting features and academic terms for matching;

building predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;

performing a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and

executing a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.

8. A computer-readable storage medium containing program instructions for student data-to-insight-to-action-to-learning analytics method, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

computing student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;

applying appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and

executing a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

9. The computer-readable storage medium of claim 8, wherein executing a multi-tier impact analysis includes computing utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window.

10. The computer-readable storage medium of claim 9, wherein executing a multi-tier impact analysis further includes determining whether the interventions are message nudges, and for the message nudges, performing natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages.

11. The computer-readable storage medium of claim 8, wherein applying the appropriate interventions to the pilot students further comprises:

monitoring incoming streams of event data to detect when any of the engagement rules are triggered;

if more than one engagement rule is triggered, prioritizing the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and

applying an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

12. The computer-readable storage medium of claim 11, wherein executing a multi-tier impact analysis includes:

aligning the applied interventions with respect to time;

creating a pool of control students that are similar to each pilot student exposed to one of the interventions;

creating groups of pilot and control students that have similar metrics; and

performing difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

13. The computer-readable storage medium of claim 12, wherein executing a multi-tier impact analysis further includes correlating the success metrics with the utility scores.

14. The computer-readable storage medium of claim 13, wherein executing a multi-tier impact analysis includes:

segmenting the students using data footprint;

selecting features and academic terms for matching;

building predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;

performing a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and

executing a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.

15. A student data-to-insight-to-action-to-learning analytics system comprising:

memory;

a processor configured to;

compute student success predictions, student engagement predictions, and student impact predictions to interventions using at least linked-event features from multiple student event data sources and an evidence-based action knowledge database, the linked-event features including student characteristic factors that are relevant to student success;

apply appropriate interventions to pilot students when engagement rules are triggered, the engagement rules being based on at least the linked-event features and multi-modal student success prediction scores; and

execute a multi-tier impact analysis on impact results of the applied interventions to update the evidence-based action knowledge database, the multi-tier impact analysis including using changes in key performance indicators (KPIs) for the pilot students after each applied intervention and dynamic matching of the pilot students exposed to the appropriate interventions to other students who were not exposed to the appropriate interventions.

16. The system of claim 15, wherein the processor is configured to compute utility scores for triggered engagement rule-intervention pairs by looking at changes in KPIs within a defined time window to execute the multi-tier impact analysis.

17. The system of claim 16, wherein the processor is configured to determine whether the interventions are message nudges, and for the message nudges, perform natural language processing on the contents of the message nudges to learn characteristics of effective and ineffective messages to execute the multi-tier impact analysis.

18. The system of claim 15, wherein the processor is configured to:

monitor incoming streams of event data to detect when any of the engagement rules are triggered;

if more than one engagement rule is triggered, prioritize the engagement rules that are triggered based corresponding utility scores and intersection with recently triggered engagement rules to derive a highest ranked engagement rule; and

apply an intervention that correspond to the highest ranked engagement rule to at least one pilot student.

19. The system of claim 18, wherein the processor is configured to:

align the applied interventions with respect to time;

create a pool of control students that are similar to each pilot student exposed to one of the interventions;

create groups of pilot and control students that have similar metrics; and

perform difference-of-difference analysis on each applied intervention for the groups of pilot and control students to produce success metrics for cells of CPT.

20. The system of claim 19, wherein the processor is configured to:

segment the students using data footprint;

selecting features and academic terms for matching;

build predictive and propensity-score models for each student-success metric and intervention program to produce prediction scores and propensity scores;

perform a matching process on the pilot and control students to ensure that the pilot and control students are indistinguishable in a statistical sense; and

execute a statistical hypothesis testing to determine if observed difference in student success rates between the pilot and control students is statistically significant.