Systems and methods for providing information incorporating reinforcement-based learning and feedback

• US 9,064,017 B2
• Filed: 06/01/2012
• Issued: 06/23/2015
• Est. Priority Date: 06/01/2011
• Status: Active Grant

• Current Assignee: D2L Corporation
• Original Assignee: D2L Corporation
• Inventors: Zhou, Jian, Bilic, Jugoslav, Bilic, Ivana, Callison, Justin, Aggarwal, Manuj, Michaud, Stephen, Ogg, Ryan, Ramirez, Juan Camilo, Sarkaria, Sarbjit
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US13/486,923
• Publication Number:
  US 20120310961A1
• Time in Patent Office: 1,117 Days
• Field of Search: 707/687, 707/693, 707/797, 707/704, 707/713, 707/736, 707/758, 707/821
• US Class Current: 1/1
• CPC Class Codes: G06F 16/2282 Tablespace storage structures; Management thereof
• CPC Class Codes: G06F 16/2379 Updates performed during online database operations; commit processing
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F

Abstract

Systems and methods are provided for feedback-driven provision of information in relation to a plurality of information resources. Some such methods involve: accessing information resources from repositories to establish a mapping between information resources; maintaining a state table comprising a plurality of state-table records, each state-table record attributing a value to a state-action pair, which comprises an indication of actions in relation to the information resources; monitoring actions of a user in relation to the information resources; receiving a feedback metric related to interaction of the user with the information resources; using the feedback metric to update the values of the state-table records; and providing output information based at least in part on the updated values of the state-table records.

Claims

1. A method, implemented by a computer, for feedback-driven provision of information in relation to a plurality of information resources, the method comprising:

accessing the information resources from one or more information resource repositories to establish a mapping between each information resource and each of the other information resources;

maintaining a state table comprising a plurality of state-table records, each state-table record attributing a value to a state-action pair, the state-action pair comprising an indication of a series of one or more actions in relation to the information resources;

monitoring actions of a first user in relation to the information resources;

receiving a feedback metric related to interaction of the first user with the information resources;

using the feedback metric as a basis for updating the values of a subset of the state-table records, the subset of the state-table records comprising state-action pairs which correspond to the monitored actions of the first user in relation to the information resources;

providing output information related to the information resources based at least in part on one or more of the updated values of the state-table records.

2. A method according to claim 1 wherein the output information comprises one or more recommended actions for a particular user in relation to the information resources, each recommended action relating to a corresponding recommended one of the information resources.

3. A method according to claim 2 wherein the particular user is different than the first user.

4. A method according to claim 3 wherein receiving the feedback metric comprises receiving a grade resulting from the first user taking an assessment, the assessment comprising one or more questions related to information contained in the information resources.

5. A method according claim 3 wherein receiving the feedback metric comprises receiving the feedback metric from an instructor based on the instructor'"'"'s assessment of a level of understanding of the first user of information contained in the information resources.

6. A method according to claim 3 wherein monitoring the actions of the first user comprises determining that the first user has interacted with a particular set of information resources and wherein receiving the feedback metric comprises receiving an assigned feedback metric after determining that the first user has interacted with the particular set of information resources.

7. A method according to claim 3 wherein monitoring the actions of the first user comprises determining that the first user has interacted with one or more particular information resources a threshold number of times and wherein receiving the feedback metric comprises receiving an assigned feedback metric after determining that the first user has interacted with the one or more particular information resources the threshold number of times.

8. A method according to claim 3 wherein receiving the feedback metric comprises receiving user-evaluation feedback from the first user, the user-evaluation feedback relating to the first user'"'"'s perception of the first user'"'"'s interaction with the information resources.

9. A method according to claim 3 wherein accessing the information resources from the one or more repositories comprises electronically accessing the information resources from the one or more information repositories.

10. A method according to claim 9 wherein the one or more information resource repositories comprise a plurality of information resource repositories, at least two of which are maintained by different entities.

11. A method according to claim 9 wherein the one or more information resource repositories comprise a plurality of information resource repositories, at least two of which are different types of information resource repositories.

12. A method according to claim 9 wherein one of the one or more information resource repositories comprises the internet generally.

13. A method according to claim 9 wherein one of the one or more information resources comprises a portion of the internet, the portion of the internet maintained by one or more entities.

14. A method according to claim 9 wherein establishing the mapping between each information resource and each of the other information resources comprises establishing mappings between individual information resources based on semantic analysis of the individual information resources.

15. A method according to claim 14 wherein establishing the mapping between individual information resources based on semantic analysis comprise determining a metric that provides an indication of similarity between each information resource and each of the other information resources.

16. A method according to claim 14 wherein establishing the mapping between each information resource and each of the other information resources comprises crawling at least one of the one or more information repositories to access the information resources.

17. A method according to claim 14 wherein establishing the mapping between each information resource and each of the other information resources comprises receiving information pushed from at least one of the one or more information repositories.

18. A method according to claim 3 wherein using the feedback metric as a basis for updating the values of the subset of the state-table records comprises using reinforcement learning to determine the updated values.

19. A method according to claim 3 wherein the subset of state-table records whose values are updated comprise state-table records comprising state-action pairs corresponding to the monitored actions of the first user since a previous time that a feedback metric was received in relation to the interaction of the first user with the information resources.

20. A method according to claim 2 comprising:

monitoring actions of each of a plurality of first users in relation to the information resources;

receiving a plurality of feedback metrics, each feedback metric related to the interaction of a corresponding one of the plurality of first users with the information resources; and

using the received plurality of feedback metrics as a basis for updating the values of the state-table records, wherein each feedback metric is used as a basis for updating the values of a corresponding subset of the state-table records which comprises state-action pairs corresponding to the monitored actions of the corresponding one of the first users in relation to the information resources.

21. A method according to claim 20 wherein the particular user is not among the plurality of first users.

22. A method according to claim 20 wherein the particular user is one of the plurality of first users.

23. A method according to claim 20 wherein using each feedback metric as a basis for updating the values of a corresponding subset of the state-table records comprises using reinforcement learning to determine the updated values.

24. A method according to claim 3 wherein monitoring the actions of the first user in relation to the information resources comprises providing a user interface through which the first user accesses the information resources and logging the actions of the first user through the user interface.

25. A method according to claim 24 wherein logging the actions of the first user through the user interface comprises logging an action for each interaction of the first user with a corresponding one of the information resources.

26. A method according to claim 25 wherein logging the actions of the first user through the user interface comprises creating a user action log comprising one or more user log records, each user log record comprising a user log state-action pair which comprises:

an action of the first user in relation to a corresponding one of the information resources; and

a historical state of the first user, the historical state of the first user comprising a set of historical actions of the first user, each historical action relating to a corresponding one of the information resources.

27. A method according to claim 3 wherein monitoring the actions of the first user in relation to the information resources comprises:

providing a user interface through which the first user accesses the information resources; and

extracting information from one or more learning management system logs created by the learning management system.

28. A method according to claim 27 wherein monitoring the actions of the first user in relation to the information resources comprises logging an action for each interaction of the first user with a corresponding one of the information resources.

29. A method according to claim 3 wherein monitoring the actions of the first user in relation to the information resources comprises permitting the first user to interact with the information resources through an independent application program present on a user-computing device;

and providing a monitoring agent on the user computing device for passively monitoring the actions of the first user in relation to the independent application program.

30. A method according to claim 29 wherein the independent application program comprises a user interface to a learning management system.

31. A method according to claim 29 wherein the independent application program comprises an internet browser.

32. A method according to claim 29 wherein monitoring the actions of the first user in relation to the information resources comprises logging an action for each interaction of the first user with a corresponding one of the information resources.

33. A method according to claim 3 comprising monitoring actions of the particular user in relation to the information resources to establish a historical state of the particular user and wherein providing the output information comprising one or more recommended actions for the particular user in relation to the information resources is based at least in part on the historical state of the particular user.

34. A method according to claim 3 comprising:

subjecting the particular user to a diagnostic assessment comprising one or more questions related to information contained in the information resources;

ascertaining one or more questions from the diagnostic assessment which were answered incorrectly by the particular user; and

wherein providing the output information comprising one or more recommended actions for the particular user in relation to the information resources is based at least in part on the one or more incorrectly answered questions.

35. A method according to claim 34 wherein providing the output information comprising one or more recommended actions for the particular user in relation to the information resources comprises:

performing semantic analysis on the one or more incorrectly answered questions to determine a mapping between each of the one or more incorrectly answered questions and each of the information resources; and

determining the one or more recommended actions based at least in part on the mapping.

36. A method according to claim 3 comprising:

receiving one or more taxonomy items and, for each taxonomy item, establishing a mapping between the taxonomy item and each of the information resources.

37. A method according to claim 36 wherein, for each taxonomy item, the mapping between the taxonomy item and each of the information resources is based on semantic analysis of the taxonomy item and the information resources which provides an indication of similarity between the taxonomy item and each of the information resources.

38. A method according to claim 37 comprising:

prior to providing the output information comprising one or more actions for the particular user, receiving a learning objective of the particular user, the learning objective indicative of a corresponding taxonomy item; and

using the mapping between the indicated taxonomy item and each of the information resources as a basis for narrowing the plurality of information resources to a subset of information resources considered to be within a threshold level of similarity to the indicated taxonomy item; and

wherein providing the output information comprising one or more actions for the particular user is limited to actions relating to the subset of information resources.

39. A method according to claim 38 comprising using the mapping between the indicated taxonomy item and each of the information resources as a basis for narrowing the state table to a subset of the plurality of state-table records considered to be within a threshold level of similarity to the indicated taxonomy item and wherein providing the output information comprising one or more actions for the particular user is based at least in part on one or more updated values of the state-table records in the subset of the plurality of state-table records.

40. A method according to claim 3 wherein providing the output information comprising one or more recommended actions for the particular user comprises providing an action path for the particular user, the action path comprising a structured plurality of recommended actions for the particular used in relation to the information resources.

41. A method according to claim 40 wherein the action path comprises an ordered plurality of recommended actions for the particular used in relation to the information resources.

42. A method according to claim 41 wherein providing the action path for the particular user comprises:

determining a target state for the particular user, the target state for the particular used comprising a plurality of target actions, each target action relating to a corresponding target one of the information resources;

repeating a process of determining a next target action for inclusion in the action path based at least in part on one or more of the updated values of the state-table records, until the action path includes all of the actions in the target state.

43. A method according to claim 42 wherein repeating the process of determining the next target action for inclusion in the action path comprises, for at least one iteration:

filtering the state-table records to obtain a subset of the state-table records whose state-actions pairs include states comprising historical actions which match an expected state of particular user assuming that the particular user is following the action plan; and

selecting the next target action for inclusion in the action path to be an action corresponding to one of the subset of the state-table records having the highest value.

44. A method according to claim 42 wherein repeating the process of determining the next target action for inclusion in the action path comprises, for at least one iteration, selecting the next target action for inclusion in the action path to be one of the target actions not already in the action path having a highest weighted average value, the weighted average value of each prospective action calculated according to:

45. A method according to claim 3 wherein providing the output information comprising one or more recommended actions for the particular user comprises providing a plurality of recommended actions and wherein providing the plurality of recommended actions comprises:

filtering the state-table records to obtain a subset of the state-table records whose state-actions pairs include states comprising historical actions which match a current state of particular user, the current state of the particular user comprising historical actions of the particular user in relation to the information resources; and

selecting at least one of the plurality of recommended actions to be an action corresponding to one of the subset of the state-table records having the highest value.

46. A method according to claim 45 wherein providing the plurality of recommended actions comprises selecting at least one of the plurality of recommended actions to be an action having a highest weighted average value, the weighted average value of each prospective action calculated according to:

47. A method according to claim 3 wherein each state-table record attributes a plurality of values to its corresponding state-action pair, each of the plurality of values maintained in a corresponding value field and wherein at least one of the value fields comprises a personalization value field characterized by an evaluation characteristic.

48. A method according to claim 47 wherein the evaluation characteristic comprises a user-profile characteristic and wherein using the feedback metric as a basis for updating the values of the subset of the state-table records comprises updating the values in the personalization value field only in circumstances where the first user exhibits the user-profile characteristic.

49. A method according to claim 48 wherein providing the output information comprising one or more recommended actions for the particular user is based at least in part on the updated values in the personalization value field in circumstances where the particular user exhibits the user-profile characteristic.

50. A method according to claim 48 wherein providing the output information comprising one or more recommended actions for the particular user is based on the updated values in the personalization value field to the exclusion of the other updated values.

51. A method according to claim 47 wherein the evaluation characteristic comprises an information resource characteristic and wherein using the feedback metric as a basis for updating the values of the subset of the state-table records comprises updating the values in the personalization value field only for monitored actions of the first user that relate to interaction with information resources that exhibit the information resource characteristic.

52. A method according to claim 51 wherein providing the output information comprising one or more recommended actions for the particular user is based at least in part on the updated values in the personalization value field in circumstances where the one or more recommended actions relate to interaction with information resources that exhibit the information resource characteristic.

53. A method according to claim 47 wherein the evaluation characteristic comprises a configurable characteristic and wherein using the feedback metric as a basis for updating the values of the subset of the state-table records comprises updating the values in the personalization value field only for circumstances that exhibit the configurable characteristic.

54. A method according to claim 53 wherein providing the output information comprising one or more recommended actions for the particular user is based at least in part on the updated values in the personalization value field in circumstances that exhibit the configurable characteristic.

55. A system for providing feedback-driven information in relation to a plurality of information resources, the system comprising:

an information resource interface for accessing the information resources from one or more information resource repositories to establish a mapping between each information resource and each of the other information resources;

a ranking engine for maintaining a state table comprising a plurality of state-table records, each state-table record attributing a value to a state-action pair, the state-action pair comprising an indication of a series of one or more actions in relation to the information resources;

an action interface for monitoring actions of a first user in relation to the information resources;

a feedback interface for receiving a feedback metric related to interaction of the first user with the information resources;

wherein the ranking engine is configured to use the feedback metric as a basis for updating the values of a subset of the state-table records, the subset of the state-table records comprising state-action pairs which correspond to the monitored actions of the first user in relation to the information resources; and

a recommendation engine for providing output information related to the information resources based at least in part on one or more of the updated values of the state-table records.