Relativistic sentiment analyzer

• US 9,336,268 B1
• Filed: 04/08/2015
• Issued: 05/10/2016
• Est. Priority Date: 04/08/2015
• Status: Active Grant

• Current Assignee: Pearson Education Incorporated (Pearson plc)
• Original Assignee: Pearson Education Incorporated (Pearson plc)
• Inventors: Moudy, Christopher, Paterson, Todd, Berns, Kevin
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Nilsson, Eric
• Application Number:
  US14/681,474
• Time in Patent Office: 398 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/244 Grouping and aggregation
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]

Abstract

Sentiment analyzer systems may include feedback analytics servers configured to receive and analyze feedback data from various client devices. Feedback data may be received and analyzed to determine feedback context and sentiment scores. In some embodiments, natural language processing neural networks may be used to determine sentiment scores for the feedback data. Feedback data also may be grouped into feedback aggregations based on context, and sentiment scores may be calculated for each feedback aggregation. Sentiment analyzer outputs and corresponding output devices may be determined based on the sentiment scores and feedback contexts.

Claims

1. A sentiment analyzer for an electronic learning system comprising:

one or more client devices of the electronic learning system, each client device comprising;

a processing unit comprising one or more processors;

an I/O subsystem configured to provide electronic learning content, and to receive user input data relating to the provided electronic learning content via one or more input devices connected to the client device; and

memory coupled with and readable by the processing unit and storing therein a set of instructions which, when executed by the processing unit, causes the client device to;

provide electronic learning content to one or more users via the I/O subsystem;

receive user feedback data relating to the provided electronic learning content via the I/O subsystem; and

transmit the user feedback data relating to the provided electronic learning content to a feedback analytics server of the electronic learning system; and

a feedback analytics server of the electronic learning system, comprising;

a processing unit comprising one or more processors; and

memory coupled with and readable by the processing unit and storing therein a set of instructions which, when executed by the processing unit, causes the feedback analytics server of the electronic learning system to;

receive a plurality of feedback data from the one or more client devices, the received feedback data corresponding to user feedback of one or more users relating to electronic learning content;

determine an associated user and a sentiment score for each of the received plurality of feedback data;

group the plurality of feedback data into one or more feedback aggregations associated with the one or more users;

calculate a sentiment score for each of the one or more feedback aggregations associated with the one or more users using a language processing engine to determine a sentiment score for text feedback data relating to the electronic learning content;

receive user records associated with each of the one or more users, the received user records relating to interactions of the one or more users with the electronic learning system occurring after the receipt of the feedback data;

store the user records and associated sentiment scores for each of the one or more users within a data store of the electronic learning system;

training a machine learning algorithm based on the stored user records and associated sentiment scores, for each of the one or more users within the data store of the electronic learning system;

receive additional feedback data from the one or more client devices, the additional feedback data including user feedback from a first user relating to electronic learning content;

calculate a sentiment score for the first user, based on the received additional feedback data;

using the stored user records and associated sentiment scores in the data store of the electronic learning system, determine a user record prediction for the first user using the trained machine learning algorithm, based on the calculated sentiment score for the first user;

determine a sentiment analyzer output for the electronic learning system and one or more output devices, based on the determined user record prediction for the first user; and

provide the determined sentiment analyzer system-output for the electronic learning system to the determined one or more output devices.

2. The sentiment analyzer system of claim 1, wherein determining the sentiment score for each of the received plurality of feedback data comprises:

providing each of the plurality of feedback data to a natural language processing (NLP) neural network; and

receiving a plurality of sentiment scores from the NLP neural network corresponding to the received plurality of feedback data.

3. The sentiment analyzer system of claim 1, wherein the received plurality of feedback data comprises multimodal user input data relating to the electronic learning content, and wherein determining the sentiment score for the each of the received plurality of feedback data comprises at least two of:

using a language processing engine to determine a sentiment score for text feedback data relating to the electronic learning content;

using a voice analyzer to determine a sentiment score for voice feedback data relating to the electronic learning content;

using a gesture analyzer to determine a sentiment score for movement feedback data relating to the electronic learning content; and

using an eye movement analyzer to determine a sentiment score for eye movement feedback data relating to the electronic learning content.

4. The sentiment analyzer system of claim 1, wherein grouping the plurality of feedback data comprises:

determining a first subset of the plurality of feedback data corresponding to feedback provided by the first user via the one or more client devices; and

grouping the first subset of the plurality of feedback data into a first feedback aggregation.

5. The sentiment analyzer system of claim 1, wherein grouping the plurality of feedback data comprises:

determining a first subset of the plurality of feedback data corresponding to feedback provided by a plurality of users relating to a first set of electronic learning content; and

grouping the first subset of the plurality of feedback data into a first feedback aggregation.

6. The sentiment analyzer system of claim 1, the memory of the feedback analytics server storing therein further instructions which, when executed by the processing unit, cause the feedback analytics server to:

group the plurality of feedback data into at least a first feedback aggregation and a second feedback aggregation, wherein the first and second feedback aggregations correspond to a same subset of the one or more users, and wherein the first feedback aggregation corresponds to a first feedback time period and the second feedback aggregation corresponds to a second feedback time period;

calculate a sentiment score for the second feedback aggregation; and

determine a change in sentiment score for the same feedback context between the first feedback time period and the second feedback time period, by comparing the sentiment score for the second feedback aggregation to the sentiment score for the second feedback aggregation, wherein the sentiment analyzer system output is determined based on the change in sentiment score.

7. The sentiment analyzer system of claim 1, wherein the received plurality of feedback is related to a presentation of live electronic learning content within the electronic learning system, and wherein determining and providing the sentiment analyzer system output to the one or more output devices comprises:

determining one or more computing devices associated with a presenter of the live electronic learning content;

determining the sentiment analyzer system output for the presenter of the live electronic learning content, based on the sentiment score for a first feedback aggregation; and

transmitting the sentiment analyzer system output to the computing devices associated with the presenter of the live electronic learning content during the presentation of the live electronic learning content.

8. The sentiment analyzer system of claim 1, further comprising a data store server comprising one or more data stores, wherein receiving the plurality of feedback data from the one or more client devices comprises:

receiving the plurality of feedback data via an event streaming service executing on the data store server.

9. The sentiment analyzer system of claim 1, wherein calculating the sentiment score for each of the one or more feedback aggregations associated with the one or more users comprises:

calculating a Z-score for the a feedback aggregation associated with a first user, based on the sentiment scores for each of the feedback data within the first feedback aggregation; and

calculating a stanine score for the first feedback aggregation based on the Z-score for the first feedback aggregation.

10. A method of analyzing feedback data of an electronic learning system comprising:

receiving a plurality of feedback data from one or more client devices, the received feedback data corresponding to user feedback of one or more users relating to electronic learning content of the electronic learning system;

determining an associated user and a sentiment score for each of the received plurality of feedback data;

grouping the plurality of feedback data into one or more feedback aggregations associated with the one or more users;

calculating a sentiment score for each of the one or more first feedback aggregations associated with the one or more users;

receiving user records associated with each of the one or more users, the received user records relating to interactions of the one or more users with the electronic learning system occurring after receipt of the feedback data;

storing the user records and associated sentiment scores for each of the one or more user within a data store of the electronic learning system;

training a machine learning algorithm based on the stored user records and associated sentiment scores, for each of the one or more users within the data store of the electronic learning system;

receiving additional feedback data from the one or more client devices, the additional feedback data including user feedback from a first user relating to electronic learning content;

calculating a sentiment score for the first user, based on the received additional feedback data;

using the stored user records and associated sentiment scores in the data store of the electronic learning system, determining a user record prediction for the first user using the trained machine learning algorithm, based on the calculated sentiment score for the first user;

determining an output and one or more output devices, based on the determined user record prediction for the first user; and

transmitting the determined output to the determined one or more output devices.

11. The method of claim 10, wherein determining the sentiment score for each of the received plurality of feedback data comprises:

providing each of the plurality of feedback data to a natural language processing (NLP) neural network; and

receiving a plurality of sentiment scores from the NLP neural network corresponding to the received plurality of feedback data.

12. The method of claim 10, wherein the received plurality of feedback data comprises multimodal user input data relating to the electronic learning content, and wherein determining the sentiment score for the each of the received plurality of feedback data comprises at least two of:

using a language processing engine to determine a sentiment score for text feedback data relating to the electronic learning content;

using a voice analyzer to determine a sentiment score for voice feedback data relating to the electronic learning content;

using a gesture analyzer to determine a sentiment score for movement feedback data relating to the electronic learning content; and

using an eye movement analyzer to determine a sentiment score for eye movement feedback data relating to the electronic learning content.

13. The method of claim 10, wherein the received plurality of feedback is related to a presentation of live electronic learning content within the electronic learning system, and wherein determining and providing the output to the one or more output devices comprises:

determining one or more computing devices associated with a presenter of the live electronic learning content;

determining the output for the presenter of the live electronic learning content, based on the sentiment score for the a feedback aggregation; and

transmitting the output to the computing devices associated with the presenter of the live electronic learning content during the presentation of live electronic learning content.

14. The method of claim 10, wherein receiving the plurality of feedback data from the one or more client devices comprises:

receiving the plurality of feedback data via an event streaming service executing on a data store server separate from the one or more client devices.

15. The method of claim 10, wherein calculating the sentiment score for each of the one or more feedback aggregations associated with the one or more users comprises:

calculating a Z-score for a first feedback aggregation associated with a first user, based on the sentiment scores for each of the feedback data within the first feedback aggregation; and

calculating a stanine score for the first feedback aggregation based on the Z-score for the first feedback aggregation.

16. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, including instructions configured to cause one or more data processors to perform actions including:

receiving a plurality of feedback data from one or more client devices, the received feedback data corresponding to user feedback of one or more users relating to electronic learning content presented via an electronic learning system;

determining an associated user a feedback context and a sentiment score for each of the received plurality of feedback data;

grouping the plurality of feedback data into one or more feedback aggregations associated with the one or more users;

calculating a sentiment score for each of the one or more feedback aggregation associated with the one or more users;

receiving user records associated with each of the one or more users, the received user records relating to interactions of the one or more users with the electronic learning system occurring after the receipt of the feedback data;

storing the user records and associated sentiment scores for each of the one or more users within a data store of the electronic learning system;

training a machine learning algorithm based on the stored user records and associated sentiment scores, for each of the one or more users within the data store of the electronic learning system;

receiving additional feedback data from the one or more client devices, the additional feedback data including user feedback from a first user relating to electronic learning content;

calculating a sentiment score for the first user, based on the received additional feedback data;

using the stored user records and associated sentiment scores in the data store of the electronic learning system, determining a user record prediction for the first user using the trained machine learning algorithm, based on the calculated sentiment score for the first user;

determining an output and one or more output devices, based on the determined user record prediction for the first user; and

transmitting the determined output to the determined one or more output devices.

17. The computer-program product as recited in claim 16, wherein

determining the sentiment score for each of the received plurality of feedback data comprises:

providing each of the plurality of feedback data to a natural language processing (NLP) neural network; and

receiving a plurality of sentiment scores from the NLP neural network corresponding to the received plurality of feedback data.

18. The computer-program product as recited in claim 16, wherein the received plurality of feedback data comprises multimodal user input data relating to the electronic learning content, and wherein determining the sentiment score for the each of the received plurality of feedback data comprises at least two of:

using a language processing engine to determine a sentiment score for text feedback data relating to the electronic learning content;

using a voice analyzer to determine a sentiment score for voice feedback data relating to the electronic learning content;

using a gesture analyzer to determine a sentiment score for movement feedback data relating to the electronic learning content; and

using an eye movement analyzer to determine a sentiment score for eye movement feedback data relating to the electronic learning content.

19. The computer-program product as recited in claim 16, wherein receiving the plurality of feedback data from the one or more client devices comprises:

receiving the plurality of feedback data via an event streaming service executing on a data store server separate from the one or more client devices.

20. The computer-program product as recited in claim 16,

wherein calculating the sentiment score for each of the one or more feedback aggregations associated with the one or more users comprises:

calculating a Z-score for a first feedback aggregation associated with a first user, based on the sentiment scores for each of the feedback data within the first feedback aggregation; and

calculating a stanine score for the first feedback aggregation based on the Z-score for the first feedback aggregation.