Organizing survey text responses

• US 10,339,160 B2
• Filed: 10/29/2015
• Issued: 07/02/2019
• Est. Priority Date: 10/29/2015
• Status: Active Grant

• Current Assignee: Qualtrics LLC (SAP SE)
• Original Assignee: Qualtrics LLC (SAP SE)
• Inventors: Starr, Jeffrey Wade, Norton, Robert David
• Primary Examiner(s): Nguyen, Merilyn P
• Application Number:
  US14/927,256
• Publication Number:
  US 20170124174A1
• Time in Patent Office: 1,342 Days
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]

Abstract

Embodiments of the present disclosure relate generally to organizing electronic text documents. In particular, one or more embodiments comprise a content management system that improves the organization of electronic text documents by intelligently and accurately categorizing electronic text documents by topic. The content management system organizes electronic text documents based on one or more topics, without the need for a human reviewer to manually classify each electronic text document, and without the need for training a classification algorithm based on a set of manually classified electronic text documents. Further, the content management system identifies and suggests topics for electronic text documents that relate to new or emerging topics.

Claims

1. A method for organizing content items, the method comprising:

identifying one or more predetermined topics that potentially relate to a plurality of content items;

automatically assigning, by at least one processor and without user input, a first subset of content items from the plurality of the content items to the one or more predetermined topics by analyzing the plurality of content items to determine the first subset of content items of the plurality of content items that correlate to the one or more predetermined topics;

identifying a second subset of content items from the plurality of content items that comprise content items unassigned to the one or more predetermined topics;

providing a first unassigned content item from the second subset of content items comprising unassigned content items to a client device associated with a user;

receiving, from the client device associated with the user, an additional topic corresponding to the first unassigned content item from the second subset of content items comprising unassigned content items, wherein the additional topic is a different topic from the one or more predetermined topics;

automatically assigning, by the at least one processor and without user input, a second content item from the plurality of content items to the additional topic by re-analyzing the second subset of content items to determine that the second content item correlates to the additional topic; and

providing, the client device associated with the user, topic assignment results organized by the one or more predetermined topics and the additional topic.

2. The method of claim 1, wherein each content item of the plurality of content items comprises a text response.

3. The method of claim 2, wherein each text response comprises a survey response to one or more survey questions of an electronic survey.

4. The method of claim 1, wherein identifying one or more predetermined topics that potentially relate to the plurality of content items comprises:

providing, to the client device associated with the user, a topic prompt for the one or more predetermined topics; and

receiving, from the client device and in response to the topic prompt, the one or more predetermined topics that potentially relate to the plurality of content items.

5. The method of claim 4, further comprising providing, to the client device, one or more content items from the second subset of content items.

6. The method of claim 5, further comprising providing, to the client device, an additional topic prompt for the additional topic based on analyzing the second subset of content items to identify the additional topic prompt.

7. The method of claim 1, wherein identifying the one or more predetermined topics that potentially relate to the plurality of content items comprises identifying the one or more predetermined topics from a stored list of topics.

8. The method of claim 7, further comprising adding the additional topic to the stored list of topics.

9. The method of claim 1, further comprising:

determining one or more suggested topics for the second subset of content items; and

providing, in connection with providing the first unassigned content item from the second subset of content items, the one or more suggested topics to the client device associated with the user.

10. The method of claim 1, wherein analyzing the plurality of content items to determine the first subset of content items of the plurality of content items that correlate to the one or more predetermined topics comprises using a topic probability model to determine the first subset of content items of the plurality of content items correlate to the one or more predetermined topics.

11. The method of claim 10, wherein using the topic probability model to determine the first subset of content items of the plurality of content items correlate to the one or more predetermined topics comprises determining an odds ratio between each of the plurality of content items and the one or more predetermined topics.

12. The method of claim 10, wherein using the topic probability model, for each topic of the one or more predetermined topics, comprises:

determining a most likely estimate probability of a topic of the one or more predetermined topics given a content item of the plurality of content items;

determining a most likely estimate probability of the topic of the one or more predetermined topics given the plurality of content items; and

determining an allocation-based probability of the one or more predetermined topics given the content item of the plurality of content items.

13. The method of claim 12, wherein the allocation-based probability is a Latent Dirichlet Allocation (LDA)-based probability.

14. The method of claim 12, wherein assigning the first subset of content items within the plurality of content items to the one or more predetermined topics comprises using the topic probability model and the one or more predetermined topics to map the first subset of content items within the plurality of content items to a topic of the one or more predetermined topics.

15. The method of claim 12, further comprising mapping the first subset of content items within the plurality of content items to the topic of the one or more predetermined topics when the first subset of content items within the plurality of content items is more likely than not to correspond to the one or more predetermined topics.

16. A method for organizing survey text responses, the method comprising:

receiving a plurality of text responses to a survey question;

identifying a list of topics comprising one or more predetermined topics that potentially relate to the plurality of text responses;

automatically assigning, by at least one processor and without user input, a first subset of text responses from the plurality of the text responses to the one or more predetermined topics by analyzing the plurality of text responses to determine which text responses of the plurality of text responses that correlate to the one or more predetermined topics;

identifying a second subset of text responses from the plurality of text responses that is assigned to an unknown topic category, wherein a given text response in the second subset of text responses is assigned to the unknown topic category when the given text response is not assigned to the one or more predetermined topics;

providing a first text response assigned to the unknown topic category from the second subset of text responses comprising text responses assigned to the unknown topic category to a client device associated with a survey reviewer;

receiving, from the client device associated with the survey reviewer, an additional topic corresponding to the first text response assigned to an unknown topic category from the second subset of text responses comprising text responses assigned to the unknown topic category, wherein the additional topic is a different topic from the one or more predetermined topics;

automatically reassigning, by the at least one processor and without user input, a second text response assigned to the unknown topic category from the plurality of text responses to the additional topic received from the client device by re-analyzing the second subset of text responses to determine that the second text response correlates to the additional topic; and

providing, to the client device associated with the survey reviewer, topic assignment results for the text responses organized by the one or more predetermined topics and the additional topic.

17. The method of claim 16, further comprising adding the additional topic to the list of topics.

18. The method of claim 16, further comprising:

determining one or more suggested topics for the second subset of text responses; and

providing, in connection with providing the first text response assigned to the unknown topic category from the second subset of text responses, one or more suggested topics to the client device associated with the survey reviewer.

19. The method of claim 18, further comprising:

determining one or more communities of words based on text content within the plurality of text responses;

determining one or more representative words for each of the one or more communities of words;

identifying a connection between the first text response assigned to the unknown topic category and at least one representative word from the one or more representative words; and

based the connection between the first text response assigned to the unknown topic category and the at least one representative word, providing the at least one representative word to the client device as part of the one or more suggested topics.

20. A survey management system for organizing survey responses, the system comprising:

at least one processor; and

at least one non-transitory computer readable storage medium storing instructions that, when executed by at least one processor, cause the system to;

receive a plurality of content items;

identify one or more predetermined topics that potentially relate to the plurality of content items;

automatically assign, without user input, a first subset of content items from the plurality of the content items to the one or more predetermined topics by analyzing the plurality of content items to determine the first subset of content items of the plurality of content items that correlate to the one or more predetermined topics;

identify a second subset of content items from the plurality of content items that comprise content items unassigned to the one or more predetermined topics;

provide a first unassigned content item from the second subset of content items comprising unassigned content items to a client device associated with a user;

receive, from the client device associated with the user, an additional topic corresponding to the first unassigned content item from the second subset of content items comprising unassigned content items, wherein the additional topic is a different topic from the one or more predetermined topics;

re-analyze the plurality of content items to determine that a second unassigned content item from the second subset of content items relates to the additional topic;

automatically assign, without user input, the second unassigned content item from the second subset of content items to the additional topic based on re-analyzing the plurality of content items; and

provide, to the client device associated with the user, topic assignment results organized by the one or more predetermined topics and the additional topic.