Learning-based data decontextualization

• US 9,342,796 B1
• Filed: 09/16/2013
• Issued: 05/17/2016
• Est. Priority Date: 09/16/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McClintock, Jon Arron, Stathakopoulos, George Nikolaos, Brezinski, Dominique Imjya
• Primary Examiner(s): Chaki, Kakali
• Assistant Examiner(s): Sobkowski, Isidore
• Application Number:
  US14/028,396
• Time in Patent Office: 974 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Techniques are described for employing a crowdsourcing framework to analyze data related to the performance or operations of computing systems, or to analyze other types of data. A question is analyzed to determine data that is relevant to the question. The relevant data may be decontextualized to remove or alter contextual information included in the data, such as sensitive, personal, or business-related data. The question and the decontextualized data may then be presented to workers in a crowdsourcing framework, and the workers may determine an answer to the question based on an analysis or an examination of the decontextualized data. The answers may be combined, correlated, or otherwise processed to determine a processed answer to the question. Machine learning techniques are employed to adjust and refine the decontextualization.

Claims

1. A computer-implemented method, comprising:

accessing a question having a predetermined answer, the question being associated with operations of at least one computing system;

accessing at least one dataset associated with the question, the at least one dataset including data describing the operations of the at least one computing system and contextual information about the data;

selecting at least one decontextualization operation from a plurality of decontextualization operations, wherein the at least one decontextualization operation at least partly alters the contextual information about the data included in the at least one dataset;

applying the at least one decontextualization operation to the at least one dataset to determine at least one modified dataset in which the contextual information about the data included in the at least one dataset is at least partly altered;

sending the at least one modified dataset including the data and the altered contextual information and the question to a plurality of worker devices associated with a plurality of workers in a crowdsourcing framework;

receiving, from the plurality of worker devices, a plurality of answers to the question, the plurality of answers generated by the plurality of workers analyzing the at least one modified dataset having the contextual information about the data at least partly altered and the data in view of the question;

incorporating, into training data, the plurality of answers and information describing the at least one decontextualization operation; and

employing the training data in machine learning to train a decontextualizer to be used in subsequent data decontextualization to answer subsequent questions using the crowdsourcing framework.

2. The method of claim 1, wherein the contextual information describes one or more of:

an attribute of the at least one computing system;

a characteristic of the data included in the at least one dataset;

information regarding an organization associated with the at least one computing system;

or

information regarding an individual associated with the at least one computing system.

3. The method of claim 1, wherein the machine learning is unsupervised machine learning.

4. The method of claim 1, wherein the selecting of the at least one decontextualization operation includes randomly or pseudo-randomly selecting the at least one decontextualization operation from the plurality of decontextualization operations.

5. A system, comprising:

at least one computing device configured to implement one or more services, wherein the one or more services are configured to;

access at least one dataset associated with a question, the at least one dataset including data and contextual information about the data and the question having a predetermined answer;

determine at least one decontextualization operation that at least partly alters the contextual information about the data included in the at least one dataset;

apply the at least one decontextualization operation to the contextual information about the data of the at least one dataset to determine at least one modified dataset in which the contextual information about the data included in the at least one dataset is at least partly altered;

send the at least one modified dataset that includes the data and the at least partly altered contextual information about the data and the question to a plurality of worker devices associated with a plurality of workers in a crowdsourcing framework;

receive, from the plurality of worker devices, a plurality of answers to the question, the plurality of answers generated by the plurality of workers analyzing the at least one modified dataset that includes the data and the at least partly altered contextual information about the data in view of the question;

incorporate, into training data, the plurality of answers and information describing the at least one decontextualization operation; and

employ the training data in machine learning to train a decontextualizer to be used in subsequent data decontextualization to answer subsequent questions using the crowdsourcing framework.

6. The system of claim 5, wherein:

the question is associated with operations of at least one computing system; and

the at least one dataset includes data describing the operations of the at least one the computing system.

7. The system of claim 6, wherein the contextual information included in the at least one dataset describes one or more of:

an attribute of the at least one computing system;

information regarding an organization associated with the at least one computing system;

or

information regarding an individual associated with the at least one computing system.

8. The system of claim 5, wherein the contextual information included in the at least one dataset describes one or more of:

a characteristic of data included in the at least one dataset;

or

an identification of data included in the at least one dataset.

9. The system of claim 5, wherein the at least one dataset includes data comprising one or more of:

a rate of requests to a web site;

an amount of time spent processing the requests;

a delay time in processing the requests;

a rate of errors in processing the requests;

a size of requests to the web site;

or

a size of responses to the requests.

10. The system of claim 5, wherein the machine learning is unsupervised machine learning.

11. The system of claim 5, wherein the machine learning is supervised or semi-supervised machine learning.

12. The system of claim 5, wherein the determining of the at least one decontextualization operation includes randomly or pseudo-randomly selecting the at least one decontextualization operation from a plurality of decontextualization operations.

13. The system of claim 5, wherein the applying of the at least one decontextualization operation to the at least one dataset includes iteratively applying a plurality of different decontextualization operations to the at least one dataset to determine a plurality of differently modified datasets in which the contextual information included in the at least one dataset is at least partly altered.

14. One or more computer-readable media storing instructions which, when executed by at least one processor, instruct the at least one processor to perform actions comprising:

accessing at least one dataset associated with a question, the at least one dataset including data and contextual information about the data;

determining at least one decontextualization operation that at least partly alters the contextual information about the data included in the at least one dataset with the data;

applying the at least one decontextualization operation to the at least one dataset to determine at least one modified dataset that includes the data and the contextual information about the data included in the at least one modified dataset is at least partly altered;

receiving, from a plurality of worker devices associated with a plurality of workers in a crowdsourcing framework, a plurality of answers to the question, the plurality of answers generated by the plurality of workers analyzing the at least one modified dataset having data and at least partly altered contextual information about the data in view of the question;

incorporating, into training data, the plurality of answers and information describing the at least one decontextualization operation; and

employing the training data in machine learning to train a decontextualizer to be used in subsequent data decontextualization to answer subsequent questions using the crowdsourcing framework.

15. The one or more computer-readable media of claim 14, wherein:

the question is associated with operations of at least one computing system; and

the at least one dataset includes data describing the operations of the at least one computing system.

16. The one or more computer-readable media of claim 14, wherein the question is a control question having a predetermined answer.

17. The one or more computer-readable media of claim 14, wherein the applying of the at least one decontextualization operation to the at least one dataset includes iteratively applying a plurality of different decontextualization operations to the at least one dataset to determine a plurality of differently modified datasets in which the contextual information included in the at least one dataset is at least partly altered.

18. The one or more computer-readable media of claim 14, wherein the contextual information included in the at least one dataset describes one or more of:

a characteristic of data included in the at least one dataset;

or

an identification of data included in the at least one dataset.

19. The one or more computer-readable media of claim 14, the actions further comprising:

modifying the question to at least partly alter the contextual information included in the question, prior to sending the question to the plurality of worker devices.

20. The one or more computer-readable media of claim 14, wherein the at least one modified dataset presents data in a multidimensional plot to the plurality of workers.