Crowdsourced analysis of decontextualized data

• US 10,002,177 B1
• Filed: 09/16/2013
• Issued: 06/19/2018
• Est. Priority Date: 09/16/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: McClintock, Jon Arron, Stathakopoulos, George Nikolaos, Brezinski, Dominique Imjya
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): Bromell, Alexandria
• Application Number:
  US14/028,360
• Time in Patent Office: 1,737 Days
• Field of Search: 707736, 709204, 726 25
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/284 Relational databases

Abstract

Techniques are described for employing a crowdsourcing framework to analyze data related to the performance or operations of computing systems, or to analyze other types of data. A question is analyzed to determine data that is relevant to the question. The relevant data may be decontextualized to remove or alter contextual information included in the data, such as sensitive, personal, or business-related data. The question and the decontextualized data may then be presented to workers in a crowdsourcing framework, and the workers may determine an answer to the question based on an analysis or an examination of the decontextualized data. The answers may be combined, correlated, or otherwise processed to determine a processed answer to the question.

Claims

1. A computer-implemented method, comprising:

receiving, at one or more servers, a question to be answered at least partly by a plurality of workers forming a crowdsourcing framework, the question associated with operations of at least one computing system;

accessing at least one dataset associated with the question, the at least one dataset including data describing the operations of the at least one computing system and contextual information providing context to the data;

receiving a selection of one or more decontextualizing operations to be performed on the at least one dataset associated with the question for altering the contextual information providing the context to the data of the at least one dataset;

transforming the at least one dataset associated with the question to at least one decontextualized dataset by performing the selected one or more decontextualizing operations on the at least one dataset associated with the question, the selected one or more decontextualizing operations at least partly altering the contextual information providing the context to the data of the at least one dataset to transform the at least one dataset to the at least one decontextualized dataset, the contextual information describing one or more of;

an attribute of the at least one computing system;

a characteristic of the data included in the at least one decontextualized dataset comprising one or more of a visual or audio representation;

information regarding an organization associated with the at least one computing system;

or

information regarding an individual associated with the at least one computing system;

transforming the question associated with the operations of the at least one computing system to a modified question by decontextualizing information included in the question, the transforming the question including at least partly altering the contextual information included in the question using the at least one decontextualized dataset transformed according to the selected one or more decontextualizing operations;

sending the at least one decontextualized dataset and the modified question altered using the at least one decontextualized dataset to a plurality of worker devices associated with the plurality of workers in the crowdsourcing framework;

receiving, at the one or more servers, a plurality of answers to the modified question from the plurality of worker devices, the plurality of answers generated by the plurality of workers analyzing the at least one decontextualized dataset in view of the modified question;

processing the plurality of answers to determine a processed answer to the modified question; and

providing the processed answer to at least one requester as a response to the question associated with the operations of the at least one computing system.

2. The method of claim 1, wherein transforming the at least one dataset associated with the question to the at least one decontextualized dataset further comprises altering at least one scale of at least one coordinate axis associated with the data in the at least one dataset to transform the at least one dataset associated with the question to the at least one decontextualized dataset.

3. The method of claim 1, wherein transforming the at least one dataset associated with the question to the at least one decontextualized dataset further comprises inverting the data in the at least one dataset to transform the at least one dataset associated with the question to the at least one decontextualized dataset with regard to at least one coordinate axis.

4. The method of claim 1, wherein the transforming the at least one dataset associated with the question to further comprises normalizing the data in the at least one dataset to transform the at least one dataset associated with the question to the at least one decontextualized dataset with regard to at least one baseline decontextualized dataset, to present at least one variance of the data from the at least one baseline decontextualized dataset.

5. The method of claim 1, wherein the transforming the at least one dataset associated with the question to further comprises representing the data in the at least one dataset to transform the at least one dataset associated with the question to the at least one decontextualized dataset as one or more of a shape, an object, a color, a design, or a sound.

6. A system, comprising:

a processor; and

a memory device communicatively coupled to the processor, the memory device storing instructions which when executed by the processor, cause the processor to instantiate at least one computing device to implement one or more services, wherein the one or more services;

access at least one dataset associated with a question from at least one requester to be answered at least partly by a plurality of workers forming a crowdsourcing framework, the question associated with operations of at least one computing system, the at least one dataset including data describing the operations of the at least one computing system and contextual information providing context to the data;

receive a selection of one or more decontextualizing operations to be performed on the at least one dataset associated with the question for altering the contextual information providing the context to the data of the at least one dataset;

transforming the at least one dataset associated with the question to at least one decontextualized dataset by performing the selected one or more decontextualizing operations on the at least one dataset associated with the question, the selected one or more decontextualizing operations at least partly altering the contextual information providing the context to the data of the at least one dataset to transform the at least one dataset to the at least one decontextualized dataset;

transforming the question associated with the operations of the at least one computing system to a modified question by decontextualizing information included in the question, the transforming the question including at least partly altering the contextual information included in the question using the at least one decontextualized dataset transformed according to the selected one or more decontextualizing operations;

send the at least one decontextualized dataset and the modified question altered using the at least one decontextualized dataset to a plurality of worker devices associated with the plurality of workers in the crowdsourcing framework;

receive, from the plurality of worker devices, a plurality of answers to the modified question, the plurality of answers generated by the plurality of workers analyzing the at least one decontextualized dataset in view of the modified question; and

process the plurality of answers to determine a processed answer to the modified question.

7. The system of claim 6, wherein the contextual information included in the at least one dataset describes one or more of:

an attribute of the at least one computing system;

information regarding an organization associated with the at least one computing system;

or

information regarding an individual associated with the at least one computing system.

8. The system of claim 6, wherein the contextual information included in the at least one dataset describes one or more of:

a visual or audio representation of a characteristic of the data included in the at least one decontextualized dataset;

or

an identification of the data included in the at least one decontextualized dataset.

9. The system of claim 6, wherein the processing of the plurality of answers to determine the processed answer to the modified question further comprises:

determining that at least a threshold proportion of the plurality of answers are a common answer; and

designating the common answer as the processed answer.

10. The system of claim 9, wherein the threshold proportion is based at least partly on the modified question.

11. The system of claim 6, wherein the data included in the at least one dataset comprises one or more of:

a rate of requests to a web site;

an amount of time spent processing the requests;

a delay time in processing the requests;

a rate of errors in processing the requests;

a size of requests to the web site;

or

a size of responses to the requests.

12. The system of claim 6, wherein the at least one decontextualized dataset presents the data in a multidimensional plot to the plurality of workers.

13. One or more non-transitory computer-readable media storing instructions which, when executed by at least one processor, instruct the at least one processor to perform actions comprising:

accessing at least one dataset including data and contextual information providing context to the data, the at least one dataset being associated with a question from at least one requester to be answered at least partly by a plurality of workers in a crowdsourcing framework;

receiving a selection of one or more decontextualizing operations to be performed on the at least one dataset associated with the question for altering the contextual information providing the context to the data of the at least one dataset;

transforming the at least one dataset associated with the question to at least one decontextualized dataset by performing the selected one or more decontextualizing operations on the at least one dataset associated with the question, the selected one or more decontextualizing operations at least partly altering the contextual information providing the context to the data of the at least one dataset to transform the at least one dataset to the at least one decontextualized dataset;

transforming the question associated with operations of the at least one computing system to a modified question by decontextualizing information included in the question, the transforming the question including at least partly altering the contextual information included in the question using the at least one decontextualized dataset transformed according to the selected one or more decontextualizing operations;

sending the at least one decontextualized dataset and the modified question at least partly altered using the at least one decontextualized dataset to a plurality of worker devices associated with the plurality of workers;

receiving, from the plurality of worker devices, a plurality of answers to the modified question, the plurality of answers generated by the plurality of workers analyzing the at least one decontextualized dataset in view of the modified question; and

processing the plurality of answers to determine a processed answer to the modified question.

14. The one or more non-transitory computer-readable media of claim 13, wherein:

transforming the question associated with the operations of the at least one computing system to the modified question includes associated the modified question with the operations of at least one computing system; and

altering the contextual information providing the context to the data includes altering the data describing the operations of the at least one computing system.

15. The one or more non-transitory computer-readable media of claim 13, the actions further comprising:

modifying transforming the at least one dataset by representing the data included in the at least one dataset as one or more of a shape, an object, a color, a design, or a sound.

16. The one or more non-transitory computer-readable media of claim 13, wherein:

the at least one decontextualized dataset includes a plot of data over time; and

the modified question requests that the plurality of workers identify a time period during which the data exhibits an aberration.

17. The one or more non-transitory computer-readable media of claim 13, wherein:

the at least one decontextualized dataset includes a first decontextualized dataset and a second decontextualized dataset; and

the modified question requests that the plurality of workers determine whether the first decontextualized dataset is at least partly correlated with the second decontextualized dataset.

18. The one or more non-transitory computer-readable media of claim 13, wherein:

the at least one decontextualized dataset includes a histogram of data; and

the modified question requests that the plurality of workers determine at least one curve that characterizes the data.

19. The one or more non-transitory computer-readable media of claim 13, wherein the modified question requests that the plurality of workers determine that the at least one decontextualized dataset is substantially similar to at least one other decontextualized dataset.

20. The one or more non-transitory computer-readable media of claim 13, wherein:

the at least one decontextualized dataset is normalized relative to at least one baseline dataset; and

the modified question requests that the plurality of workers determine whether the at least one decontextualized dataset exhibits at least one variance from a norm corresponding to the at least one baseline dataset.