Mechanical Turk integrated IDE, systems and method

• US 9,436,738 B2
• Filed: 04/19/2013
• Issued: 09/06/2016
• Est. Priority Date: 04/19/2012
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Ehsani, Farzad, Witt-Ehsani, Silke Maren, Master, Demitrios Leo
• Primary Examiner(s): Tecklu, Isaac T
• Application Number:
  US13/866,444
• Publication Number:
  US 20130282757A1
• Time in Patent Office: 1,236 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/168 Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs query results presentation G06F16/156
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 8/41 Compilation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A Mechanical Turk-Integrated Development Environment system is disclosed. An integrated development environment (IDE) can include one or more interfaces capable of communicating with a mechanical turk engine. As a developer creates applications within the IDE, the developer can use the IDE to submit one or more requests to the mechanical turk engine. The engine constructs a mechanical turk project based on the requests and provides project tasks to workers. The results of the tasks can then be compiled and integrated back into the developer'"'"'s application via the IDE. An example use includes constructing large domain specific data sets that can be applied to spoken dialog interfaces.

Claims

1. An application development system comprising:

an application developer engine including a computing device having a computer processor configured to execute stored software instructions; and

a mechanical turk engine including a computing device having a computer processor configured to execute stored software instructions said mechanical turk engine communicatively coupled with the application developer engine and at least one member interface, wherein the mechanical turk engine is configured to;

receive a mechanical turk request;

convert the mechanical turk request into a mechanical turk project;

receive mechanical turk project results from at least one member interface;

validate the mechanical turk project results;

convert mechanical turk project results data into a module formatted for compilation by the application developer engine; and

return project results to the application developer engine.

2. The system of claim 1, wherein the mechanical turk engine comprises a computing device configured as an Integrated Development Environment (IDE).

3. The system of claim 1, wherein the application developer engine and the mechanical turk engine are communicatively coupled over a network.

4. The system of claim 3, wherein the mechanical turk engine comprises an entity unrelated to the application developer engine.

5. The system of claim 1, wherein the mechanical turk request comprises an automatically generated project definition.

6. The system of claim 1, wherein the project results comprise a serialized data stream.

7. The system of claim 1, wherein the project results comprise raw data.

8. The system of claim 1, wherein the project results comprise the domain module.

9. The system of claim 1, wherein the mechanical turk engine comprises the application module generator.

10. The system of claim 1, wherein the application developer engine comprises the application module generator.

11. The system of claim 1, wherein the application developer engine is further configured to generate an application comprising the project results.

12. The system of claim 11, wherein the application configures a computing device on which the application executes to obtain the project results.

13. The system of claim 12, wherein the computing device modifies its behavior according to the application integrating the project results.

14. The system of claim 1, wherein the project results comprise conceptual mappings.

15. The system of claim 14, wherein the conceptual mappings represent a meaning obtained from at least one modality of communication.

16. The system of claim 15, wherein the at least one modality of communication comprises at least one of the following:

human speech, text, visual data, kinesthetic data, auditory data, taste data, ambient data, tactile data, haptic data and location data.

17. The system of claim 1 where the mechanical turk engine is further configured to expand initial concepts based at least in part on a phrase thesaurus.

18. The system of claim 1 where the mechanical turk engine is configured to expand initial concepts based at least in part on external data sources that match a domain of the application.

19. The system of claim 1 further configured to convert the mechanical turk project results into a validation project.

20. The system of claim 19 further configured to receive validation project results from at least one member interface.

21. The system of claim 20 wherein a mechanical turk project results validation is based, at least in part, on validation project results.

22. The system of claim 20 wherein a mechanical turk project results validation is based, at least in part, on two or more validation project results.

23. The system of claim 1 wherein mechanical turk project results are stored with data annotations.

24. The system of claim 23 wherein said data annotations comprise at least one of the following:

project ID, timestamp, mechanical turk member identification and mechanical turk member demographics.

25. The system of claim 23 wherein the mechanical turk project results are validated, at least in part is, based on said data annotations.