Mechanical Turk integrated IDE, systems and methods

• US 10,147,038 B2
• Filed: 08/05/2016
• Issued: 12/04/2018
• Est. Priority Date: 04/19/2012
• Status: Active Grant

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Ehsani, Farzad, Witt-Ehsani, Silke Maren, Master, Demitrios Leo
• Primary Examiner(s): TECKLU, ISAAC TUKU
• Application Number:
  US15/230,131
• Publication Number:
  US 20160342898A1
• Time in Patent Office: 851 Days
• Field of Search: None
• CPC Class Codes: G06F 16/168 Details of user interfaces specifically adapted to file systems, e.g. browsing and visualisation, 2d or 3d GUIs query results presentation G06F16/156
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/2471 Distributed queries
• CPC Class Codes: G06F 8/41 Compilation
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition

Abstract

A Mechanical Turk-Integrated Development Environment system is disclosed. An integrated development environment (IDE) can include one or more interfaces capable of communicating with a mechanical turk engine. As a developer creates applications within the IDE, the developer can use the IDE to submit one or more requests to the mechanical turk engine. The engine constructs a mechanical turk project based on the requests and provides project tasks to workers. The results of the tasks can then be compiled and integrated back into the developer'"'"'s application via the IDE. An example use includes constructing large domain specific data sets that can be applied to spoken dialog interfaces.

Claims

1. A computer implemented method of integrating a trained model trained on a training data set into an application, the method comprising:

instantiating, by a project engine, a mechanical turk project that includes a plurality of tasks available to project members;

receiving, by the project engine, project results data from the project members;

generating, by the project engine, a training data set from the project results data;

training, by the project engine, a trained model from the training data set;

converting, by an application generator module, the project results data in the form of the trained model into a trained application module formatted for integration into an application; and

causing, by the application generator module, integration of the trained application module into the application.

2. The method of claim 1, wherein the project engine comprises at least one of the following:

an integrated development environment, a mechanical turk engine, and an application developer engine.

3. The method of claim 1, wherein the plurality of tasks includes tasks based on at least one of the following modalities of data:

human speech, text, visual data, kinesthetic data, auditory data, and location data.

4. The method of claim 1, wherein the plurality of tasks comprise game tasks.

5. The method of claim 4, wherein the plurality of tasks comprise character action tasks.

6. The method of claim 1, wherein the plurality of tasks comprise question-answer tasks.

7. The method of claim 1, wherein the plurality of tasks comprise picture tasks.

8. The method of claim 7, wherein the picture tasks include taking picture tasks.

9. The method of claim 1, wherein the plurality of tasks comprise categorization tasks.

10. The method of claim 9, wherein the categorization tasks comprise food categorization tasks.

11. The method of claim 1, wherein the plurality of tasks comprise conceptual mapping tasks.

12. The method of claim 1, further comprising validating the project results data.

13. The method of claim 12, further comprising formatting the validated project results data into the training data set.

14. The method of claim 12, wherein the mechanical turk project comprises a mechanical turk project results validation project.

15. The method of claim 14, wherein the plurality of tasks comprise validation tasks.

16. The method of claim 1, wherein the training data set comprises a domain-specific training set.

17. The method of claim 1, wherein the project members comprise trained members.

18. The method of claim 17, wherein the trained members comprises subject matter experts.

19. The method of claim 1, wherein the project results data includes at least one of the following:

raw data, text data, audio data, image data, serial stream data, annotation data, scripts, data files, and data records.

20. The method of claim 1, further comprising updating the project results data as time passes.

21. The method of claim 20, wherein the step of updating the project results data includes updating the results based on usage of the application.

22. The method of claim 21, further comprising updating the trained model based on the updated project results data.

23. The method of claim 22, further comprising updating the application based on the updated trained model.

24. The method of claim 1, wherein the application module comprises at least one of the following:

a source control file, a script file, an image file, an audio file, a text file, a database record, a library of callable functions, compiled code, a serialized data structure, and a domain module.

25. The method of claim 1, wherein the step of converting the project results data includes compiling the project results data in the form of the trained model into the trained application module.