SYSTEMS AND METHODS FOR DISTRIBUTED DATA ANNOTATION

• US 20130097164A1
• Filed: 10/12/2012
• Published: 04/18/2013
• Est. Priority Date: 10/12/2011
• Status: Active Grant

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Perona, Pietro, Welinder, Peter
• Granted Patent: US 8,706,729 B2
• Time in Patent Office: Days
• US Class Current: 707/736
• CPC Class Codes: G06F 16/14 Details of searching files based on file metadata
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/908 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/93 Document management systems
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 18/2185 the supervisor being an automated module, e.g. intelligent oracle
• CPC Class Codes: G06F 18/41 Interactive pattern learning with a human teacher
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 40/67 for remote operation

Abstract

Systems and methods for distributed data annotation in accordance embodiments of the invention are disclosed. In one embodiment of the invention, a distributed data annotation server system includes a storage device configured to store source data, one or more annotators, annotation tasks and a processor, wherein a distributed data annotation application configures the processor to receive source data including one or more pieces of source data, select one or more annotators, create one or more annotation tasks for the selected annotators and source data, request one or more annotations for the source data using the annotation tasks, receive annotations, determine source data metadata for at least one piece of source data using the received annotations, generate annotator metadata for at least one annotator using the received annotations and the source data, and estimate the ground truth for the source data using the source data metadata and the annotator metadata.

Claims

1. A distributed data annotation server system, comprising:

at least one storage device configured to store source data, one or more annotators, annotation tasks, and a distributed data annotation application; and

a processor;

wherein the distributed data annotation application configures the processor to;

receive source data, where the source data comprises one or more pieces of source data;

select one or more annotators for at least one piece of source data;

create one or more annotation tasks for the selected annotators and at least one piece of source data;

request one or more annotations for at least one piece of source data using the created annotation tasks;

receive annotations for at least one piece of source data;

determine source data metadata for at least one piece of source data using the received annotations, where the source data metadata includes source data characteristics;

generate annotator metadata for at least one annotator using the received annotations and at least one piece of source data, where the annotator metadata includes identified annotator characteristics; and

estimate the ground truth for at least one piece of source data using the source data metadata and the annotator metadata.

2. The distributed data annotation server system of claim 1, wherein the distributed data annotation application further configures the processor to estimate the ground truth for at least one piece of source data using the source data metadata and the annotator metadata by iteratively:

updating source data metadata for at least one piece of source data based upon at least the previously generated source data metadata and annotator metadata;

updating annotator metadata for at least one annotator based upon at least the previously generated source data metadata and annotator metadata; and

estimating the ground truth for at least one piece of source data using the updated source data metadata and the updated annotator metadata when a termination condition occurs.

3. The distributed data annotation server system of claim 2, wherein the distributed data annotation application configures the processor to determine source data characteristics in the source data metadata and identify annotator characteristics in the annotator metadata using an iterative maximum a posteriori estimation and the received annotations.

4. The distributed data annotation server system of claim 3, wherein the iterative maximum a posteriori estimation is selected from the group consisting of gradient ascent, gradient descent, and estimation-maximization.

5. The distributed data annotation server system of claim 1, wherein the distributed data annotation application further configures the processor to update source data metadata for at least one piece of source data using the received annotations and the annotator metadata.

6. The distributed data annotation server system of claim 5, wherein the source data metadata includes a measure of the difficulty of describing the source data.

7. The distributed data annotation server system of claim 6, wherein the source data metadata further comprises source data characteristics selected from the group consisting of annotations applied to the piece of source data, features of the source data, and annotators who have previously annotated the piece of source data.

8. The distributed data annotation server system of claim 1, wherein the distributed data annotation application further configures the processor to update annotator metadata for at least one annotator using the received annotations and the source data metadata.

9. The distributed data annotation server system of claim 8, wherein the annotator metadata includes a measure of the competence of the annotator

10. The distributed data annotation server system of claim 9, wherein the annotator metadata further comprises annotator characteristics selected from the group consisting of the expertise of the annotator, the bias of the annotator regarding mislabeling of source data, annotations previously provided by the annotator, and references to source data previously annotated by the annotator.

11. The distributed data annotation server system of claim 1, wherein the distributed data annotation application further configures the processor to determine a confidence threshold value regarding the ground truth of at least one piece of source data.

12. The distributed data annotation server system of claim 1, wherein the distributed data annotation software further configures the processor to determine a cost for performing the annotation task.

13. The distributed data annotation server system of claim 1, wherein:

the annotation task is a human intelligence task; and

the distributed data annotation software further configures the processor to request one or more annotations by submitting at least one annotation task to a human intelligence task marketplace.

14. The distributed data annotation server system of claim 1, wherein:

the annotation task is a machine intelligence task; and

the distributed data annotation software further configures the processor to request one or more annotations by submitting at least one annotation task to an annotation device configured to perform machine intelligence tasks.

15. The distributed data annotation server system of claim 1, wherein selecting one or more annotators for at least one piece of source data comprises selecting one or more annotators based on at least one annotator characteristic in the annotator metadata describing the one or more annotators.

16. The distributed data annotation server system of claim 1, wherein selecting one or more annotators for at least one piece of source data comprises selecting one or more annotators based on at least one source data characteristic in the source data metadata.

17. A method for distributed data annotation, comprising:

receiving source data using a distributed data annotation server system, where the source data comprises one or more pieces of source data;

selecting one or more annotators for at least one piece of source data using the distributed data annotation server system;

creating one or more annotation tasks for the selected annotators and at least one piece of source data using the distributed data annotation server system;

requesting one or more annotations for at least one piece of source data using the created annotation tasks and the distributed data annotation server system;

receiving annotations for at least one piece of source data using the distributed data annotation server system;

determining source data metadata for at least one piece of source data using the received annotations and the distributed data annotation server system, where the source data metadata includes source data characteristics;

generating annotator metadata for at least one annotator using the received annotations, at least one piece of source data, and the distributed data annotation server system, where the annotator metadata includes identified annotator characteristics; and

estimating the ground truth for at least one piece of source data using the source data metadata, the annotator metadata, and the distributed data annotation server system.

18. The method of claim 17, wherein estimating the ground truth for at least one piece of source data using the source data metadata and the annotator metadata further comprises iteratively:

updating source data metadata for at least one piece of source data using the distributed data annotation server system based upon at least the previously generated source data metadata and annotator metadata;

updating annotator metadata for at least one annotator and the distributed data annotation server system based upon at least the previously generated source data metadata and annotator metadata; and

estimating the ground truth for at least one piece of source data using the distributed data annotation server system based upon the updated source data metadata and the updated annotator metadata when a termination condition occurs.

19. The method of claim 18, further comprising determining source data characteristics in the source data metadata and identifying annotator characteristics in the annotator metadata using an iterative maximum a posteriori estimation, the received annotations, and the distributed data annotation server system.

20. The method of claim 19, wherein the iterative maximum a posteriori estimation is selected from the group consisting of gradient ascent, gradient descent, and estimation-maximization.

21. The method of claim 17, further comprising updating source data metadata for at least one piece of source data using the received annotations, the annotator metadata, and the distributed data annotation server system.

22. The method of claim 21, wherein the source data metadata includes a measure of the difficulty of describing the source data.

23. The method of claim 22, wherein the source data metadata further comprises source data characteristics selected from the group consisting of annotations applied to the piece of source data, features of the source data, and annotators who have previously annotated the piece of source data.

24. The method of claim 17, further comprising updating annotator metadata for at least one annotator using the received annotations, the source data metadata, and the distributed data annotation server system.

25. The method of claim 24, wherein the annotator metadata includes a measure of the competence of the annotator

26. The method of claim 25, wherein the annotator metadata further comprises annotator characteristics selected from the group consisting of the expertise of the annotator, the bias of the annotator regarding mislabeling of source data, annotations previously provided by the annotator, and references to source data previously annotated by the annotator.

27. The method of claim 17, further comprising determining a confidence threshold value regarding the ground truth of at least one piece of source data using the distributed data annotation server system.

28. The method of claim 17, further comprising determining a cost for performing the annotation task using the distributed data annotation server system.

29. The method of claim 17, wherein:

the annotation task is a human intelligence task; and

requesting one or more annotations further comprises submitting at least one annotation task to a human intelligence task marketplace using the distributed data annotation server system.

30. The method of claim 17, wherein:

the annotation task is a machine intelligence task; and

requesting one or more annotations further comprises submitting at least one annotation task to an annotation device using the distributed data annotation server system, where the annotation device is configured to perform machine intelligence tasks.

31. The method of claim 17, wherein selecting one or more annotators for at least one piece of source data comprises selecting one or more annotators based on at least one annotator characteristic in the annotator metadata describing the one or more annotators using the distributed data annotation server system.

32. The method of claim 17, wherein selecting one or more annotators for at least one piece of source data comprises selecting one or more annotators based on at least one source data characteristic in the source data metadata using the distributed data annotation server system.

33. A distributed data annotation server system, comprising:

at least one storage device configured to store source data, one or more annotators, annotation tasks, and a distributed data annotation application; and

a processor;

wherein the distributed data annotation application configures the processor to;

receive source data, where the source data comprises one or more pieces of source data;

select one or more annotators for at least one piece of source data;

create one or more annotation tasks for the selected annotators and at least one piece of source data;

request one or more annotations for at least one piece of source data using the created annotation tasks;

receive annotations for at least one piece of source data; and

perform an iterative a posteriori estimation of the ground truth for at least one piece of source data, the difficulty of describing the at least one piece of source data and the competence of the selected annotators that annotated the at least one piece of source data based upon the received annotations.