System and method for image annotation

• US 10,360,257 B2
• Filed: 08/08/2017
• Issued: 07/23/2019
• Est. Priority Date: 08/08/2017
• Status: Active Grant

• Current Assignee: TuSimple, Inc. (TuSimple Holdings, Inc.)
• Original Assignee: Tusimple
• Inventors: Hou, Xiaodi, Liu, Siyuan, Zhou, Kai
• Primary Examiner(s): Hong, Stephen S
• Assistant Examiner(s): Ludwig, Matthew J
• Application Number:
  US15/672,217
• Publication Number:
  US 20190050428A1
• Time in Patent Office: 714 Days
• Field of Search: 715230, 715231, 715233
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06Q 30/0283 Price estimation or determination

Abstract

A system and method for implementing an image annotation platform are disclosed. A particular embodiment includes: registering a plurality of labelers to which annotation tasks are assigned; assigning annotation tasks to the plurality of labelers; determining if the annotation tasks can be closed or re-assigned to the plurality of labelers; aggregating annotations provided by the plurality of labelers as a result of the closed annotation tasks; evaluating a level of performance of the plurality of labelers in providing the annotations; and calculating payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

Claims

1. A system comprising:

a data processor; and

an image annotation module, executable by the data processor, the image annotation module being configured to perform an image annotation operation, the image annotation operation being configured to;

register a plurality of labelers to which annotation tasks are assigned;

assign annotation tasks to the plurality of labelers;

randomly assigning evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;

enabling the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;

determine if the annotation tasks can be closed or re-assigned to the plurality of labelers;

aggregate annotations provided by the plurality of labelers as a result of the closed annotation tasks;

evaluate a level of performance of the plurality of labelers in providing the annotations; and

calculate payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

2. The system of claim 1 wherein the plurality of labelers are in networked data communication with the data processor.

3. The system of claim 1 wherein at least one of the assigned annotation tasks is an evaluation task.

4. The system of claim 1 being further configured to enable each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

5. The system of claim 1 being further configured to determine if an annotation task needs to be re-assigned to another labeler.

6. The system of claim 1 wherein the annotations correspond to object labels identified in an input image.

7. The system of claim 1 being further configured to receive images from a video feed from an autonomous vehicle.

8. A method comprising:

registering a plurality of labelers to which annotation tasks are assigned;

assigning annotation tasks to the plurality of labelers;

randomly assigning evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;

enabling the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;

determining if the annotation tasks can be closed or re-assigned to the plurality of labelers;

aggregating annotations provided by the plurality of labelers as a result of the closed annotation tasks;

evaluating a level of performance of the plurality of labelers in providing the annotations; and

calculating payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

9. The method of claim 8 wherein the plurality of labelers are in networked data communication.

10. The method of claim 8 wherein at least one of the assigned annotation tasks is an evaluation task.

11. The method of claim 8 including enabling each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

12. The method of claim 8 including determining if an annotation task needs to be re-assigned to another labeler.

13. The method of claim 8 wherein the annotations correspond to object labels identified in an input image.

14. The method of claim 8 including receiving images from a video feed from an autonomous vehicle.

15. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

register a plurality of labelers to which annotation tasks are assigned;

assign annotation tasks to the plurality of labelers;

randomly assign evaluation tasks to the plurality of labelers, the evaluation tasks configured to evaluate the quality of the annotations provided by the plurality of labelers;

enable the plurality of labelers in an annotation verification chain to add, delete, or modify annotations provided by sequentially prior labelers;

determine if the annotation tasks can be closed or re-assigned to the plurality of labelers;

aggregate annotations provided by the plurality of labelers as a result of the closed annotation tasks;

evaluate a level of performance of the plurality of labelers in providing the annotations; and

calculate payments for the plurality of labelers based on the quantity and quality of the annotations provided by the plurality of labelers.

16. The machine-useable storage medium of claim 15 wherein the plurality of labelers are in networked data communication with the data processor.

17. The machine-useable storage medium of claim 15 wherein at least one of the assigned annotation tasks is an evaluation task.

18. The machine-useable storage medium of claim 15 wherein the instructions being further configured to enable each of the plurality of labelers to add, delete, or modify annotations provided by prior labelers.

19. The machine-useable storage medium of claim 15 wherein the instructions being further configured to determine if an annotation task needs to be re-assigned to another labeler.

20. The machine-useable storage medium of claim 15 wherein the instructions being further configured to receive images from a video feed from an autonomous vehicle.