Real time object scanning using a mobile phone and cloud-based visual search engine

• US 9,633,272 B2
• Filed: 02/15/2013
• Issued: 04/25/2017
• Est. Priority Date: 02/15/2013
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Zhong, Yu, Garrigues, Pierre, Culpepper, Benjamin Jackson
• Primary Examiner(s): Park, Soo Jin
• Application Number:
  US13/768,051
• Publication Number:
  US 20140233854A1
• Time in Patent Office: 1,530 Days
• Field of Search: None
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A system for tagging an object comprises and interface and a processor. The interface is configured to receive an image. The processor is configured to determine a key frame. Determining a key frame comprises determining that the image is stable. The processor is configured to determine a tag for an item in the key frame.

Claims

1. A system, comprising:

an interface configured to receive, via a network, an image generated by a camera of a mobile device;

a memory; and

a processor, at least one of the processor or the memory being configured to;

determine a key frame of the image, wherein determining a key frame comprises determining that the image is stable;

determine a tag for an item in the key frame based, at least in part, upon the tag being received from one or more human taggers;

determine a validity of the tag received from the one or more human taggers based, at least in part, upon a status of the one or more human taggers from which the tag was received; and

providing tagging results to the mobile device in real time based, at least in part, on the validity of the tag received from the one or more human taggers;

wherein a plurality of human taggers including the human taggers are divided into separate groups according to one or more characteristics including their average time of providing tags.

2. A system as in claim 1, wherein determining that the image is stable comprises determining that at least a particular number of a plurality of features of the image remain that have not moved more than a threshold.

3. A system as in claim 2, wherein a percentage of the plurality of features is above a threshold for a predetermined number of frames.

4. A system as in claim 2, wherein a number of the plurality of features is above a threshold for a predetermined number of frames.

5. A system as in claim 2, wherein each of the plurality of features is in a current frame and a prior frame.

6. A system as in claim 1, at least one of the processor or the memory being configured to:

determine movement associated with a plurality of features of the image;

in the event that a specific feature of the plurality of features has movement above a threshold, remove the specific feature from the plurality of features such that a reduced set of features is generated;

determine a tag for an item in the key frame based, at least in part, upon at least a portion of the reduced set of features;

wherein the specific feature is in the current frame and in a prior frame.

7. A system as in claim 1, wherein the mobile device comprises a mobile phone.

8. A system as in claim 1, wherein the mobile device comprises a tablet.

9. A system as in claim 1, wherein the key frame has visual content different from visual content from a previous key frame.

10. A system as in claim 1, the human taggers being associated with a particular specialty, wherein the status of each of the human taggers indicates whether the human tagger is an expert.

11. A system as in claim 1, wherein the one or more human taggers have a common set of characteristics including a particular location.

12. A computer-implemented method, comprising:

receiving, via a network, an image from a mobile device;

determining, by a processor, a key frame of the image, wherein determining a key frame comprises determining that an image is stable;

determining a tag for an item in the key frame based, at least in part, upon the tag being received from one or more human taggers;

determining a validity of the tag received from the one or more human taggers based, at least in part, upon a status of the one or more human taggers from which the tag was received; and

providing tagging results to the mobile device in real time based, at least in part, on the validity of the tag received from the one or more human taggers;

wherein the one or more human taggers share a common set of characteristics, the common set of characteristics including a particular average time to provide results;

wherein a plurality of human taggers including the human taggers are divided into separate groups according to one or more characteristics including their average time of providing tags.

13. A computer-implemented method as in claim 12, wherein the tag is determined using a human computation module configured to communicate with the one or more human taggers.

14. A computer-implemented method as in claim 13, further comprising:

requesting, by the human computation module, the tag from the one or more human taggers.

15. A computer-implemented method as in claim 14, wherein the one or more human taggers have a common set of characteristics, the common set of characteristics including at least one of a location or specialty.

16. A computer-implemented method as in claim 13, wherein the tag is determined using the human computation module after determining that the tag could not be determined using a computer vision module.

17. A computer-implemented method as in claim 12, wherein the status of each of the human taggers indicates whether the human tagger is a reliable tagger.

18. A computer-implemented method as in claim 12, the human taggers being associated with a particular specialty, wherein the status of each of the human taggers indicates whether the human tagger is an expert.

19. A computer-implemented method as in claim 12, further comprising:

marking the tag received from the one or more human taggers with a validity status based on the status of the one or more human taggers.

20. A computer-implemented method as in claim 12, wherein the tag is provided for display via the mobile device.

21. The method as recited in claim 12, wherein the common set of characteristics comprise a particular location and a particular specialty.

22. A computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for:

receiving, via a network, an image from a mobile device;

determining, by a processor, a key frame of the image, wherein determining a key frame comprises determining that an image is stable;

determining a tag for an item in the key frame based, at least in part, upon the tag being received from one or more human taggers;

determining a validity of the tag received from the one or more human taggers based, at least in part, upon a status of the one or more human taggers from which the tag was received; and

providing tagging results to the mobile device in real time based, at least in part, on the validity of the tag received from the one or more human taggers;

wherein a plurality of human taggers including the human taggers are divided into separate groups according to one or more characteristics including their average time of providing tags.

23. A computer program product as in claim 22, wherein the image is stable if the image has been still for at least a predetermined number of frames.

24. A computer program product as in claim 22, further comprising:

determining whether to add the image to an object database in association with the tag; and

storing the image in the object database in association with the tag according to a result of determining whether to add the image to the object database in association with the tag.

25. A computer program product as in claim 22, further comprising instructions for:

extracting features at object contours of the image;

removing features from the extracted features that are above a blurriness threshold such that a reduced set of features is generated; and

determining a tag for an item in the key frame based, at least in part, upon at least a portion of the reduced set of features.