VISUAL SEARCH IN REAL WORLD USING OPTICAL SEE-THROUGH HEAD MOUNTED DISPLAY WITH AUGMENTED REALITY AND USER INTERACTION TRACKING

• US 20150049113A1
• Filed: 01/09/2014
• Published: 02/19/2015
• Est. Priority Date: 08/19/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: RAHMAN, Md Sazzadur, LIU, Kexi, RENSCHLER, Martin H.
• Granted Patent: US 10,152,495 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, an apparatus, and a computer program product conduct online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD). An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. The portion includes searchable content, such as a barcode. The user interaction may be an eye gaze or a gesture. A user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region. The apparatus captures an image of the portion. The identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view. The apparatus transmits the image to a visual search engine.

Claims

1. A method of conducting an online visual search through an augmented reality (AR) device having an optical see-through head mounted display (HMD), said method comprising:

identifying a portion of an object in a field of view of the HMD based on user interaction with the HMD;

capturing an image of the portion, wherein the size of the image is less than the size of the object in the field of view; and

transmitting the image to a search engine.

2. The method of claim 1, wherein the user interaction comprises an eye gaze and identifying a portion of an object comprises:

tracking the eye gaze of the user to determine a location of the eye gaze;

locating a region of the object corresponding to the location of the eye gaze; and

detecting the portion within the region.

3. The method of claim 1, wherein the user interaction comprises a gesture and identifying a portion of an object comprises:

tracking the gesture of the user to determine a location of the gesture;

locating a region of the object corresponding to the location of the gesture; and

detecting the portion within the region.

4. The method of claim 1, further comprising displaying an augmented reality on the HMD in the vicinity of the portion.

5. The method of claim 4, wherein the augmented reality comprises a boundary around the portion.

6. The method of claim 1, further comprising receiving an indication to initiate an online visual search of the object.

7. The method of claim 1, further comprising:

receiving search results from the search engine; and

displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.

8. An apparatus for conducting an online visual search through an augmented reality (AR) device having an optical see-through head mounted display (HMD), said apparatus comprising:

means for identifying a portion of an object in a field of view of the HMD based on user interaction with the HMD;

means for capturing an image of the portion, wherein the size of the image is less than the size of the object in the field of view; and

means for transmitting the image to a search engine.

9. The apparatus of claim 8, wherein the user interaction comprises an eye gaze and the means for identifying a portion of an object is configured to:

track the eye gaze of the user to determine a location of the eye gaze;

locate a region of the object corresponding to the location of the eye gaze; and

detect the portion within the region.

10. The apparatus of claim 8, wherein the user interaction comprises a gesture and the means for identifying a portion of an object is configured to:

track the gesture of the user to determine a location of the gesture;

locate a region of the object corresponding to the location of the gesture; and

detect the portion within the region.

11. The apparatus of claim 8, further comprising means for displaying an augmented reality on the HMD in the vicinity of the portion.

12. The apparatus of claim 11, wherein the augmented reality comprises a boundary around the portion.

13. The apparatus of claim 8, further comprising means for receiving an indication to initiate an online visual search of the object.

14. The apparatus of claim 8, further comprising:

means for receiving search results from the search engine; and

means for displaying the search results as an augmented reality image on the HMD screen in the vicinity of the object.

15. An apparatus for conducting an online visual search through an augmented reality (AR) device having an optical see-through head mounted display (HMD), said apparatus comprising:

a memory; and

at least one processor coupled to the memory and configured to;

identify a portion of an object in a field of view of the HMD based on user interaction with the HMD;

capture an image of the portion, wherein the size of the image is less than the size of the object in the field of view; and

transmit the image to a search engine.

16. The apparatus of claim 15, wherein the user interaction comprises an eye gaze and the processor identifies a portion of an object by being further configured to:

track the eye gaze of the user to determine a location of the eye gaze;

locate a region of the object corresponding to the location of the eye gaze; and

detect the portion within the region.

17. The apparatus of claim 15, wherein the user interaction comprises a gesture and the processor identifies a portion of an object by being further configured to:

track the gesture of the user to determine a location of the gesture;

locate a region of the object corresponding to the location of the gesture; and

detect the portion within the region.

18. The apparatus of claim 15, the processor further configured to display an augmented reality on the HMD in the vicinity of the portion.

19. The apparatus of claim 18, wherein the augmented reality comprises a boundary around the portion.

20. The apparatus of claim 15, the processor further configured to receive an indication to initiate an online visual search of the object.

21. The apparatus of claim 15, the processor further configured to:

receive search results from the search engine; and

display the search results as an augmented reality image on the HMD screen in the vicinity of the object.

22. A computer program product for conducting an online visual search through an augmented reality (AR) device having an optical see-through head mounted display (HMD), said product comprising:

a computer-readable medium comprising code for;

identifying a portion of an object in a field of view of the HMD based on user interaction with the HMD;

capturing an image of the portion, wherein the size of the image is less than the size of the object in the field of view; and

transmitting the image to a search engine.