Visual search in real world using optical see-through head mounted display with augmented reality and user interaction tracking

• US 10,152,495 B2
• Filed: 01/09/2014
• Issued: 12/11/2018
• Est. Priority Date: 08/19/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Rahman, Md Sazzadur, Liu, Kexi, Renschler, Martin H.
• Primary Examiner(s): Robinson, Chante
• Application Number:
  US14/151,664
• Publication Number:
  US 20150049113A1
• Time in Patent Office: 1,797 Days
• Field of Search: 345156, 345 8, 345418, 345629-635
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 16/332 Query formulation
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9554 by using bar codes
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method, an apparatus, and a computer program product conduct online visual searches through an augmented reality (AR) device having an optical see-through head mounted display (HMD). An apparatus identifies a portion of an object in a field of view of the HMD based on user interaction with the HMD. The portion includes searchable content, such as a barcode. The user interaction may be an eye gaze or a gesture. A user interaction point in relation to the HMD screen is tracked to locate a region of the object that includes the portion and the portion is detected within the region. The apparatus captures an image of the portion. The identified portion of the object does not encompass the entirety of the object. Accordingly, the size of the image is less than the size of the object in the field of view. The apparatus transmits the image to a visual search engine.

Claims

1. A method of conducting an online visual search through an augmented reality (AR) device having a display, said method comprising:

capturing, via an image capture device of the AR device, a scene in a field of view of the display;

identifying, via at least one processor of the AR device, a portion of the scene based on a first user interaction with the display;

displaying AR content on the display in response to the first user interaction, the AR content comprising indicia associated with the identified portion of the scene;

receiving, after displaying the AR content on the display, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction with a search icon displayed on the display, the second user interaction occurring after the first user interaction, wherein the second user interaction comprises a non-eye gaze gesture; and

transmitting, by the AR device in response to the second user interaction, an image of the identified portion of the scene to a search engine, wherein the image includes the identified portion of the scene and does not include content in the field of view of the display outside of the identified portion of the scene.

2. The method of claim 1, wherein the first user interaction comprises an eye gaze and the identifying the portion of the scene comprises:

tracking the eye gaze of a user to determine a location of the eye gaze;

locating a region of the scene corresponding to the location of the eye gaze; and

detecting the portion of the scene within the region.

3. The method of claim 1, wherein the first user interaction comprises a gesture and the identifying the portion of the scene comprises:

tracking the gesture of a user to determine a location of the gesture;

locating a region of the scene corresponding to the location of the gesture; and

detecting the portion of the scene within the region.

4. The method of claim 1, wherein the indicia comprises a boundary around the portion of the scene.

5. The method of claim 1, further comprising:

receiving search results from the search engine; and

displaying the search results as an AR image on the display.

6. The method of claim 1, further comprising:

processing, via the at least one processor of the AR device in response to the second user interaction, the scene to generate the image of the identified portion of the scene for transmission.

7. The method of claim 6, wherein a transceiver of the AR device is configured to transmit the image of the portion of the scene to the search engine, and wherein the method further comprises:

providing, via the at least one processor of the AR device, the image of the identified portion of the scene to the transceiver of the AR device.

8. The method of claim 1, wherein the image capture device is a video camera.

9. An apparatus for conducting an online visual search through an augmented reality (AR) device having a display, said apparatus comprising:

means for capturing a scene in a field of view of the display;

means for identifying a portion of the scene based on a first user interaction with the display;

means for displaying AR content on the display in response to the first user interaction, the AR content comprising indicia associated with the identified portion of the scene;

means for receiving, after displaying the AR content on the display, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction with a search icon displayed on the display, the second user interaction occurring after the first user interaction, wherein the second user interaction comprises a non-eye gaze gesture; and

means for transmitting, in response to the second user interaction, an image of the identified portion of the scene to a search engine, wherein the image includes the identified portion of the scene and does not include content in the field of view of the display outside of the identified portion of the scene.

10. The apparatus of claim 9, wherein the first user interaction comprises an eye gaze and the means for identifying the portion of the scene is configured to:

track the eye gaze of a user to determine a location of the eye gaze;

locate a region of the scene corresponding to the location of the eye gaze; and

detect the portion of the scene within the region.

11. The apparatus of claim 9, wherein the first user interaction comprises a gesture and the means for identifying the portion of the scene is configured to:

track the gesture of a user to determine a location of the gesture;

locate a region of the scene corresponding to the location of the gesture; and

detect the portion of the scene within the region.

12. The apparatus of claim 9, wherein the indicia comprises a boundary around the portion of scene.

13. The apparatus of claim 9, further comprising:

means for receiving search results from the search engine; and

means for displaying the search results as an AR image on the display.

14. The apparatus of claim 9, further comprising:

means for processing, in response to the second user interaction, the scene to generate the image of the identified portion of the scene for transmission.

15. The apparatus of claim 14, further comprising:

means for providing the image of the identified portion of the scene to the means for transmitting the image of the identified portion of the scene to the search engine.

16. An apparatus for conducting an online visual search through an augmented reality (AR) device having a display, said apparatus comprising:

a memory;

an image capture device configured to capture a scene in a field of view of the display;

a transceiver; and

at least one processor coupled to the memory and transceiver, wherein the at least one processor is configured to;

identify a portion of the scene based on a first user interaction with the display;

display AR content on the display in response to the first user interaction, the AR content comprising indicia associated with the identified portion of the scene;

receive, after the AR content is displayed on the display, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction with a search icon displayed on the display, the second user interaction occurring after the first user interaction, wherein the second user interaction comprises a non-eye gaze gesture; and

cause, in response to the second user interaction, the transceiver to transmit an image of the identified portion of the scene to a search engine, wherein the image includes the identified portion of the scene and does not include content in the field of view of the display outside of the identified portion of the scene.

17. The apparatus of claim 16, wherein the first user interaction comprises an eye gaze and to identify the portion of the scene, the at least one processor is configured to:

track the eye gaze of a user to determine a location of the eye gaze;

locate a region of the scene corresponding to the location of the eye gaze; and

detect the portion of the scene within the region.

18. The apparatus of claim 16, wherein the first user interaction comprises a gesture and to identify the portion of the scene, the at least one processor is configured to:

track the gesture of a user to determine a location of the gesture;

locate a region of the scene corresponding to the location of the gesture; and

detect the portion of the scene within the region.

19. The apparatus of claim 16, wherein the indicia comprises a boundary around the scene.

20. The apparatus of claim 16, wherein the at least one processor is configured to:

receive search results from the search engine; and

display the search results as an AR image on the display.

21. The apparatus of claim 16, wherein the at least one processor is configured to:

process, in response to the second user interaction, the scene to generate the image of the identified portion of the scene for transmission.

22. The apparatus of claim 21, wherein the at least one processor is configured to:

provide the image of the identified portion of the scene to the transceiver of the AR device.

23. The apparatus of claim 16, wherein the image capture device is a video camera.

24. A non-transitory computer-readable medium having instructions stored thereon that, when executed, cause at least one processor of an augmented reality (AR) device having a display to:

cause an image capture device of the AR device to capture a scene in a field of view of the display;

identify, via the at least one processor of the AR device, a portion of the scene based on a first user interaction with the display;

display AR content on the display in response to the first user interaction, the AR content comprising indicia associated with the identified portion of the scene;

receive, after the AR content is displayed on the display, an indication to initiate an online visual search of the identified portion of the scene based on a second user interaction with a search icon displayed on the display, the second user interaction occurring after the first user interaction, wherein the second user interaction comprises a non-eye gaze gesture; and

cause, in response to the second user interaction, the AR device to transmit an image of the identified portion of the scene to a search engine, wherein the image includes the identified portion of the scene and does not include content in the field of view of the display outside of the identified portion of the scene.

25. The non-transitory computer-readable medium of claim 24, further comprising instructions stored thereon that, when executed, cause the at least one processor of the AR device to:

process, in response to the second user interaction, the scene to generate the image of the identified portion of the scene for transmission.

26. The non-transitory computer-readable medium of claim 25, wherein a transceiver of the AR device is configured to transmit the image of the portion of the scene to the search engine, and wherein the non-transitory computer-readable medium further comprises instructions that, when executed, cause the at least one processor of the AR device to:

provide the image of the identified portion of the scene to the transceiver of the AR device.

27. The non-transitory computer-readable medium of claim 24, wherein the image capture device is a video camera.