Device and method for hierarchical object recognition

• US 10,319,151 B2
• Filed: 07/07/2017
• Issued: 06/11/2019
• Est. Priority Date: 07/07/2017
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Motorola Solutions, Inc.
• Inventors: Lim, Bing Qin, Khoo, Teik Kean, Tham, Mun Yew
• Primary Examiner(s): Lett, Thomas J
• Application Number:
  US15/643,789
• Publication Number:
  US 20190012836A1
• Time in Patent Office: 704 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 12/08 in hierarchically structured memory systems, e.g. virtual memory systems
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/235 based on user input or interaction
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

A device and method for hierarchical object recognition is provided. The device comprises: an augmented reality display device including a camera, a display device, and an eye-tracking device; an input device; a memory storing a hierarchical object recognition library arranged in a plurality of levels; and a controller. The controller receives, using the input device, an indication of a selected level of the hierarchical object recognition library. The controller determines, using the eye-tracking device, an eye-gaze direction. The controller recognizes at least one object in an image from the camera in the eye-gaze direction by comparing at least a region of the image in the eye-gaze direction with the selected level of the hierarchical object recognition library. The controller controls the display device to indicate a recognized object in the eye-gaze direction.

Claims

1. A device comprising:

an augmented reality display device including a camera, a display device, and an eye-tracking device;

an input device;

a memory storing a hierarchical object recognition library arranged in a plurality of levels;

a communication interface; and

a controller configured to;

receive, using the input device, an indication of a selected level of the hierarchical object recognition library;

determine, using the eye-tracking device, an eye-gaze direction;

recognize at least one object in an image from the camera in the eye-gaze direction by comparing at least a region of the image in the eye-gaze direction with the selected level of the hierarchical object recognition library;

control the display device to indicate a recognized object in the eye-gaze direction; and

when an object to be recognized is partially out of a field-of-view of the camera, transmit, using the communication interface, a request to a server to acquire a respective image of the object to be recognized using one or more other cameras.

2. The device of claim 1, wherein the controller is further configured to control the display device to indicate the recognized object in the eye-gaze direction by providing an outline around the recognized object.

3. The device of claim 1, wherein the selected level of the hierarchical object recognition library indicates a size of objects to be recognized in the eye-gaze direction.

4. The device of claim 1, wherein the plurality of levels of the hierarchical object recognition library are organized according to object size.

5. The device of claim 1, wherein the plurality of levels of the hierarchical object recognition library are organized according to one or more of object components and object sub-components.

6. The device of claim 1, wherein the controller is further configured to, as the eye-gaze direction changes, recognize the at least one object in the image in a changed eye-gaze direction, and update the display device to indicate a currently recognized object in the changed eye-gaze direction, rather than a previously recognized object in a previous eye-gaze direction.

7. The device of claim 1, wherein the controller is further configured to recognize the at least one object in the image from the camera in the eye-gaze direction using edge detection.

8. The device of claim 1, wherein the controller is further configured to, when no objects are recognized using the selected level of the hierarchical object recognition library, control the display device to indicate an object in the eye-gaze direction based on an object size corresponding to the selected level.

9. The device of claim 1, wherein the input device comprises a touch panel, and the controller is further configured to receive the indication of the selected level of the hierarchical object recognition library by detecting one or more of touch input and sliding touch input at the touch panel.

10. A method comprising:

at a device comprising;

an augmented reality display device including a camera, a display device, and an eye-tracking device;

an input device;

a memory storing a hierarchical object recognition library arranged in a plurality of levels; and

a controller, receiving, using the input device, an indication of a selected level of the hierarchical object recognition library;

determining, using the eye-tracking device, an eye-gaze direction;

recognizing, using the controller, at least one object in an image from the camera in the eye-gaze direction by comparing at least a region of the image in the eye-gaze direction with the selected level of the hierarchical object recognition library;

controlling, using the controller, the display device to indicate a recognized object in the eye-gaze direction; and

when an object to be recognized is partially out of a field-of-view of the camera, transmitting, using a communication interface of the device, a request to a server to acquire a respective image of the object to be recognized using one or more other cameras.

11. The method of claim 10, further comprising controlling, using the controller, the display device to indicate the recognized object in the eye-gaze direction by providing an outline around the recognized object.

12. The method of claim 10, wherein the selected level of the hierarchical object recognition library indicates a size of objects to be recognized in the eye-gaze direction.

13. The method of claim 10, wherein the plurality of levels of the hierarchical object recognition library are organized according to object size.

14. The method of claim 10, wherein the plurality of levels of the hierarchical object recognition library are organized according to one or more of object components and object sub-components.

15. The method of claim 10, further comprising, as the eye-gaze direction changes, recognizing, using the controller, the at least one object in the image in a changed eye-gaze direction, and update the display device to indicate a currently recognized object in the changed eye-gaze direction, rather than a previously recognized object in a previous eye-gaze direction.

16. The method of claim 10, further comprising recognizing, using the controller, the at least one object in the image from the camera in the eye-gaze direction using edge detection.

17. The method of claim 10, further comprising, when no objects are recognized using the selected level of the hierarchical object recognition library, controlling, using the controller, the display device to indicate an object in the eye-gaze direction based on an object size corresponding to the selected level.

18. A device comprising:

a display device;

an input device;

a memory storing a hierarchical object recognition library arranged in a plurality of levels; and

a controller communicatively coupled to the display device, the input device, and the memory, the controller configured to;

receive an image including a plurality of objects;

receive, using the input device, an indication of a selected level of the hierarchical object recognition library;

determine an eye-gaze direction;

recognize at least one of the objects in the image by comparing at least a region of the image in the eye-gaze direction with the selected level of the hierarchical object recognition library;

control the display device to indicate a recognized object in the eye-gaze direction; and

when an object to be recognized is partially out of a field-of-view of a camera that acquired the image, transmit a request to a server to acquire a respective image of the object to be recognized using one or more other cameras.