Gaze assisted object recognition

• US 9,563,272 B2
• Filed: 04/18/2016
• Issued: 02/07/2017
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Karakotsios, Kenneth M., Noble, Isaac S.
• Primary Examiner(s): DANIELSEN, NATHAN ANDREW
• Application Number:
  US15/132,170
• Publication Number:
  US 20160283778A1
• Time in Patent Office: 295 Days
• Field of Search: 345156-158, 351/209, 715/784, 359/630, 382/103
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Claims

1. A computer-implemented method, comprising:

under control of one or more computing systems configured with executable instructions, receiving image data from a camera;

determining, from the image data and using at least one processor of a computing device, a relative position of a user in a first portion of a field of view of the camera;

determining, from the image data, that an object is in a second portion of the field of view;

determining, based at least in part on the relative position of the user, that a face of the user is directed toward the object;

recognizing, based at least in part on the image data, a characteristic of the object;

obtaining first information associated with the object;

providing second information, the second information related to the first information;

receiving, via a user command, a selection associated with the second information; and

causing the object to be operated.

2. The computer-implemented method of claim 1, further comprising connecting the computing device to the object by at least a wireless electronic control channel.

3. The computer-implemented method of claim 1, further comprising connecting the computing device to the object by at least a wired electronic control channel.

4. The computer-implemented method of claim 1, further comprising receiving the image data from at least one of a video camera, a digital camera, an infrared-capable camera or a stereoscopic camera.

5. The computer-implemented method of claim 1, further comprising providing the second information using at least a speaker.

6. The computer-implemented method of claim 1, further comprising receiving the selection associated with the second information using at least a microphone.

7. The computer-implemented method of claim 1, further comprising:

receiving the image data from a video camera, the image data including at least a portion of a face of the user;

locating position information for the user'"'"'s eyes in the image data; and

determining that the face of the user is directed toward the object based at least in part on the position information for the user'"'"'s eyes.

8. The computer-implemented method of claim 1, further comprising:

identifying the object using at least one of an image recognition algorithm, a pattern recognition algorithm, a barcode detection algorithm, or an optical character recognition algorithm.

9. The computer-implemented method of claim 1, further comprising receiving the selection associated with the second information by at least one of a motion, a gesture, an audio command, or a physical input.

10. The computer-implemented method of claim 1, further comprising providing at least a portion of the image data to a service configured to recognize the characteristic of the object.

11. A computing system, comprising:

a camera;

at least one processor; and

a memory device including instructions that, when executed by the at least one processor, cause the computing system to;

receive image data from the camera;

determine, from the image data and using the at least one processor, a relative position of a user in a first portion of a field of view of the camera;

determine, from the image data, that an object is in a second portion of the field of view;

determine, based at least in part on the relative position of the user, that a face of the user is directed toward the object;

recognize, based at least in part on the image data, a characteristic of the object;

obtain first information associated with the object;

provide second information, the second information related to the first information;

receive, via a user input, a selection associated with the second information; and

causing the object to be operated.

12. The computing system according to claim 11, further comprising an electronic communication channel connecting the computing system to the object.

13. The computing system according to claim 12, wherein the electronic communication channel is a wireless electronic control channel.

14. The computing system according to claim 12, wherein the electronic communication channel is a wired electronic control channel.

15. The computing system according to claim 12, wherein the instructions that, when executed by the at least one processor, further cause the computing system to provide a control scheme via the electronic communication channel enabling the computing system to function as a control mechanism for the object.

16. The computing system according to claim 11, wherein the camera includes at least one of a video camera, a digital camera, an infrared-capable camera and a stereoscopic camera.

17. The computing system according to claim 16, wherein the instructions that, when executed by the at least one processor, further cause the computing system to receive the image data from the at least one of a video camera, a digital camera, an infrared-capable camera or a stereoscopic camera.

18. The computing system according to claim 11, further comprising:

at least one audio component, including at least one of a speaker and a microphone.

19. The computing system according to claim 18, wherein the instructions that, when executed by the at least one processor, further cause the computing system to provide the second information using at least the speaker.

20. The computing system according to claim 18, wherein the instructions that, when executed by the at least one processor, further cause the computing system to receive the selection using at least the microphone.