Gaze assisted object recognition

• US 9,317,113 B1
• Filed: 05/31/2012
• Issued: 04/19/2016
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Karakotsios, Kenneth M., Noble, Isaac S.
• Primary Examiner(s): Danielsen, Nathan
• Application Number:
  US13/485,195
• Time in Patent Office: 1,419 Days
• Field of Search: 345 7- 9, 345156-158, 345/419, 345/633, 351/209, 359/630, 382/103, 715/784
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof

Abstract

An electronic device can attempt to determine a gaze direction of a user, which can be used to determine an object of interest to the user. Determining the gaze direction helps to reduce the search space and reduce processing requirements for identifying the object. Image information can be captured that includes the object of interest, which then can be analyzed to recognize the type of object. Upon recognizing the object, the user can be provided with information about the object, which in some cases can depend at least in part upon a current context or location of the object. If the object is a networked device, the user can potentially be provided with input options for controlling the device.

Claims

1. A computer-implemented method for recognizing at least one object, comprising:

capturing a first image including at least a portion of a face of a user using a first camera on a first side of a computing device, the face is within a first field of view of the computing device;

capturing a second image including at least one object using a second camera on a second side of the computing device, the at least one object is within a second field of view of the computing device that is different than the first field of view;

determining, using at least one processor of the computing device and based at least in part upon where the face is in the first field of view, that the face is directed toward the at least one object;

determining, based at least in part upon the second image, a location of the at least one object;

obtaining information for the at least one object;

providing the information for the at least one object to the user;

receiving, via a user command, a selection associated with the information; and

causing the at least one object to be operated.

2. The computer-implemented method of claim 1, wherein the at least one object is an electronic device with which the computing device is able to communicate over at least one communication channel, and

wherein the information indicates at least one command enabling the user to provide input to the electronic device using the computing device.

3. The computer-implemented method of claim 2, wherein providing the information to the user includes providing a control scheme via the computing device, the control scheme enabling the computing device to function as a control mechanism for the electronic device.

4. The computer-implemented method of claim 2, further comprising:

determining a duration of time that the face is directed toward the at least one object; and

initiating the recognizing of the at least one object after the duration exceeds a threshold value.

5. The computer-implemented method of claim 2, wherein the at least one object includes a book, and wherein the information for the recognized at least one object includes at least title and author information.

6. A computer-implemented method, comprising:

under control of one or more computing systems configured with executable instructions,

determining, using at least one processor of the computing device and based at least in part on a relative position of a user in a first field of view of the computing device, that a face of a user is directed toward an object;

determining the object is within a second field of view of the computing device, the first field of view and the second field of view are different;

recognizing a characteristic of the object;

obtaining information associated with the object;

providing the information;

receiving, via a user command, a selection associated with the information; and

causing the object to be operated.

7. The computer-implemented method of claim 6, further comprising:

capturing video using at least one camera, the video including at least a portion of the face of the user;

locating position information for the user'"'"'s eyes in the video; and

determining that the face of the user is directed toward the object based at least in part upon the position information for the user'"'"'s eyes.

8. The computer-implemented method of claim 7, wherein the at least one camera includes a single digital camera, a plurality of digital cameras adapted to capture stereoscopic images, or at least one infrared-capable camera.

9. The computer-implemented method of claim 7, wherein the at least one camera is located on the computing device or on a pair of glasses worn by the user, the pair of glasses including a communication mechanism enabling the pair of glasses to communicate with the computing device.

10. The computer-implemented method of claim 9, wherein the pair of glasses includes one infrared emitter and one infrared sensor for each eye of the user.

11. The computer-implemented method of claim 9, wherein the pair of glasses communicate the information to the user using one or more speakers of the pair of glasses or a heads-up display provided on at least one lens of the pair of glasses.

12. The computer-implemented method of claim 9, wherein the at least one camera is located on the computing device and the computing device communicates the information to the user using at least one of a display of the computing device or one or more speakers of the computing device.

13. The computer-implemented method of claim 6, wherein the object is an electronic device, and

wherein the information indicates at least one input enabling the user to adjust at least one functional aspect of the electronic device.

14. The computer-implemented method of claim 6, wherein the object is identified using at least one of an image recognition algorithm, a facial recognition algorithm, a pattern recognition algorithm, a barcode detection algorithm, or an optical character recognition algorithm.

15. The computer-implemented method of claim 6, further comprising:

capturing image information of at least one object within a field of view of at least one camera; and

determining a location of each of the at least one object.

16. The computer-implemented method of claim 15, wherein the at least one camera includes a single digital camera or a plurality of digital cameras adapted to capture stereoscopic images.

17. The computer-implemented method of claim 15, wherein the at least one camera is located on the computing device or on a pair of glasses worn by the user, the pair of glasses including a communication mechanism enabling the pair of glasses to communicate with the computing device.

18. The computer-implemented method of claim 6, further comprising receiving a specified input corresponding with the object.

19. The computer-implemented method of claim 18, wherein the specified input includes at least one of a motion, a gesture, an audio command, or a physical input to the computing device.

20. The computing device of claim 19, wherein the at least one camera includes at least one of at least one digital camera, a plurality of cameras for capturing stereoscopic images, or at least one infrared-capable camera.

21. The computer-implemented method of claim 6, further comprising estimating that the face of the user is directed toward the object is based at least in part upon a direction of a head of the user.

22. A computing device, comprising:

a processor;

at least one camera; and

a memory device including instructions that, when executed by the processor, cause the computing device to;

determine, using at least a first image captured by the at least one camera, and a relative position of a user in the first image, that a face of the user is directed toward an object;

recognize a characteristic of the object, using at least a second image, to obtain information about the object;

provide the information;

receive, via a user command, a selection associated with the information; and

cause the object to be operated.

23. The computing device of claim 22, further comprising providing at least a portion of the at least the second image to a service configured to perform at least one of image matching or object recognition on the at least the second image in order to identify the object.

24. The computing device of claim 22, wherein a location of the object and that the face of the user is directed toward the object is determined based at least in part upon a relative orientation of the computing device to the user and the object.

25. A non-transitory computer-readable storage medium including instructions that, when executed by a processor of a computing device, cause the computing device to:

capture a first image including at least a portion of a face of a user using a first camera on a first side of the computing device, the face is within a first field of view of the computing device;

capture a second image including an object using a second camera on a second side of the computing device, the object is within a second field of view of the computing device that is different than the first field of view;

determine, based at least in part upon where the face is in the first field of view, that the face is directed toward the object;

determine, based at least in part upon the second image, a location of the object;

obtain information associated with the object;

provide the information associated with the object to the user;

receive, via a user command, a selection associated with the information; and

cause the object to be operated.

26. The non-transitory computer-readable storage medium of claim 25, further comprising determining saccade of the eyes of the user corresponding to positions within a boundary of the object for at least a period of time.

27. The non-transitory computer-readable storage medium of claim 25, wherein the object is an electronic device, and

wherein the information indicates at least one input enabling the user to adjust at least one functional aspect of the electronic device.