Determining relative motion as input

• US 9,557,811 B1
• Filed: 10/31/2014
• Issued: 01/31/2017
• Est. Priority Date: 05/24/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bozarth, Bradley J.
• Primary Examiner(s): Bukowski, Kenneth
• Application Number:
  US14/530,438
• Time in Patent Office: 823 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2360/144 the light being ambient light

Abstract

Input can be provided to a computing device based upon relative movement of a user or other object with respect to the device. In some embodiments, infrared radiation is used to determine measurable aspects of the eyes or other of a user. Since the human retina is a retro-reflector for certain wavelengths, using two different wavelengths or two measurement angles can allow user pupils to be quickly located and measured without requiring resource-intensive analysis of full color images captured using ambient light, which can be important for portable, low power, or relatively inexpensive computing devices. Various embodiments provide differing levels of precision and design that can be used with different devices.

Claims

1. A computer-implemented method of enabling a user to provide a control input to an electronic device, comprising:

under control of one or more computing systems configured with executable instructions,

illuminating, by a first infrared (IR) source of the electronic device, at least a portion of a face;

illuminating, by a second IR source of the electronic device, at least the portion of the face;

determining that a third IR sensor is covered, based at least in part on the third IR sensor not receiving reflected IR light;

capturing, by a first IR sensor at a first time, a first image including reflected IR light from the first IR source;

capturing, by a second IR sensor at the first time, a second image including reflected IR light from the second IR source;

determining a first portion of the first image, the first portion including facial features that meet a minimum level of distinctiveness;

locating a second portion of the second image, the second portion of the second image corresponding to the first portion of the first image;

aligning the first portion of the first image and the second portion of the second image, based at least in part on first information;

determining a first position of the face based at least on the first image;

determining a second position of the face based at least on the second image;

determining a relative position of the face based at least on a difference between the first position of the face and the second position of the face; and

determining a control input for the electronic device based on the relative position of the face.

2. The computer-implemented method of claim 1, wherein the first information includes information relating to at least a color, a brightness, a hue, a light level, one or more coordinates, or a pixel.

3. The computer-implemented method of claim 1, wherein aligning the first portion of the first image and the second portion of the second image further comprises:

identifying a first target portion in the second image based at least in part on one or more coordinates of the first portion of the first image;

determining that a first match score between the first portion of the first image and the first target portion in the second image is lower than a threshold match score;

identifying a second target portion in the second image;

determining that a second match score between the first portion of the first image and the second target portion in the second image is above the threshold match score; and

using the second target portion in the second image as the second portion of the second image.

4. The computer-implemented method of claim 1, wherein determining the first portion of the first image further comprises:

determining a first size of the first portion of the first image;

determining that the facial features included in the first portion of the first image do not meet the minimum level of distinctiveness;

determining a second portion of the first image, the second portion having a second size, the second size larger than the first size;

determining that the facial features included in the second portion of the first image meet the minimum level of distinctiveness; and

using the second portion of the first image in place of the first portion of the first image.

5. The computer-implemented method of claim 1, further comprising determining an approximate distance between the electronic device and the portion of the face based at least on the first portion of the first image and the second portion of the second image.

6. The computer-implemented method of claim 1, further comprising:

illuminating, by a third IR source of the electronic device, at least the portion of the face; and

capturing, by a fourth IR sensor, a third image including reflected IR light from the third IR source.

7. The computer-implemented method of claim 1, further comprising:

capturing, by a camera, an ambient light image including at least the portion of the face;

determining an approximate head position of the face with respect to the electronic device based at least in part upon the ambient light image; and

determining the relative position of the face based at least on the ambient light image, the first portion of the first image and the second portion of the second image.

8. The computer-implemented method of claim 1, further comprising:

illuminating, by a third IR source of the electronic device, at least the portion of the face; and

capturing, by the second IR sensor, a third image including reflected IR light from the third IR source.

9. The computer-implemented method of claim 1, further comprising:

illuminating, by the first IR source, radiation within a first range of wavelengths capable of being substantially reflected by areas of the face; and

illuminating, by the second IR source, radiation with a second range of wavelengths capable of being substantially absorbed by the areas of the face.

10. A computing device, comprising:

a first infrared (IR) source;

a second IR source;

a first IR sensor;

a second IR sensor;

a third IR sensor;

at least one processor;

memory including non-transitory instructions that, when executed by the processor, cause the computing device to;

illuminate, by the first IR source, at least a portion of a face;

determine that the third IR sensor is not receiving reflected IR light;

determine, based at least on the third IR sensor not receiving reflected IR light that the third IR sensor is covered;

illuminate, by the second IR source, at least the portion of the face;

capture, by the first IR sensor at a first time, a first image including reflected IR light from the first IR source;

capture, by the second IR sensor at the first time, a second image including reflected IR from the second IR source;

determine a first portion of the first image, the first portion including facial features that meet a minimum level of distinctiveness;

locate a second portion of the second image, the second portion of the second image corresponding to the first portion of the first image;

align the first portion of the first image and the second portion of the second image, based at least in part on first information;

determine a first position of the face based at least on the first image;

determine a second position of the face based at least on the second image;

determine a relative position of the face based at least on a difference between the first position of the face and the second position of the face; and

determine a control input for the computing device based on the relative position of the face.

11. The computing device of claim 10, wherein the first information includes information relating to at least a color, a brightness, a hue, a light level, one or more coordinates, or a pixel.

12. The computing device of claim 10, wherein the memory further includes non-transitory instructions that cause the computing device to:

identify a first target portion in the second image based at least in part on one or more coordinates of the first portion of the first image;

determine that a first match score between the first portion of the first image and the first target portion in the second image is lower than a threshold match score;

identify a second target portion in the second image;

determine that a second match score between the first portion of the first image and the second target portion in the second image is above the threshold match score; and

use the second target portion in the second image as the second portion of the second image.

13. The computing device of claim 10, wherein the memory further includes non-transitory instructions that cause the computing device to:

determine a first size of the first portion of the first image;

determine that the facial features included in the first portion of the first image do not meet the minimum level of distinctiveness;

determine a second portion of the first image, the second portion having a second size, the second size larger than the first size;

determine that the facial features included in the second portion of the first image meet the minimum level of distinctiveness; and

use the second portion of the first image in place of the first portion of the first image.

14. The computing device of claim 10, wherein the first IR sensor is positioned substantially adjacent to the first IR source, and the second IR sensor is positioned substantially adjacent to the second IR source.

15. The computing device of claim 10, wherein the first IR source emits radiation within a first range of wavelengths capable of being substantially reflected by areas of the face, and wherein the second IR source emits radiation within a second range of wavelengths capable of being substantially absorbed by the areas of the face.

16. The computing device of claim 10, further comprising:

a third IR source,

wherein the memory further includes non-transitory instructions that cause the computing device to;

illuminate, by the third IR source, at least the portion of the face; and

capture a third image using the first IR sensor, the third image including reflected IR from the third IR source.

17. The computing device of claim 10, wherein the memory further includes non-transitory instructions that cause the computing device to:

determine an approximate distance between the computing device and the portion of the face based at least on the first portion of the first image and the second portion of the second image.

18. The computing device of claim 10, further comprising:

a third IR source,

wherein the memory further includes non-transitory instructions that cause the computing device to;

determine that the third IR sensor is not receiving reflected IR light from the third IR source.

19. The computing device of claim 10, further comprising:

a camera,

wherein the memory further includes non-transitory instructions that cause the computing device to;

capture, using the camera, at least one ambient light image including at least the portion of the face; and

determine an approximate head position of the face with respect to the computing device based at least in part upon the at least one ambient light image.