Robust user detection and tracking

• US 9,832,452 B1
• Filed: 08/12/2013
• Issued: 11/28/2017
• Est. Priority Date: 08/12/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fotland, David Allen, Stafford, David Wayne, Chaboud, Matthew Liang, Touloumtzis, Michael Paul
• Primary Examiner(s): Teitelbaum, Michael
• Application Number:
  US13/965,126
• Time in Patent Office: 1,569 Days
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

Systems and approaches are provided for robustly detecting and tracking a user. Image data can be captured and processed to provide an estimated position and/or orientation of the user. Other sensor data, such as from an accelerometer and/or gyroscope, can be determined for a more robust estimation of the user'"'"'s position and/or orientation. Multiple user detection processes and/or motion estimation approaches and their corresponding confidence levels can also be combined to determine a final estimated position and orientation of the user. The multiple user pose estimations and/or motion estimations can be combined via an approach such as probabilistic system modeling and maximum likelihood estimation.

Claims

1. A computer-implemented method, comprising:

determining a portion of image data to initiate detection of one or more representations of an object in the image data based at least in part upon first motion data from at least one inertial sensor;

determining, at a first rate, first position data associated with the object relative to at least one display screen by initiating detection of one or more representations of the object in the portion of the image data;

determining, at a second rate greater than the first rate, second position data associated with the object relative to a latest determined position of the first position data based at least in part upon second motion data from the at least one inertial sensor; and

adjusting content displayed on the at least one display screen in response to determining that a change between a latest determined position of the second position data and a previously determined position of the second position data satisfies a threshold.

2. The computer-implemented method of claim 1, further comprising:

determining one or more regions of the image data that exclude a representation of the object based at least in part upon the first motion data; and

bypassing the one or more regions for determining the latest determined position of the first position data.

7. The computer-implemented method of claim 1, further comprising obtaining three-dimensional positional data using at least one of structured lighting, a proximity sensor, a distance sensor, a depth sensor, a range finder, an ultrasonic transceiver, or stereo disparity between a first image and a second image.

8. The computer-implemented method of claim 1, further comprising:

determining that the object is not within a field of view of a camera; and

suspending a tracking process for the object using the image data.

9. The computer-implemented method of claim 8, further comprising:

determining that the object is within the field of view of the camera; and

resuming the tracking process for the object using the image data.

10. The computer-implemented method of claim 1, further comprising:

determining an amount of ambient light from an ambient light sensor; and

selecting, based at least in part on the amount of ambient light, a tracking process for the object.

11. The computer-implemented method of claim 1, further comprising applying a Kalman filter using the first position data and the second position data.

3. A non-transitory computer-readable storage medium storing instructions, the instructions, upon being executed by a processor, causing a computing system to:

determine a first portion of image data that excludes a representation of an object based at least in part upon first motion data from at least one inertial sensor;

determine, at a first rate, first position data associated with the object relative to at least one display screen by analyzing a second portion of the image data that excludes the first portion of the image data;

determine, at a second rate, second position data associated with the object relative to a latest determined position of the first position data based at least in part upon second motion data from the at least one inertial sensor; and

adjust content displayed on the at least one display screen in response to determining that a change between a latest determined position of the second position data and a previously determined position of the second position data satisfies a threshold.

4. The non-transitory computer-readable storage medium of claim 3, wherein the instructions when executed further cause the computing system to:

determine a third portion of the image data that includes the representation of the object based at least in part upon the first motion data; and

initiate detection of the representation of the object in the third portion of the image data.

12. The non-transitory computer-readable storage medium of claim 3, further comprising obtain three-dimensional positional data using at least one of structured lighting, a proximity sensor, a distance sensor, a depth sensor, a range finder, an ultrasonic transceiver, or stereo disparity between a first image and a second image.

13. The non-transitory computer-readable storage medium of claim 3, further comprising:

determine that the object is not within a field of view of a camera; and

suspend a tracking process for the object using the image data.

14. The non-transitory computer-readable storage medium of claim 13, further comprising:

determine that the object is within the field of view of the camera; and

resume the tracking process for the object using the image data.

15. The non-transitory computer-readable storage medium of claim 3, further comprising:

determine an amount of ambient light from an ambient light sensor; and

select, based at least in part on the amount of ambient light, a tracking process for the object.

16. The non-transitory computer-readable storage medium of claim 3, further comprising apply a Kalman filter using the first position data and the second position data.

5. A computing system, comprising:

at least one processor;

at least one camera;

at least one sensor;

at least one display screen; and

memory including instructions that, upon being executed by the at least one processor, cause the computing system to;

determine a portion of image data, captured by the at least one camera, to initiate detection of one or more representations of an object in the image data based at least in part upon first motion data from the at least one sensor;

determine, at a first rate, first position data associated with the object relative to the at least one display screen by initiating detection of one or more representations of the object in the portion of the image data;

determine, at a second rate, second position data associated with the object relative to a latest determined position of the first position data based at least in part upon second sensor data from the at least one sensor; and

adjust content displayed on the at least one display screen in response to determining that a change between a latest determined position of the second position data and a previously determined position of the second position data satisfies a threshold.

6. The computing system of claim 5, wherein the instructions when executed further cause the computing system to:

determine one or more regions of the image data that exclude a representation of the object based at least in part upon the first motion data; and

bypass the one or more regions for determining the latest determined position of the first position data.

17. The computing system of claim 5, wherein the instructions when executed further cause the computing system to:

obtain three-dimensional positional data using at least one of structured lighting, a proximity sensor, a distance sensor, a depth sensor, a range finder, an ultrasonic transceiver, or stereo disparity between a first image and a second image.

18. The computing system of claim 5, wherein the instructions when executed further cause the computing system to:

determine that the object is not within a field of view of the camera; and

suspend a tracking process for the object using the image data.

19. The computing system of claim 18, wherein the instructions when executed further cause the computing system to:

determine that the object is within the field of view of the camera; and

resume the tracking process for the object using the image data.

20. The computing system of claim 5, further comprising an ambient light sensor, wherein the instructions when executed further cause the computing system to:

determine an amount of ambient light from the ambient light sensor; and

select, based at least in part on the amount of ambient light, a tracking process for the object.

21. The computing system of claim 5, wherein the instructions when executed further cause the computing system to:

apply a Kalman filter using the first position data and the second position data.