Tracking head position for rendering content

• US 9,274,597 B1
• Filed: 12/20/2011
• Issued: 03/01/2016
• Est. Priority Date: 12/20/2011
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Karakotsios, Kenneth M., Ivanchenko, Volodymyr V., Noble, Isaac S.
• Primary Examiner(s): Chang, Kent
• Assistant Examiner(s): Shah, Sujit
• Application Number:
  US13/332,172
• Time in Patent Office: 1,533 Days
• Field of Search: 345/158
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

Content displayed on a display element of a computing device can be rendered from a viewpoint that updates with changes in the relative position of a user'"'"'s head, or other such feature, with respect to the device. At least one camera of the device can attempt to track the relative position in order to determine the proper viewpoint. When the user'"'"'s head can no longer be seen, or when confidence in the determined relative position drops below an acceptable value, for example, the device can utilize information from another source, such as a motion or rotation sensor, to attempt to estimate the current position of the user'"'"'s head. Physical limitations of the user with respect to the device also can be used to improve the accuracy of the determination.

Claims

1. A computer-implemented method of rendering content on a display screen of a computing device, comprising:

under control of a computing device configured with executable instructions,

capturing video information using a camera of the computing device;

detecting at least one first feature in the video information indicative of a head of a user;

determining a first relative position of the at least one first feature with respect to the computing device;

rendering the content on the display screen of the computing device to correspond to a viewpoint of the user at the first relative position;

at a first time, predicting that the at least one first feature will be outside a field of view of the camera at a second time based at least in part upon analyzing the video information;

storing to a buffer first motion information obtained from at least one sensor of the computing device from the first time to the second time based at least in part upon predicting that the at least one first feature will be outside the field of view of the camera at the second time;

at the second time, determining that the at least one first feature is outside the field of view of the camera based at least in part upon analyzing the video information;

estimating a second relative position of the at least one first feature with respect to the computing device based at least in part upon where the at least one first feature was last tracked in the video information with at least a minimum level of confidence and a motion of the computing device determined based at least in part upon the first motion information pulled from the buffer and second motion information obtained from the at least one sensor after the second time;

determining an updated viewpoint of the user based at least in part upon the second relative position; and

rendering the content on the display screen to correspond to the updated viewpoint.

2. The computer-implemented method of claim 1, wherein the motion includes at least one of a translation or a rotation of the computing device.

3. The computer-implemented method of claim 1, wherein the motion is a weighted combination that is further based at least in part upon an environmental condition, the environmental condition including at least one of an amount of lighting or a presence of a point light source.

4. The computer-implemented method of claim 1, wherein the at least one sensor includes at least one of an electronic gyroscope, an electronic compass, an inertial sensor, an accelerometer, a shadow sensor, motion detector, a microphone array, or a distance sensor.

5. A computer-implemented method of displaying content, comprising:

under control of one or more computing systems configured with executable instructions,

capturing image information using a camera of a computing device;

analyzing the image information to determine a first relative position of a first feature of a user with respect to the computing device;

at a first time, predicting that the first feature of the user will be outside a field of view of the camera at a second time;

buffering first motion information obtained from at least one sensor of the computing device from the first time to the second time;

at the second time, estimating a second relative position of the first feature with respect to the computing device based at least in part upon where the first feature was last tracked in the image information with at least a minimum level of confidence and a motion of the computing device determined based at least in part upon the first motion information and second motion information obtained from the at least one sensor after the second time; and

rendering content on a display element of the computing device to correspond to a viewpoint of the user, the viewpoint being determined based at least in part upon the second relative position of the first feature of the user.

6. The computer-implemented method of claim 5, wherein the motion is a weighted combination that is further based at least in part upon an environmental condition.

7. The computer-implemented method of claim 6, wherein the environmental condition includes at least one of an amount of lighting or a presence of a point light source.

8. The computer-implemented method of claim 5, wherein the at least one sensor of the computing device includes at least one of an electronic gyroscope, an electronic compass, an inertial sensor, an accelerometer, a shadow sensor, a motion detector, a microphone array, or a distance sensor.

9. The computer-implemented method of claim 5, wherein the motion of the computing device is determined by analyzing second image information captured by the at least one sensor to determine relative movement of at least one background object or light source with respect to the computing device.

10. The computer-implemented method of claim 5, further comprising:

analyzing the image information with a feature detection algorithm to locate the first feature in the image information; and

determining the first relative position of the first feature of the user based at least in part upon a position of the first feature in the image information and the field of view of the camera.

11. The computer-implemented method of claim 5, further comprising:

illuminating the first feature with at least one of visible light or infrared radiation while capturing the image information.

12. The computer-implemented method of claim 5, wherein the at least one sensor further provides contact information indicating whether the user is currently contacting the computing device, the contact information being used to further estimate the second relative position of the first feature of the user.

13. A computing device, comprising:

a processor;

a display screen;

a camera;

a motion sensor; and

a memory device including instructions that, upon being executed by the processor, cause the computing device to;

capture image information using the camera;

analyze the image information to determine a first viewpoint of a user with respect to the computing device;

at a first time, predict that a head of the user will be outside a field of view of the camera at a second time;

buffer first motion information obtained from the motion sensor from the first time to the second time;

at the second time, estimate a second viewpoint of the user with respect to the computing device based at least in part upon where the head of the user was last tracked in the image information with at least a minimum level of confidence and a motion of the computing device determined based at least in part upon the first motion information and second motion information obtained from the motion sensor after the second time; and

render content on the display screen to correspond to the second viewpoint of the user.

14. The computing device of claim 13, wherein the motion sensor includes at least one of an electronic gyroscope, an electronic compass, an inertial sensor, an accelerometer, a shadow sensor, a motion detector, a microphone array, or a distance sensor.

15. The computing device of claim 13, wherein the instructions, upon being executed to cause the computing device to estimate the second viewpoint of the user, include causing the computing device to:

estimate a relative position of the head of the user with respect to the computing device.

16. A non-transitory computer-readable storage medium storing instructions for rendering content for a user viewpoint, the instructions when executed by a processor of a computing system causing the computing system to:

capture video information using a camera of a computing device;

analyze the video information to determine a first relative viewpoint of a user with respect to the computing device;

at a first time, predict that a head of a user will be outside a field of view of the camera at a second time;

buffer first motion information obtained from at least one motion sensor of the computing device from the first time to the second time;

at the second time, estimate a second relative viewpoint of the user with respect to the computing device based at least in part upon where the head of the user was last tracked in the video information with at least a minimum level of confidence and a motion of the computing device determined based at least in part upon the first motion information and second motion information obtained from the at least one motion sensor after the second time; and

render content on a display element of the computing device to correspond to the second relative viewpoint of the user.

17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions, upon being executed, further cause the computing system to:

analyze the video information with a feature detection algorithm to locate the head of the user in the video information; and

determine the first relative viewpoint of the user based at least in part upon a position of the head of the user in the video information and the field of view of the camera.

18. The non-transitory computer-readable storage medium of claim 16, wherein the instructions, upon being executed, further cause the computing system to:

illuminate the head of the user with at least one of visible light or infrared radiation while capturing the video information.

19. The non-transitory computer-readable storage medium of claim 16, wherein the instructions, upon being executed, further cause the computing system to:

obtain contact information indicating whether the user is currently contacting the computing system, the contact information being used to further estimate the second relative viewpoint of the user.