Dynamic object tracking for user interfaces

• US 10,055,013 B2
• Filed: 09/17/2013
• Issued: 08/21/2018
• Est. Priority Date: 09/17/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Ramaswamy, Sharadh, Weber, Jason Robert
• Primary Examiner(s): Merkoulova, Olga
• Application Number:
  US14/029,612
• Publication Number:
  US 20150077323A1
• Time in Patent Office: 1,799 Days
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

Systems and approaches provide for user interfaces that are based on object tracking. For example, the object may be a user'"'"'s head or face. As the user moves his head or face and/or tilts a computing device, the content displayed on the computing device will adapt to the user'"'"'s perspective. The content may include three-dimensional (3D) graphical elements projected onto a two-dimensional (2D) plane and/or the graphical elements can be associated with textural shading, shadowing or reflections that change according to user or device motion to give the user the impression that the user is interacting with the graphical elements in 3D environment. A state of motion of the device can be determined and jitter and/or latency corresponding to the rendering of content can be altered so as to minimize or decrease jitter when the device is stationary and/or to decrease or minimize latency when the device is in motion.

Claims

1. A computing device, comprising:

one or more processors;

a display screen;

one or more cameras;

one or more inertial sensors; and

memory including instructions that, when executed by the one or more processors, cause the computing device to;

estimate a first position of a head with respect to the computing device based at least in part upon one or more first images captured by the one or more cameras;

display a first view of content on the display screen based at least in part upon the first position of the head with respect to the computing device;

obtain motion data, captured by the one or more inertial sensors, corresponding to motion of the computing device;

determine, using the motion data, that the computing device is in motion;

determine, using the motion data, one or more performance parameters for tracking the head in image data, the one or more performance parameters controlling an amount of latency with respect to display of the content, the one or more performance parameters including at least an image sampling rate;

estimate a second position of the head with respect to the computing device based at least in part upon the one or more performance parameters, the first position of the head with respect to the computing device, and one or more second images captured by the one or more cameras; and

display a second view of the content on the display screen based at least in part upon the second position of the head with respect to the computing device.

2. The computing device of claim 1, wherein the one or more performance parameters for tracking the head in image data correspond to:

less jitter when the motion of the computing device is determined to be less than a threshold, and

less latency when the motion of the computing device is determined to exceed the threshold.

3. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

select a tracking process from a plurality of tracking processes to track the head, the plurality of tracking processes including at least one of feature point tracking, template matching, optical flow analysis, mean-shift tracking, Jepson tracking, principal component analysis (PCA), or support vector machines (SVM), the tracking process being selected based at least in part upon the motion data, wherein the instructions when executed to cause the computing device to estimate the second position of the head with respect to the computing device is further based at least in part upon the tracking process.

4. The computing device of claim 1, wherein:

the first view of the content includes one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the head; and

the second view of the content includes the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the second position of the head.

5. A computer-implemented method for tracking an object, comprising:

estimating a first position of an object relative to a computing device based at least in part upon one or more images;

obtaining motion data corresponding to motion of the computing device;

determining, based at least in part upon the motion data, that the computing device is in motion;

determining at least one tracking algorithm for tracking the object in image data, the at least one tracking algorithm controlling an amount of latency with respect to a display of content, wherein the at least one tracking algorithm is selected from among a plurality of tracking algorithms, the plurality of tracking algorithms including at least two of feature point tracking, template matching, optical flow analysis, mean-shift tracking, Jepson tracking, principal component analysis (PCA), or support vector machines (SVM); and

estimating a second position of the object relative to the computing device using the at least one tracking algorithm.

6. The computer-implemented method of claim 5, wherein determining the at least one tracking algorithm includes:

determining one or more performance parameters for tracking the object based at least in part upon the motion data.

7. The computer-implemented method of claim 6, wherein the one or more performance parameters control jitter corresponding to the display of content and control the amount of latency corresponding to the display of content.

8. The computer-implemented method of claim 7, wherein the one or more performance parameters correspond to:

less jitter when the motion data indicates a first state of motion of the computing device, and

less latency when the motion data indicates a second state of the motion of the computing device.

9. The computer-implemented method of claim 5, wherein the object is a head, the method further comprising:

estimating a first position of a head with respect to the computing device based at least in part upon the one or more images captured by one or more cameras; and

estimating a second position of the head with respect to the computing device based at least in part upon a second one or more images captured by the one or more cameras, wherein tracking the object using the at least one tracking algorithm includes tracking the head between the one or more images and the second one or more images.

10. The computer-implemented method of claim 9, further comprising:

causing to be displayed a first view of content rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the head; and

causing to be displayed a second view of the content rendered according to a second three-dimensional perspective that is based at least in part upon the second position of the head.

11. The computer-implemented method of claim 9, further comprising:

computing stereo disparity data for at least one of the first set of one or more images or the second set of one or more images, wherein estimating the second position of the head with respect to the computing device is further based at least in part upon the stereo disparity data.

12. The computer-implemented method of claim 9, wherein estimating the first position of the head includes:

determining a first plurality of points of the head represented in a first image of the one or more images corresponding to a second plurality of points of the head represented in a second image of the one or more images;

calculating a fundamental matrix or an essential matrix using the first plurality of points and the second plurality of points; and

calculating one or more three-dimensional points corresponding to the head by triangulation using a first camera matrix corresponding to a first camera used to capture the first image, a second camera matrix corresponding to a second camera used to capture the second image, the first plurality of points, and the second plurality of points.

13. The computer-implemented method of claim 5, wherein the motion data is obtained from a sensor of the computing device.

19. The computer-implemented method of claim 5, further comprising:

obtaining second motion data corresponding to motion of the computing device;

determining, based at least in part upon the second motion data, that the computing device is in a stationary position;

determining a second tracking algorithm for tracking the object, the second tracking algorithm controlling an amount of jitter with respect to the display of content; and

estimating a third position of the object relative to the computing device using the second tracking algorithm.

14. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors of a computing device, cause the computing device to:

estimate a first position of an object relative to a computing device based at least in part upon one or more images;

obtain motion data corresponding to motion of the computing device;

determine, based at least in part upon the motion data, that the computing device is in motion;

determining at least one tracking algorithm for tracking the object in image data, the at least one tracking algorithm controlling an amount of latency with respect to a display of content, wherein the determining further comprises determining one or more performance parameters for tracking the object based at least in part upon the motion data, the one or more performance parameters including at least an image sampling rate; and

estimate a second position of the object relative to the computing device using the at least one tracking algorithm.

15. The non-transitory computer-readable storage medium of claim 14, wherein the instructions when executed to cause the computing device to determine the at least one tracking algorithm include causing the computing device to:

select, based at least in part upon the motion data, the at least one tracking algorithm from among a plurality of tracking algorithms, the plurality of tracking algorithms including at least one of feature point tracking, template matching, optical flow analysis, mean-shift tracking, Jepson tracking, principal component analysis (PCA), or support vector machines (SVM).

16. The non-transitory computer-readable storage medium of claim 14, wherein the instructions when executed to cause the computing device to determine the at least one tracking algorithm include causing the computing device to:

determine one or more performance parameters for tracking the object, the one or more performance parameters controlling jitter corresponding to the display of content and controlling the amount of latency corresponding to the display of content.

17. The non-transitory computer-readable storage medium of claim 14, wherein the instructions when executed to cause the computing to estimate the first position of the object include causing the computing device to:

determine a first plurality of points of the object represented in a first image of the one or more images corresponding to a second plurality of points of the object represented in a second image of the one or more images;

calculate a fundamental matrix or an essential matrix using the first plurality of points and the second plurality of points; and

calculate one or more three-dimensional points corresponding to the object by triangulation using a first camera matrix corresponding to a first camera used to capture the first image, a second camera matrix corresponding to a second camera used to capture the second image, the first plurality of points, and the second plurality of points.

18. The non-transitory computer-readable storage medium of claim 17, wherein the object is a head, and the instructions when executed further cause the computing device to:

cause to be displayed a first view of content rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the head; and

cause to be displayed a second view of the content rendered according to a second three-dimensional perspective that is based at least in part upon the second position of the head.

20. The non-transitory computer-readable storage medium of claim 14, wherein the instructions when executed further cause the computing device to:

obtain second motion data corresponding to motion of the computing device;

determine, based at least in part upon the second motion data, that the computing device is in a stationary position;

determine a second tracking algorithm for tracking the object, the second tracking algorithm controlling an amount of jitter with respect to the display of content; and

estimate a third position of the object relative to the computing device using the second tracking algorithm.

21. A computer-implemented method for tracking an object, comprising:

estimating a first position of an object relative to a computing device based at least in part upon one or more images;

obtaining motion data corresponding to motion of the computing device;

determining, based at least in part upon the motion data, that the computing device is in motion;

dynamically adjusting, based on the motion data, one or more performance parameters of at least one tracking algorithm for tracking the object in image data, the one or more parameters controlling an amount of latency with respect to a display of content; and

estimating a second position of the object relative to the computing device using the at least one tracking algorithm.