Multi-tracker object tracking

• US 9,269,012 B2
• Filed: 08/22/2013
• Issued: 02/23/2016
• Est. Priority Date: 08/22/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fotland, David Allen
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Deaderick, Elliott
• Application Number:
  US13/973,913
• Publication Number:
  US 20150055821A1
• Time in Patent Office: 915 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Systems and approaches are provided for tracking an object using multiple tracking processes. By combining multiple lightweight tracking processes, object tracking can be robust, use a limited amount of power, and enable a computing device to respond to input corresponding to the motion of the object in real time. The multiple tracking processes can be run in parallel to determine the position of the object by selecting the results of the best performing tracker under certain heuristics or combining the results of multiple tracking processes in various ways. Further, other sensor data of a computing device can be used to improve the results provided by one or more of the tracking processes.

Claims

1. A computing device, comprising:

one or more processors;

one or more cameras; and

memory including instructions that, upon being executed by the one or more processors, cause the computing device to;

capture, at a first time, one or more first images using the one or more cameras;

detect at least one portion of a representation of a face of a user in the one or more first images using a facial detection process;

capture, at a second time, one or more second images using the one or more cameras;

track the at least one portion of the representation of the face of the user between the one or more first images and the one or more second images by concurrently executing a plurality of types of object tracking processes, the plurality of types of object tracking processes including two or more of a feature-based tracking process, an elliptical tracking process, a template matching tracking process, an optical flow tracking process, or a silhouette or contour tracking process;

select an object tracking process from among the plurality of object tracking processes based at least in part upon the object tracking process best satisfying a condition; and

determine a position of the face of the user based at least in part upon where the selected object tracking process tracked the at least one portion of the representation of the face of the user in the one or more second images.

2. The computing device of claim 1, wherein the condition corresponds to at least one of a highest level of confidence, a lowest error rate, an average corresponding to where the plurality of types of object tracking processes tracked the at least one portion of the representation of the face of the user in the one or more second images, and a consensus corresponding to where a majority of the plurality of types of object tracking processes tracked the at least one portion of the representation of the face of the user in the one or more second images.

3. The computing device of claim 1, further comprising:

a display screen operable to display content in two dimensions,

wherein the instructions upon being executed further cause the computing device to;

determine a first position of the face of the user based at least in part upon where the at least one portion of the representation of the face of the user is detected in the one or more first images;

display a first view of the content on the display screen, the first view of the content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the face of the user; and

display a second view of the content on the display screen, the second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the face of the user.

4. A computer-implemented method for tracking an object, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining data corresponding to where at least one portion of a representation of the object is located in one or more first images;

obtaining one or more second images, at least one of the second images including the at least one portion of the representation of the object;

tracking the at least one portion of the representation of the object between the one or more first images and the one or more second images by concurrently executing a plurality of types of object tracking processes, the plurality of types of object tracking processes including two or more of a feature-based tracking process, an elliptical tracking process, a template matching tracking process, an optical flow tracking process, or a silhouette or contour tracking process;

selecting an object tracking process from among the plurality of object tracking processes based at least in part upon the object tracking process best satisfying a condition; and

determining a position of the object based at least in part upon where the selected object tracking process tracked the at least one portion of the representation of the object in the one or more second images.

5. The computer-implemented method of claim 4, wherein obtaining the data corresponding to where the at least one portion of the representation of the object is located in the one or more first images includes:

obtaining the one or more first images including the at least one portion of the representation of the object;

detecting the at least one portion of the representation of the object in the one or more first images; and

providing the data corresponding to where the at least one portion of the representation of the object is located in the one or more first images based upon where the at least one portion of the representation of the object is detected in the one or more first images.

6. The computer-implemented method of claim 4, wherein determining the position of the object includes:

determining a first plurality of points of the at least one portion of the representation the object in the one or more first images respectively corresponding to a second plurality of points of the at least one portion of the representation of the object in the one or more second images;

calculating a fundamental matrix or an essential matrix using the first plurality of points and the second plurality of points; and

determining one or more three-dimensional points corresponding to the object by triangulation using a first camera matrix used to capture the one or more first images, a second camera matrix used to capture the one or more second images, the first plurality of points, and the second plurality of points.

7. The computer-implemented method of claim 4, wherein the condition corresponds to at least one of a highest level of confidence, a lowest error rate, an average corresponding to where the plurality of types of object tracking processes tracked the at least one portion of the representation of the object in the one or more second images, and a majority corresponding to where the plurality of types of object tracking processes tracked the at least one portion of the representation of the object in the one or more second images.

8. The computer-implemented method of claim 4, further comprising:

determining at least one of a motion or an orientation corresponding to one or more cameras used to capture the one or more second images; and

causing at least one of the plurality of types of object tracking processes to focus on at least one portion of the one or more second images to track the at least one portion of the representation of the object, the at least one portion of the one or more second images corresponding to the at least one of the motion or the orientation.

9. The computer-implemented method of claim 4, further comprising:

determining an amount of ambient light of an environment in which one or more cameras used to capture the one or more second images are situated; and

weighting second data corresponding to at least one of the plurality of types of object tracking processes based at least in part upon the amount of ambient light.

10. The computer-implemented method of claim 4, further comprising:

weighting respective data corresponding to each of the plurality of types of object tracking processes,

wherein the position of the object is further based at least in part upon each weight of the respective data corresponding to each of the plurality of types of object tracking processes.

11. The computer-implemented method of claim 10, wherein the weight of the respective data corresponding to each of the at least one subset of the plurality of types of object tracking processes is based at least in part upon at least one of a respective level of confidence for each of the at least one subset of the plurality of types of object tracking processes, a respective error rate for each of the at least one subset of the plurality of types of object tracking processes, a respective amount of memory used by each of the at least one subset of the plurality of types of object tracking processes, a respective amount of processing used by each of the at least one subset of the plurality of types of object tracking processes, or a respective amount of power used by each of the at least one subset of the plurality of types of object tracking processes.

12. The computer-implemented method of claim 4, further comprising:

determining a first position of the object based at least in part upon the data corresponding to where the at least one portion of the representation of the object is located in the one or more first images;

displaying a first view of content, the first view of the content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the object; and

displaying a second view of the content, the second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the object.

13. The computer-implemented method of claim 4, further comprising:

applying at least one of deblurring, denoising, histogram equalization, brightness optimization, contrast optimization, color-cast correction, or reduction or removal of specular reflection to at least one of the second images.

14. A non-transitory computer-readable storage medium storing instructions for tracking an object, the instructions upon being executed by a processor causing the processor to:

obtain one or more first images, at least one of the first images including at least one portion of a representation of the object;

detect the at least one portion of the representation of the object in the one or more first images;

obtain one or more second images, at least one of the second images including the at least one portion of the representation of the object;

track the at least one portion of the representation of the object between the one or more first images and the one or more second images by concurrently executing a plurality of types of object tracking processes, the plurality of types of object tracking processes including two or more of a feature-based tracking process, an elliptical tracking process, a template matching tracking process, an optical flow tracking process, or a silhouette or contour tracking process;

select an object tracking process from among the plurality of object tracking processes based at least in part upon the object tracking process best satisfying a condition; and

determine a position of the object based at least in part upon where the selected object tracking process tracked the at least one portion of the representation of the object in the one or more second images.

15. The non-transitory computer-readable storage medium of claim 14, wherein the instructions upon being executed further cause the processor to:

determine at least one of a motion or an orientation corresponding to one or more cameras used to capture the one or more second images; and

cause at least one of the plurality of types of object tracking processes to focus on at least one portion of the one or more second images to track the at least one portion of the representation of the object, the at least one portion of the one or more second images corresponding to the at least one of the motion or the orientation.

16. The non-transitory computer-readable storage medium of claim 14, wherein the instructions upon being executed further cause the processor to:

determine an amount of ambient light of an environment in which one or more cameras used to capture the one or more second images are situated; and

weight data corresponding to at least one of the plurality of types of object tracking processes based at least in part upon the amount of ambient light.

17. The non-transitory computer-readable storage medium of claim 14, wherein the condition corresponds to at least one of a highest level of confidence, a lowest error rate, a lowest amount of memory used, a lowest amount of processing used, and a lowest amount of power used.

18. The non-transitory computer-readable storage medium of claim 14, wherein the instructions upon being executed further cause the processor to:

determine a first position of the object based at least in part upon where the at least one portion of the representation of the object is detected in the one or more first images;

display a first view of content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the object; and

display a second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the object.