MULTI-TRACKER OBJECT TRACKING

• US 20150055821A1
• Filed: 08/22/2013
• Published: 02/26/2015
• Est. Priority Date: 08/22/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Fotland, David Allen
• Granted Patent: US 9,269,012 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Systems and approaches are provided for tracking an object using multiple tracking processes. By combining multiple lightweight tracking processes, object tracking can be robust, use a limited amount of power, and enable a computing device to respond to input corresponding to the motion of the object in real time. The multiple tracking processes can be run in parallel to determine the position of the object by selecting the results of the best performing tracker under certain heuristics or combining the results of multiple tracking processes in various ways. Further, other sensor data of a computing device can be used to improve the results provided by one or more of the tracking processes.

Claims

1. A computing device, comprising:

one or more processors;

one or more cameras; and

memory including instructions that, when executed by the one or more processors, cause the computing device to;

capture, at a first time, one or more first images using the one or more cameras;

detect at least a portion of a representation of a face of a user in the one or more first images using a facial detection process;

capture, at a second time, one or more second images using the one or more cameras;

track the at least the portion of the representation of the face of the user between the one or more first images and the one or more second images by concurrently executing a plurality of facial tracking processes; and

determine a position of the face of the user based at least in part upon where at least one of the plurality of facial tracking processes tracked the at least the portion of the representation of the face of the user in the one or more second images.

2. The computing device of claim 1, wherein the instructions when executed to cause the computing device to track the at least the portion of the representation of the face of the user include causing the computing device to perform at least two or more of:

detecting the at least the portion of the representation of the face of the user in the one or more second images using a facial detection process based on at least one of feature detection, background subtraction, segmentation, or supervised learning;

determining one or more first points of the at least the portion of the representation the face of the user in the one or more first images respectively corresponding to one or more points of the at least the portion of the representation of the face of the user in the one or more second images;

searching for a kernel in the one or more second images, the kernel corresponding to the at least the portion of the representation of the face of the user in the one or more first images;

or

searching for a silhouette or contours in the one or more second images, the silhouette or the contours corresponding to the at least the portion of the representation the face of the user in the one or more first images.

3. The computing device of claim 1, wherein the instructions when executed to cause the computing device to determine the position of the face of the user include causing the computing device to analyze data of each of the plurality of facial tracking processes to determine at least one of:

a maximum of a first condition, a minimum of a second condition, an average corresponding to where the plurality of facial tracking processes tracked the at least the portion of the representation of the face of the user in the one or more second images, or a consensus corresponding to where a majority the plurality of facial tracking processes tracked the at least the portion of the representation of the face of the user in the one or more second images.

4. The computing device of claim 1, further comprising:

a display screen operable to display content in two dimensions,

wherein the instructions when executed further cause the computing device to;

determine a first position of the face of the user based at least in part upon where the at least the portion of the representation of the face of the user is detected in the one or more first images;

display a first view of the content on the display screen, the first view of the content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the face of the user; and

display a second view of the content on the display screen, the second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the face of the user.

5. A computer-implemented method for tracking an object, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining data corresponding to where at least a portion of a representation of the object is located in one or more first images;

obtaining one or more second images, at least one of the second images including the at least the portion of the representation of the object;

tracking the at least the portion of the representation of the object between the one or more first images and the one or more second images by concurrently executing a plurality of object tracking processes; and

determining a position of the object based at least in part upon where at least one of the plurality of tracking processes tracked the at least the portion of the representation of the object in the one or more second images.

6. The computer-implemented method of claim 5, wherein obtaining the data corresponding to where the at least the portion of the representation of the object is located in the one or more first images includes:

obtaining the one or more first images, at least one of the first images including the at least the portion of the representation of the object;

detecting the at least the portion of the representation of the object in the one or more first images; and

providing the data corresponding to where the at least the portion of the representation of the object is located in the one or more first images based upon where the at least the first portion of the representation of the object is detected in the one or more first images.

7. The computer-implemented method of claim 5, wherein tracking the at least the portion of the representation of the object includes at least two or more of:

detecting the at least the portion of the representation of the object in the one or more second images using an object detection process;

determining one or more first points of the at least the portion of the representation the object in the one or more first images respectively corresponding to one or more points of the at least the portion of the representation of the object in the one or more second images;

searching for a kernel in the one or more second images, the kernel corresponding to the at least the portion of the representation of the object in the one or more first images;

or

searching for a silhouette or contours in the one or more second images, the silhouette or the contours corresponding to the at least the portion of the representation the object in the one or more first images.

8. The computer-implemented method of claim 5, wherein determining the position of the object includes:

determining a first plurality of points of the at least the portion of the representation the object in the one or more first images respectively corresponding to a second plurality of points of the at least the portion of the representation of the object in the one or more second images;

calculating a fundamental matrix or an essential matrix using the first plurality of points and the second plurality of points; and

determining one or more three-dimensional points corresponding to the object by triangulation using a first camera matrix used to capture the one or more second images, a second camera matrix used to capture the one or more second images, the first plurality of points, and the second plurality of points.

9. The computer-implemented method of claim 5, wherein determining the position of the object includes analyzing second data of each of the plurality of object tracking processes to determine at least one of:

a maximum of a first condition, a minimum of a second condition, an average corresponding to where the plurality of object tracking processes tracked the at least the portion of the representation of the object in the one or more second images, or a majority corresponding to where the plurality of object tracking processes tracked the at least the portion of the representation of the object in the one or more second images.

10. The computer-implemented method of claim 5, further comprising:

determining at least one of a motion or an orientation corresponding to one or more cameras used to capture the one or more second images; and

causing at least one of the plurality of object tracking processes to focus on at least one portion of the one or more second images to track the at least the portion of the representation of the object, the at least one portion of the one or more second images corresponding to the at least one of the motion or the orientation.

11. The computer-implemented method of claim 5, further comprising:

determining an amount of ambient light of an environment in which one or more cameras used to capture the one or more second images are situated; and

weight second data corresponding to at least one of the plurality of object tracking processes based at least in part upon the amount of ambient light.

12. The computer-implemented method of claim 5, further comprising:

weighting respective data corresponding to each of at least a subset of the plurality of object tracking processes,

wherein determining the position of the object is further based at least in part upon each weight of the respective data corresponding to each of the at least the subset of the plurality of object tracking processes.

13. The computer-implemented method of claim 12, wherein the weight of the respective data corresponding to each of the at least the subset of the plurality of object tracking processes is based at least in part upon at least one of a respective level of confidence for each of the at least the subset of the plurality of object tracking processes, a respective error rate for each of the at least the subset of the plurality of object tracking processes, a respective amount of memory used by each of the at least the subset of the plurality of object tracking processes, a respective amount of processing used by each of the at least the subset of the plurality of object tracking processes, or a respective amount of power used by each of the at least the subset of the plurality of object tracking processes.

14. The computer-implemented method of claim 5, further comprising:

determining a first position of the object based at least in part upon the data corresponding to where the at least the portion of the representation of the object is located in the one or more first images;

causing to be displayed a first view of content, the first view of the content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the object; and

causing to be displayed a second view of the content, the second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the object.

15. The computer-implemented method of claim 5, further comprising:

applying at least one of deblurring, denoising, histogram equalization, brightness optimization, contrast optimization, color-cast correction, or reduction or removal of specular reflection to at least one of the second images.

16. A non-transitory computer-readable storage medium storing instructions for tracking an object, the instructions when executed by a processor causing the processor to:

obtain one or more first images, at least one of the first images including at least a portion of a representation of the object;

detect the at least the portion of the representation of the object in the one or more first images;

obtain one or more second images, at least one of the second images including at least the portion of the representation of the object;

track the at least the portion of the representation of the object between the one or more first images and the one or more second images by concurrently executing a plurality of object tracking processes; and

determine a position of the object based at least in part upon where at least one of the plurality of tracking processes tracked the at least the portion of the representation of the object in the one or more second images.

17. The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed further cause the processor to:

determine at least one of a motion or an orientation corresponding to one or more cameras used to capture the one or more second images; and

cause at least one of the plurality of object tracking processes to focus on at least one portion of the one or more second images to track the at least the portion of the representation of the object, the at least one portion of the one or more second images corresponding to the at least one of the motion or the orientation.

18. The non-transitory computer-readable storage medium of claim 16, wherein the instructions when executed further cause the processor to:

determine an amount of ambient light of an environment in which one or more cameras used to capture the one or more second images are situated; and

weight data corresponding to at least one of the plurality of object tracking processes based at least in part upon the amount of ambient light.

19. The non-transitory computer-readable storage medium 16, wherein the instructions when executed further cause the processor to:

weighting respective data corresponding to each of at least a subset of the plurality of object tracking processes, a weight of the respective data corresponding to each of the at least the subset of the plurality of object tracking processes being based at least in part upon at least one of a respective level of confidence for each of the at least the subset of the plurality of object tracking processes, a respective error rate for each of the at least the subset of the plurality of object tracking processes, a respective amount of memory used by each of the at least the subset of the plurality of object tracking processes, a respective amount of processing used by each of the at least the subset of the plurality of object tracking processes, or a respective amount of power used by each of the at least the subset of the plurality of object tracking processes,

wherein the position of the object is further determined based at least in part upon the weight of the respective data corresponding to each of the at least the subset of the plurality of object tracking processes.

20. The non-transitory computer-readable storage medium 16, wherein the instructions when executed further cause the processor to:

determine a first position of the object based at least in part upon where the at least the portion of the representation of the object is detected in the one or more first images;

cause to be displayed a first view of content, the first view of the content including one or more graphical elements rendered according to a first three-dimensional perspective that is based at least in part upon the first position of the object; and

cause to be displayed a second view of the content, the second view of the content including the one or more graphical elements rendered according to a second three-dimensional perspective that is based at least in part upon the position of the object.