Object tracking techniques

• US 9,563,955 B1
• Filed: 05/15/2013
• Issued: 02/07/2017
• Est. Priority Date: 05/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamarshi, Vijay, Krishnasamy, Prasanna Venkatesh, Tikare, Amit
• Primary Examiner(s): Seth, Manav
• Application Number:
  US13/895,025
• Time in Patent Office: 1,364 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: G01B 11/026 by measuring distance between sensor and object G01B11/0608 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

Techniques for efficiently identifying objects of interest in an environment and, thereafter, tracking the location and/or orientation of those objects. As described below, a system may analyze images captured by a camera to identify objects that may be represented by the images. These objects may be identified in the images based on their size, color, and/or other physical attributes. After identifying these potential objects, the system may define a region around each object for further inspection. Thereafter, portions of a depth map of the environment corresponding to these regions may be analyzed to determine whether any of the objects identified from the images are “objects of interest”—or objects that the system has previously been instructed to track. These objects of interest may include portable projection surfaces, a user'"'"'s hand, or any other physical object. The techniques identify these objects with reference to the respective depth signatures of these objects.

Claims

1. A system comprising:

one or more processors;

a camera to obtain images of an environment, the environment including an object of interest;

a depth sensor to create depth maps of the environment, a first depth map of the depth maps corresponding to a first image of the images obtained by the camera; and

one or more computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

identifying, from the first image, a potential object for the object of interest;

defining a region within the first image that includes the potential object identified in the first image, the region comprising less than an entirety of the first image; and

analyzing a portion of the first depth map associated with the first image to determine that the potential object is the object of interest, the portion of the first depth map representing the region identified within the first image that includes the potential object.

2. The system of claim 1, the acts further comprising analyzing the portion of the first depth image to determine an orientation of the object of interest.

3. The system of claim 1, the acts further comprising tracking the object of interest within at least a subsequent region of interest of a second image obtained by the camera.

4. The system of claim 1, wherein the identifying of the region within the first image that includes the potential object is computationally less expensive than the analyzing of the portion of the first depth map to determine that the potential object is the object of interest.

5. The system of claim 1, wherein:

the object of interest comprises a hand of a user in the environment; and

the acts further comprise;

tracking the object of interest within at least a subsequent region of interest of a second image obtained by the camera; and

identifying a gesture made by the hand of the user based at least in part on the tracking object of interest.

6. A method comprising:

under control of one or more computing devices configured with executable instructions,

obtaining images of an environment that includes an object of interest;

identifying, from a first image of the images, the object of interest;

defining a region within the first image that includes the object of interest;

analyzing a portion of a first depth map of the environment to confirm that the region of the first image represents the object of interest, the portion of the first depth map corresponding to the region of the first image; and

tracking the object of interest over time at least partly in response to confirming that the region of the first image represents the object of interest.

7. The method of claim 6, further comprising analyzing the first depth map to determine an orientation of the object of interest.

8. The method of claim 6, further comprising analyzing the portion of the first depth map to determine a pose of the object of interest.

9. The method of claim 6, wherein the tracking comprises analyzing a respective portion of a second depth map of a second image to determine a subsequent location of the object of interest over time, the respective portion corresponding to a respective region of the second image that represents the object of interest.

10. The method of claim 6, wherein the tracking comprises analyzing a respective portion of a second depth map of a second image to determine a subsequent location of the object of interest over time and to determine an orientation of the object of interest over time, the respective portion corresponding to a respective region of the second image that represents the object of interest.

11. The method of claim 10, further comprising projecting content toward the subsequent location of the object of interest, wherein the projecting is based at least in part on the orientation of the object of interest.

12. The method of claim 6, wherein the object of interest comprises a hand of a user, and the method further comprising determining a gesture of the hand of the user based at least in part on the object of interest.

13. The method of claim 6, wherein the object of interest comprises a portable projection surface, and the method further comprising projecting content onto the portable projection surface based at least in part on the object of interest.

14. One or more non-transitory computer-readable media maintaining instructions executable by one or more processors to perform operations comprising:

obtaining images of an environment that includes an object of interest;

identifying, from a first image of the images, the object of interest;

defining a region within the first image that includes the object of interest;

analyzing a portion of a first depth map of the environment to confirm that the region of the first image represents the object of interest, the portion of the first depth map corresponding to the region of the first image; and

based at least in part on confirming that the region of the first image represents the object of interest, tracking the object of interest over time.

15. The one or more non-transitory computer-readable media of claim 14, the operations further comprising analyzing the first depth map to determine an orientation of the object of interest.

16. The one or more non-transitory computer-readable media of claim 14, the operations further comprising analyzing the portion of the first depth map to determine a pose of the object of interest.

17. The one or more non-transitory computer-readable media of claim 14, wherein the tracking comprises analyzing a respective portion of a second depth map of a second image to the first image to determine a subsequent location of the object of interest over time, the respective portion corresponding to a respective region of the second image that represents the object of interest.

18. The one or more non-transitory computer-readable media of claim 14, wherein the tracking comprises analyzing a respective portion of a second depth map of a second image to the first image to determine a subsequent location of the object of interest over time and to determine an orientation of the object of interest over time, the respective portion corresponding to a respective region of the second image that represents the object of interest.

19. The one or more non-transitory computer-readable media of claim 14, wherein the object of interest comprises a hand of a user, and the operations further comprising determining a gesture of the hand of the user based at least in part on the object of interest.

20. The one or more non-transitory computer-readable media of claim 14, wherein the object of interest comprises a portable projection surface, and the operations further comprising projecting content onto the portable projection surface based at least in part on the object of interest.