Object recognition techniques

• US 10,671,846 B1
• Filed: 02/06/2017
• Issued: 06/02/2020
• Est. Priority Date: 05/15/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamarshi, Vijay, Krishnasamy, Prasanna Venkatesh, Tikare, Amit
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/425,660
• Time in Patent Office: 1,212 Days
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: G01B 11/026 by measuring distance between sensor and object G01B11/0608 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: G06V 2201/07 Target detection
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

Techniques for efficiently identifying objects of interest in an environment and, thereafter, determining the location and/or orientation of those objects. As described below, a system may analyze images captured by a camera to identify objects that may be represented by the images. These objects may be identified in the images based on their size, color, and/or other physical attributes. After identifying these potential objects, the system may define a region around each object for further inspection. Thereafter, portions of a depth map of the environment corresponding to these regions may be analyzed to determine whether any of the objects identified from the images are “objects of interest”—or objects that the system has previously been instructed to track. These objects of interest may include portable projection surfaces, a user'"'"'s hand, or any other physical object. The techniques identify these objects with reference to the respective depth signatures of these objects.

Claims

1. A system comprising:

one or more processors;

a first sensor to obtain image data representing an environment;

a second sensor to obtain depth map data of the environment; and

one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

identifying a potential object that is represented by the image data;

determining a region within the image data that represents the potential object;

based at least in part on determining the region represents the potential object, analyzing a portion of the depth map data corresponding to the region; and

determining that the potential object is an object of interest based, at least in part, on analyzing the portion of the depth map data.

2. The system of claim 1, the acts further comprising determining, based at least in part on analyzing the portion of the depth map data, at least one of:

a pose of the object of interest;

an orientation of the object of interest;

or

a location of the object of interest within the environment.

3. The system of claim 1, wherein the second sensor further obtains additional depth map data of the environment, and wherein the acts further comprising:

determining a first location of the object of interest within the environment based, at least in part, on analyzing the portion of the depth map data;

analyzing at least a portion of the additional depth map data corresponding to the object of interest; and

determining a second location of the object of interest within the environment based, at least in part, on analyzing the portion of the additional depth map data.

4. The system of claim 1, further comprising:

a projector,

wherein the first sensor further obtains additional image data representing the object of interest, and the acts further comprising;

identifying a gesture made by the object of interest using the additional image data; and

causing the projector to project content based, at least in part, on the gesture.

5. The system of claim 1, wherein the region comprises less than an entirety of the image data, and the portion comprises less than an entirety of the depth map data.

20. The system of claim 1, wherein determining the region within the image data that represents the potential object is based, at least in part, on comparing a depth signature associated with the portion object to depths found in the depth map.

6. A method comprising:

receiving image data of an environment;

determining that the image data represents a potential object;

determining a region within the image data corresponding to the potential object;

based at least in part on determining the region corresponds to the potential object, analyzing the portion of the depth map data corresponding to the region; and

determining that the potential object is an object of interest based, at least in part, on analyzing the portion of the depth map data.

7. The method of claim 6, further comprising determining, based at least in part on analyzing the portion of the depth map data, at least one of:

a pose of the object of interest;

an orientation of the object of interest;

or

a location of the object of interest within the environment.

8. The method of claim 6, further comprising:

determining a location of the object of interest within the environment based, at least in part, on analyzing the portion of the depth map data; and

causing a projector to project content at the location.

9. The method of claim 6, wherein the depth map data is first depth map data, and the method further comprises:

determining a first location of the object of interest within the environment based, at least in part, on analyzing the portion of the first depth map data;

analyzing at least a portion of second depth map data corresponding to the object of interest; and

determining a second location of the object of interest within the environment based, at least in part, on analyzing the portion of the second depth map data.

10. The method of claim 6, further comprising storing at least one of:

an indication of the object of interest;

information indicating a pose of the object of interest;

information indicating an orientation of the object of interest;

or

information indicating a location within the environment of the object of interest.

11. The method of claim 6, wherein the image data is first image data and the depth map data is first depth map data, and the method further comprises:

receiving second image data representing the environment;

determining a region within the second image data that represents at least the object of interest;

analyzing at least a portion of second depth map data corresponding to the region within the second image data; and

determining a location of the object of interest within the environment based, at least in part, on analyzing the portion of the second depth map data.

12. The method of claim 6, further comprising:

receiving additional image data representing the object of interest;

identifying a gesture made by the object of interest using the additional image data; and

causing a projector to project content based, at least in part, on the gesture.

13. The method of claim 6, wherein the potential object is a first potential object and object of interest is a first object of interest, and wherein the method further comprises:

determining that the image data represents a second potential object;

determining an additional region within the image data that represents at least the second potential object;

analyzing at least an additional portion of the depth map data corresponding to the additional region; and

determining that the second potential object is a second object of interest based, at least in part, on analyzing the additional portion of the depth map data.

14. A system comprising:

one or more processors; and

one or more computer-readable media storing instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

obtaining image data representing a potential object within an environment;

determining a region of the image data that represents the potential object;

based at least in part on determining the region represents the potential object, identifying a portion of depth map data using the region;

analyzing the portion of the depth map data; and

determining that the potential object is an object of interest based, at least in part, on analyzing the portion of the depth map data.

15. The system of claim 14, the acts further comprising determining, based at least in part on analyzing the portion of the depth map data, at least one of:

a pose of the object of interest;

an orientation of the object of interest;

or

a location of the object of interest within the environment.

16. The system of claim 14, wherein the depth map data is first depth map data, and the acts further comprising:

determining a first location of the object of interest within the environment based, at least in part, on analyzing the portion of the first depth map data;

analyzing a portion of second depth map data corresponding to the object of interest; and

determining a second location of the object of interest within the environment based, at least in part, on analyzing the portion of the second depth map data.

17. The system of claim 14, wherein the image data is first image data, the depth map data is first depth map data, and the potential object is a first potential object, and wherein the acts further comprising:

obtaining second image data representing a second potential object within the environment;

determining a region within the second image data that represents at least the second potential object;

analyzing at least a portion of a second depth map data corresponding to the region within the second image data;

determining that the second potential object does not correspond to the object of interest; and

storing information that indicates that the second potential object does not correspond to the object of interest.

18. The system of claim 14, the acts further comprising:

analyzing the image data to determine at least one of a size, a color, or other physical attribute of the potential object; and

identifying that the image data represents the potential object based, at least in part, on the at least one of the size, the color, or the other physical attribute.

19. The system of claim 14, the acts further comprising:

determining a location of the object of interest within the environment based, at least in part, on analyzing the portion of the depth map data; and

causing a projector to project content at the location.