Object detection and identification

• US 10,262,230 B1
• Filed: 02/26/2016
• Issued: 04/16/2019
• Est. Priority Date: 08/24/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yan, Renwei, Chang, Samuel Henry, Crump, Edward Dietz
• Primary Examiner(s): Seth, Manav
• Application Number:
  US15/055,310
• Time in Patent Office: 1,145 Days
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A two-dimensional (2D) image and a three-dimensional (3D) of an environment may be captured. Upon identifying a location and/or contour of an object from the 3D image, the object from the 3D image may be mapped onto the 2D image. The object, including its location and contour, may be identified from the 2D image. Based at least partly on a comparison between the object from the 3D image and the object from the 2D image, a disparity may be calculated. The location and contour of the object may be determined when it is determined that the disparity is less than or equal to a predetermined threshold. Otherwise, the object from the 3D image may be remapped onto the 2D image.

Claims

1. A system comprising:

one or more sensors to capture a first image of an environment and a second image of the environment, the first image depicting a depth of one or more objects within the environment;

memory;

one or more processors; and

one or more computer-executable instructions maintained in the memory and executable by the one or more processors to;

determine, from the first image, a set of points that corresponds to an object depicted in the first image;

determine, based at least partly on the set of points, a portion of the second image that depicts the object, the portion being a subset of an entirety of the second image and including a boundary that surrounds the object;

determine pixel data associated with the portion of the second image; and

determine, based at least partly on the pixel data, one or more characteristics associated with the object.

2. The system as recited in claim 1, further comprising an object database that stores one or more second characteristics associated with a second object, and wherein the one or more computer-executable instructions are further executable by the one or more processors to identify the object based at least partly on the one or more characteristics and the one or more second characteristics.

3. The system as recited in claim 1, wherein determining the one or more characteristics associated with the object comprises determining at least one of a boundary of the object, a size of the object, a shape of the object, a location of the object, or a color of at least a second portion of the object.

4. The system as recited in claim 1, wherein the first image is a three-dimensional (3D) image and the second image is a two-dimensional (2D) image, and wherein the one or more sensors include a 3D camera to capture the 3D image and a 2D camera to capture the 2D image.

5. The system as recited in claim 1, wherein the set of points represents a first contour of the object from the first image and the one or more characteristics represent a second contour of the object from the second image, the second contour corresponding to the first contour.

6. The system as recited in claim 1, wherein the one or more computer-executable instructions are further executable by the one or more processors to:

identify a second set of points within the portion of the second image; and

determine that the second set of points corresponds to the set of points.

7. The system as recited in claim 1, further comprising a projector, and wherein the one or more computer-executable instructions are further executable by the one or more processors to:

identify a second set of points within the portion of the second image;

determine, based at least partly on the second set of points, a location of the object within the environment; and

cause the projector to project content onto the object based at least partly on the location of the object.

8. The system as recited in claim 7, wherein the one or more computer-executable instructions are further executable by the one or more processors to determine the location of the object based at least partly on a comparison of the set of points and a second set of points that corresponds to the object in the second image.

9. A method comprising:

capturing, by a first camera, a first image that depicts a depth of one or more objects within an environment;

capturing, by a second camera, a second image;

determining, from the first image, a set of points that corresponds to an object depicted in the first image;

determining, based at least partly on the set of points, that a boundary of the object depicted in the first image is incomplete;

determining, based at least partly on the set of points and the boundary of the object depicted in the first image being incomplete, a second set of points that corresponds to the object depicted in the second image; and

determining, based at least partly on the second set of points, one or more characteristics associated with the object.

10. The method as recited in claim 9, wherein determining the one or more characteristics associated with the object comprises determining at least one of the boundary of the object, a shape of the object, a size of the object, a location of the object within the environment, or a color of at least a portion of the object.

11. The method as recited in claim 9, further comprising:

determining one or more second characteristics that are associated with a second object and that are stored in an objects database; and

determining an identity of the object based at least partly on the one or more characteristics and the one or more second characteristics.

12. The method as recited in claim 9, wherein determining the second set of points comprises identifying, based at least partly on the set of points, a portion of the second image that depicts the object.

13. The method as recited in claim 9, further comprising determining an identity of the object based at least partly on the one or more characteristics.

14. The method as recited in claim 9, further comprising determining that the one or more characteristics include at least one of the boundary of the object, a shape of the object, a size of the object, a location of the object within the environment, or a color of at least a portion of the object.

15. The method as recited in claim 9, further comprising:

generating, based at least partly on the first image, a depth map that represents a position and relative distance between the one or more objects within the environment; and

generating, based at least partly on the second image, an RGB image or a greyscale image.

16. The method as recited in claim 9, further comprising determining a first frame rate associated with the first camera and a second frame rate associated with the second camera that cause the first image and the second image to be captured at a same time.

17. A method comprising:

determining, from a first image that depicts a depth of one or more objects within an environment, a set of points that corresponds to an object depicted in the first image;

determining, based at least partly on the set of points, a portion of the second image that includes a second set of points that corresponds to the object depicted in the second image, the portion being a subset of an entirety of the second image and including a boundary that surrounds the object;

determining, based at least partly on the second set of points, one or more characteristics associated with the object; and

determining, based at least partly on the one or more characteristics, an identity of the object.

18. The method as recited in claim 17, wherein determining the identity of the object comprises:

determining, from an objects database, one or more second characteristics associated with a second object and one or more third characteristics associated with a third object; and

determining that the one or more characteristics match at least one of the one or more second characteristics or the one or more third characteristics.

19. The method as recited in claim 17, wherein determining the one or more characteristics associated with the object comprises determining at least one of a boundary of the object, a shape of the object, a size of the object, a location of the object within the environment, or a color of at least a portion of the object.

20. The method as recited in claim 17, further comprising:

determining, based at least partly on the second set of points, a location of the object within the environment; and

causing a projector to project content onto the object based at least partly on the location of the object.