Object detection and identification

• US 9,275,302 B1
• Filed: 08/24/2012
• Issued: 03/01/2016
• Est. Priority Date: 08/24/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Yan, Renwei, Chang, Samuel Henry, Crump, Edward Dietz
• Primary Examiner(s): Seth, Manav
• Application Number:
  US13/594,245
• Time in Patent Office: 1,285 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/48 Matching video sequences
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A two-dimensional (2D) image and a three-dimensional (3D) of an environment may be captured. Upon identifying a location and/or contour of an object from the 3D image, the object from the 3D image may be mapped onto the 2D image. The object, including its location and contour, may be identified from the 2D image. Based at least partly on a comparison between the object from the 3D image and the object from the 2D image, a disparity may be calculated. The location and contour of the object may be determined when it is determined that the disparity is less than or equal to a predetermined threshold. Otherwise, the object from the 3D image may be remapped onto the 2D image.

Claims

1. A system comprising:

one or more cameras to capture a two-dimensional (2D) image of an environment and a three-dimensional (3D) image of the environment;

memory;

one or more processors;

an object processing module maintained in the memory and executable by the one or more processors to;

identify a first contour of an object from the 3D image;

map the first contour of the object from the 3D image onto the 2D image;

identify a second contour of the object from the 2D image;

compare the first contour of the object from the 3D image to the second contour of the object from the 2D image; and

determine a disparity between the first contour of the object from the 3D image and the second contour of the object from the 2D image; and

an object determination module maintained in the memory and executable by the one or more processors to determine a final contour of the object based at least partly on a determination that the disparity is less than or equal to a threshold.

2. The system as recited in claim 1, wherein the final contour of the object is based on at least one of the first contour of the object from the 3D image or the second contour of the object from the 2D image.

3. The system as recited in claim 1, further comprising:

a projector; and

an output module maintained in the memory and executable by the one or more processors to cause the projector to project content onto the object according to the final contour of the object.

4. The system as recited in claim 1, wherein the object determination module is further executable by the one or more processors to remap the first contour from the 3D image onto the 2D image based at least partly on a determination that the disparity is greater than the threshold.

5. A method comprising:

under control of one or more processors of a computing device,

determining, from a three-dimensional (3D) image that depicts an environment, a set of points of the 3D image corresponding to an object depicted in the 3D image;

mapping the set of points of the 3D image onto a two-dimensional (2D) image that also depicts the environment;

determining a set of points of the 2D image corresponding to the object depicted in the 2D image; and

determining a contour of the object based at least in part on a comparison of the set of points of the 3D image and the set of points of the 2D image.

6. The method as recited in claim 5, further comprising determining the contour of the object based at least in part on a disparity between the set of points from the 3D image to the set of points from the 2D image.

7. The method as recited in claim 5, further comprising;

determining a location of the object within the environment; and

projecting content onto the object based at least partly on the location of the object.

8. The method as recited in claim 5, further comprising:

receiving data relating to a plurality of objects associated with the environment; and

identifying the object from the 3D image by comparing the set of points from the 3D image to the data relating to the plurality of objects.

9. The method as recited in claim 5, further comprising;

determining a disparity between the set of points of the 3D image and the set of points of the 2D image; and

determining the contour of the object at least partly in response to determining that the disparity is less than or equal to a threshold.

10. The method as recited in claim 9, further comprising remapping the set of points from the 3D image onto the set of points of the 2D image based at least partly on a determination that the disparity is greater than the threshold.

11. The method as recited in claim 5, further comprising determining a location of the object within the environment based at least in part on the comparison of the set of points of the 3D image and the set of points of the 2D image.

12. One or more non-transitory computer-readable media having computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

capturing a two-dimensional (2D) image and a three-dimensional (3D) image of an environment;

identifying an object depicted in the 3D image;

mapping the 3D image onto the 2D image to identify the object in the 2D image; and

determining a contour of the object based at least in part on a comparison of a set of points associated with the 3D image that corresponds to the object depicted in the 3D image and a set of points associated with the 2D image that corresponds to the object depicted in the 2D image.

13. The one or more non-transitory computer-readable media as recited in claim 12, wherein the 2D image and the 3D image are captured substantially simultaneously.

14. The one or more non-transitory computer-readable media as recited in claim 12, wherein the operations further comprise:

identifying a location of the object and the contour of the object from the 3D image; and

mapping the 3D image onto the 2D image according to the location of the object from the 3D image.

15. The one or more non-transitory computer-readable media as recited in claim 12, wherein the operations further comprise:

receiving data relating to a plurality of objects associated with the environment; and

identifying the object depicted in the 3D image by comparing the set of points associated with the 3D image to the data relating to the plurality of objects.

16. The one or more non-transitory computer-readable media as recited in claim 12, wherein the operations further comprise comparing a location of the object and a contour of the object from the 3D image to a corresponding location of the object and a corresponding contour of the object from the 2D image to determine a disparity between the location of the object and the contour of the object from the 3D image and the location of the object and the contour of the object from the 2D image.

17. The one or more non-transitory computer-readable media as recited in claim 16, wherein the operations further comprise determining a location of the object within the environment and the contour of the object based at least partly on a determination that the disparity is less than or equal to a predetermined threshold.

18. The one or more non-transitory computer-readable media as recited in claim 16, wherein the operations further comprise remapping the object from the 3D image onto the 2D image based at least partly on a determination that the disparity is greater than a predetermined threshold.

19. A system comprising:

memory;

one or more processors;

a projector;

an object processing module maintained in the memory and executable by the one or more processors to;

determine an object from a three-dimensional (3D) image of an environment; and

map the object from the 3D image onto a two-dimensional (2D) image of the environment;

an object determination module maintained in the memory and executable by the one or more processors to determine a contour of the object; and

an output module maintained in the memory and executable by the one or more processors cause the projector to project content onto the object based at least partly on the contour of the object.

20. The system as recited in claim 19, further comprising:

a 3D camera to capture the 3D image; and

a 2D camera to capture the 2D image.

21. The system as recited in claim 19, wherein the object processing module is further executable by the one or more processors to determine a disparity between a determined location of the object from the 3D image and a determined location of the object from the 2D image.

22. The system as recited in claim 21, wherein the object determination module is further executable by the one or more processors to:

determine that the disparity is greater than a predetermined threshold; and

remap the object from the 3D image onto the 2D image using the location of the object from the 3D image based at least in part in response to determining that the disparity is greater than the predetermined threshold.

23. The system as recited in claim 21, wherein the output module is further executable by the one or more processors to:

determine a location of the object within the environment; and

project the content onto the object based at least partly on the location of the object.