Surface modeling with structured light

• US 8,970,693 B1
• Filed: 12/15/2011
• Issued: 03/03/2015
• Est. Priority Date: 12/15/2011
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Rawles, LLC (Amazon.com, Inc.)
• Inventors: Coley, Christopher, Worley, William Spencer III, Chang, Samuel Henry
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Le, Peter D
• Application Number:
  US13/327,141
• Time in Patent Office: 1,174 Days
• Field of Search: 348/136
• US Class Current: 348/136
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light

Abstract

Structured light patterns are projected onto an object and images of the structured light interacting with the surface of the object are acquired, as well as grayscale information. Edges within the structured light patterns are detected and depth sample points on the edges are used to determine distance to those sample points. The grayscale information is used to construct surface normals. From these normals relative surface contours such as curves or slopes may be determined. A model of a surface of the object is generated using the distances and the contours.

Claims

1. A system comprising:

one or more processors;

a projector coupled to the one or more processors and configured to project first and second structured light patterns which overlay, at least in part, a surface of an object within an environment;

a camera coupled to the one or more processors and configured to acquire images of the object while the first and second structured light patterns are overlaid onto the object;

a memory coupled to the one or more processors; and

a module stored within the memory and containing instructions, that when executed on the one or more processors perform at least the following acts;

acquire, from the camera, a first image of the environment when the first structured light pattern is projected at least partially onto the object, the first structured light pattern comprising two or more edges;

acquire, from the camera, a second image of the environment when the second structured light pattern is projected at least partially onto the object, the second structured light pattern also comprising two or more edges;

obtain a grayscale image of the object and detect the two or more edges of the first and second structured light patterns based at least in part on the first image and the second image;

determine distances between the camera and to two or more points on the object based at least in part on a distance within the acquired images between the two or more edges as detected in the first and second structured light patterns;

determine, based at least in part on the grayscale image, one or more surface normals of a portion of the surface of the object between the two or more points;

determine a contour between the two or more edges on the surface based at least in part on the one or more surface normals; and

construct at least a portion of a surface model of the object based at least in part on the distances between the camera and the two or more points and the contour.

2. The system of claim 1, wherein the first structured light pattern and the second structured light pattern are projected such that the patterns when projected are imperceptible by a human eye.

3. The system of claim 1, wherein the contour comprises one or more curves or one or more straight segments having a slope.

4. The system of claim 1, wherein each of the one or more of the surface normals correspond to an individual pixel from at least a portion of pixels in the grayscale image.

5. The system of claim 1, wherein the determining of the one or more surface normals comprises assuming the surface of the object to be Lambertian.

6. The system of claim 1, wherein the first structured light pattern and the second structured light pattern are configured such that the patterns when projected have alternating values of luminance, chrominance, or luminance and chrominance.

7. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

receiving a first image of an environment when a first structured light pattern having edges is projected onto the environment;

receiving a second image of the environment when a second structured light pattern having edges is projected onto the environment;

selecting a plurality of depth sample points at the edges formed by the first and second structured light patterns and determining distances from a designated origin point to the depth sample points;

determining, based at least in part on one or more surface normals determined from a grayscale image of the environment, a contour between at least two of the depth sample points; and

constructing a surface model of at least a portion of the environment based at least in part on the contour and the distances between the designated origin point and the depth sample points.

8. The one or more non-transitory computer-readable storage media of claim 7, wherein the determining the distances from the designated origin point to the depth sample points are based at least in part upon a distance between the depth sample points.

9. The one or more non-transitory computer-readable storage media of claim 7, wherein the grayscale image is obtained at least in part from the first and second images.

10. The one or more non-transitory computer-readable storage media of claim 7, wherein at least a portion of the first and second structured light patterns are projected onto the environment using different substantially monochromatic colors.

11. The one or more non-transitory computer-readable storage media of claim 10, wherein the substantially monochromatic colors include red, green, or blue.

12. The one or more non-transitory computer-readable storage media of claim 7, wherein the first and second structured light patterns when projected onto the environment are imperceptible to a human.

13. The one or more non-transitory computer-readable storage media of claim 7, wherein the determining the contour is constrained by a depth differential derived from the distances between the designated origin point and the plurality of depth sample points.

14. The one or more non-transitory computer-readable storage media of claim 7, wherein the contour comprise a slope or a relative angle of an object surface within the environment.

15. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

identifying, in an environment, a known object having a known shape at a known position in the environment;

acquiring a first image of the known object when illuminated by a first structured light pattern comprising a plurality of edges;

acquiring a second image of the known object when illuminated by a second structured light pattern comprising a plurality of edges;

obtaining a grayscale image of the known object at least in part from the first and second images;

determining a distance between a designated origin point and one or more points on the known object based at least in part on a plurality of depth sample points at the edges of the first and second structured light patterns;

determining a contour of at least a portion of the known object based at least in part on one or more surface normals derived from the grayscale image;

calibrating the determined distance using the known position of the known object in the environment; and

calibrating the determined contour using the known shape of the known object.

16. The one or more non-transitory computer-readable storage media of claim 15, wherein the identifying comprises imaging and recognizing one or more optical markings disposed on the known object.

17. The one or more non-transitory computer-readable storage media of claim 15, wherein the first and second structured light patterns are configured such that they are imperceptible to a human when projected.

18. One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform acts comprising:

measuring a distance from a designated origin point to an object with a non-imaging sensor to determine a measured distance;

acquiring a first image of the object when illuminated by a first structured light pattern having a plurality of edges;

acquiring a second image of the object when illuminated by a second structured light pattern having a plurality of edges;

generating a determined distance to one or more depth sample points disposed at the edges of the first and second of structured light patterns;

calibrating the determined distance using the measured distance;

obtaining a grayscale image at least in part from the first and second images; and

determining a contour between at least two depth sample points based at least in part on one or more surface normals derived from the grayscale image and constrained by the measured distance between the designated origin point and the at least two depth sample points.

19. The one or more non-transitory computer-readable storage media of claim 18, wherein the non-imaging sensor is configured to measure the distance from the designated origin point to the object with an acoustic signal.