Characterization of a scene with structured light

• US 9,766,057 B1
• Filed: 09/29/2014
• Issued: 09/19/2017
• Est. Priority Date: 12/23/2010
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Coley, Christopher, Betts-LaCroix, Jonathan, Worley, III, William Spencer
• Primary Examiner(s): CHOWDHURY, SULTAN U.
• Application Number:
  US14/500,627
• Time in Patent Office: 1,086 Days
• CPC Class Codes: G01B 11/022 by means of tv-camera scanning
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G03B 21/147 Optical correction of image distortions, e.g. keystone
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/012 Dimensioning, tolerancing
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Objects and their placement within a particular scene may be characterized using structured light with no or minimal manual intervention or calibration to the scene. The structured light pattern may be rendered with visible light, yet imperceptible to users, by generating a set of opposing excursions in one or more image properties for durations which are shorter than the integration time of the human eye. A set of assumptions and ongoing testing of those assumptions against physical objects identified within the environment allows characterization of the scene and the objects therein.

Claims

1. A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing instructions executable by the one or more processors, wherein the instructions program the one or more processors to implement a service to;

project a pre-determined structured light pattern onto a scene;

acquire a distortion pattern produced by the pre-determined structured light pattern interacting with one or more objects within the scene;

compare the distortion pattern with the pre-determined structured light pattern to determine structured light data;

determine a topology of the scene based on the structured light data, the topology being a surface representation of the scene including depth of field data;

determine one or more object parameters of an object of the one or more objects;

identify the object based on the topology of the scene by comparing the one or more object parameters to dimensional data about one or more known objects;

determine one or more preliminary dimensions of the object based at least in part upon identifying the object;

determine a plurality of tentative dimensions of an unidentified object of the one or more objects based at least in part upon the one or more preliminary dimensions and one or more relative positions of the object and the unidentified object, the plurality of tentative dimensions including a first tentative dimension of the unidentified object and a second tentative dimension of the unidentified object that is different from the first tentative dimension;

perform a test of the plurality of tentative dimensions against a pre-determined set of assumptions about at least one known object of the one or more known objects;

discard the first tentative dimension based on a result of the test contradicting the pre-determined set of assumptions; and

determine the second tentative dimension and a particular preliminary dimension of the one or more preliminary dimensions as final dimensions based at least in part on a determination that an error associated with the particular preliminary dimension and the particular tentative dimension is below a pre-determined error threshold value.

2. The system as recited in claim 1, wherein the one or more object parameters include one or more of:

a shape of at least one object of the one or more objects;

a dimension of at least one object of the one or more objects;

an identifying marking of at least one object of the one or more objects; and

a reflectivity of at least one object of the one or more objects.

3. The system as recited in claim 1, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including a particular assumption comprising a relation of a first magnitude of a first dimension of a first known object relative to a second magnitude of a second dimension of a second known object, and wherein the test comprises:

determining that a third magnitude of the first tentative dimension relative to a fourth magnitude of a third dimension of a particular identified object does not match the relation of the particular assumption; and

determining that a fifth magnitude of the second tentative dimension relative to the fourth magnitude of the third dimension of the particular identified object does match the relation of the particular assumption.

4. The system as recited in claim 1, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including an assumption regarding a first location within a physical environment where a first known object is unexpected to reside and a second location within the physical environment where the first known object is expected to reside, and wherein the test comprises:

determining a first tentative position of the unidentified object based at least in part on a first determination that the first tentative dimension is one of the final dimensions;

determining that the first tentative position of the unidentified object matches the first location within the physical environment;

determining a second tentative position of the unidentified object based at least in part on a second determination that the second tentative dimension is one of the final dimensions; and

determining that the second tentative position of the unidentified object matches the second location within the physical environment.

5. One or more non-transitory computer-readable storage media storing computer-executable instructions that, when executed, cause one or more processors to:

identify one or more objects within a scene based on a topology of the scene;

determine a preliminary dimension of the one or more objects within the scene;

determine a tentative dimension of one or more unidentified objects within the scene based at least in part upon the preliminary dimension of the one or more objects and one or more relative positions of the one or more objects and the one or more unidentified objects;

perform a test of the tentative dimension against a pre-determined set of assumptions; and

determine the tentative dimension and the preliminary dimension as final dimensions based at least in part on a determination that the preliminary dimension and the tentative dimension are within a pre-determined error threshold value.

6. The one or more non-transitory computer-readable storage media of claim 5, wherein the identifying of the one or more objects within the scene comprises:

determining one or more object parameters of objects within the scene; and

identifying the one or more objects by comparing the one or more object parameters to data about one or more known objects.

7. The one or more non-transitory computer-readable storage media of claim 5, wherein object parameters associated with an object of the one or more objects include one or more of:

a shape of the object;

one or more dimensions of the object;

one or more identifying markings of the object; and

a reflectivity of the object.

8. The one or more non-transitory computer-readable storage media of claim 7, wherein the instructions, when executed, further cause the one or more processors to project a pre-determined structured light pattern onto the scene, wherein projecting the pre-determined structured light pattern onto the scene comprises:

embedding the pre-determined structured light pattern within a plurality of visible light sub-frames such that the plurality of visible light sub-frames contain a set of opposing excursions in one or more image properties, wherein the set of opposing excursions are deviations from image properties of base frames corresponding to the plurality of visible light sub-frames and wherein the set of opposing excursions vary between pluralities of visible light sub-frames corresponding to at least two different base frames; and

projecting the plurality of visible light sub-frames onto the scene.

9. The one or more non-transitory computer-readable storage media of claim 5, wherein the topology comprises a surface representation of the scene including depth information.

10. The one or more non-transitory computer-readable storage media of claim 5, the instructions, when executed, further causing the one or more processors to:

generate a spatial model of the scene based at least in part upon the topology and the final dimensions, wherein the spatial model includes relative locations and final dimensions of the one or more objects and the one or more unidentified objects within a three-dimensional spatial representation of the scene.

11. The one or more non-transitory computer-readable storage media of claim 5, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including a particular assumption comprising a relation of a first magnitude of a first dimension of a first known object relative to a second magnitude of a second dimension of a second known object, and wherein the test comprises:

determining whether a third magnitude of the tentative dimension relative to a fourth magnitude of a third dimension of a particular identified object matches the relation of the particular assumption.

12. The one or more non-transitory computer-readable storage media of claim 5, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including an assumption regarding a location within a physical environment where a first known object is expected or unexpected to reside, and wherein the test comprises:

determining a tentative position of a particular unidentified object of the one or more unidentified objects based at least in part on a determination that the first tentative dimension is one of the final dimensions of the particular unidentified object; and

determining whether the tentative position of the particular unidentified object matches the first location.

13. The one or more non-transitory computer-readable storage media of claim 5, the instructions, when executed, further causing the one or more processors to:

performing a second test of second tentative dimension of the one or more unidentified objects against the pre-determined set of assumptions; and

discard the second tentative dimension based on a contradictory dimensional result generated from the second test of the second tentative dimension.

14. A method comprising:

identifying one or more objects within a scene based on a topology of the scene;

determining a preliminary dimension of an object of the one or more objects;

determining a tentative dimension of an undimensioned object within the scene based at least in part upon the preliminary dimension of the object;

testing the preliminary dimension of the object against a pre-determined set of assumptions about the one or more objects; and

determining the tentative dimension and the preliminary dimension as final dimensions based at least in part on a determination that the preliminary dimension and the tentative dimension are within a pre-determined error threshold value.

15. The method of claim 14, further comprising:

testing a second preliminary dimension of the object against the pre-determined set of assumptions; and

discarding the second preliminary dimension based on a contradictory dimensional result generated from the testing of the second preliminary dimension.

16. The method of claim 14, further comprising:

testing the tentative dimension of the one or more objects against a second pre-determined set of assumptions about the undimensioned object;

testing a second tentative dimension of the undimensioned object against the second pre-determined set of assumptions; and

discarding the second tentative dimension based on a contradictory dimensional result generated from the testing of the second tentative dimension.

17. The method of claim 14, wherein identifying the one or more objects within the scene comprises:

determining one or more object parameters of the one or more objects; and

identifying the one or more objects by comparing the one or more object parameters to data about one or more known objects.

18. The method of claim 14, further comprising determining one or more object parameters of the object, wherein the one or more object parameters include one or more of:

a shape of the object;

one or more dimensions of the object;

one or more identifying markings of the object; and

a reflectivity of the object.

19. The method of claim 14, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including a particular assumption comprising a relation of a first magnitude of a first dimension of a first known object relative to a second magnitude of a second dimension of a second known object, and wherein testing comprises:

determining whether a third magnitude of the tentative dimension relative to a fourth magnitude of a dimension of a particular dimensioned object matches the relation of the particular assumption.

20. The method of claim 19, wherein the pre-determined set of assumptions comprise a pre-determined set of rules describing behavior of physical objects including an assumption regarding a location within a physical environment where a first known object is expected or unexpected to reside, and wherein the testing comprises:

determining a tentative position of a particular undimensioned object based at least in part on a determination that the tentative dimension is one of the final dimensions; and

determining whether the tentative position of the particular undimensioned object matches the location.