Augmented reality images based on color and depth information

• US 9,818,224 B1
• Filed: 06/20/2013
• Issued: 11/14/2017
• Est. Priority Date: 06/20/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Worley, Connor Spencer Blue, Pauley, Devin Bertrum
• Primary Examiner(s): Harrison, Chante
• Application Number:
  US13/922,999
• Time in Patent Office: 1,608 Days
• Field of Search: 345619, 345660, 345420, 345424, 345633, 345629, 345632, 705 2681, 705 261, 705 265, 705 272, 705 271, 348239
• CPC Class Codes: G06Q 30/0601 Electronic shopping [e-shopping]
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals

Abstract

Techniques are described for generating a composite image that depicts a first object with a second object. Two-dimensional images of the first object are captured, along with depth information that includes three-dimensional coordinates of points on the surface of the first object. Based on the depth information, a polygonal model may be determined for the first object including color information determined from the images. The polygonal model of the first object may be placed with a polygonal model of the second object in a virtual scene, and ray tracing operations may generate a plurality of pixels for the composite image. In cases where the first object represents a user and the second object represents a product, the composite image may provide a substantially realistic preview of the user with the product.

Claims

1. A computer-implemented method, comprising:

accessing one or more images of a first object, the one or more images captured using at least one camera included in a user device;

accessing depth information determined by at least one depth sensor of the user device measuring distance from the user device to at least a portion of the first object;

receiving, from the user device, a composite image request including an object identifier (ID) identifying a second object to be rendered with the first object and user specified position information indicating a preferred position at which to display the second object relative to the first object in a composite image;

processing the depth information to determine a first plurality of polygons representing at least a portion of a surface of the first object;

accessing a three-dimensional model of the second object using the object ID identifying the second object, the three-dimensional model including three-dimensional coordinates of points associated with a second plurality of polygons representing at least a portion of a surface of the second object identified by the object ID;

arranging the second plurality of polygons representing at least the portion of the surface of the second object relative to the first plurality of polygons representing at least the portion of the surface of the first object in a virtual scene based on the user specified position information specified in the composite image request;

generating the composite image of the virtual scene, wherein the composite image comprises a plurality of pixels, and wherein values for the plurality of pixels are determined by ray tracing one or more virtual light rays from at least one virtual light source toward the second plurality of polygons representing at least the portion of the surface of the second object identified by the object ID and the first plurality of polygons representing at least the portion of the surface of the first object in the virtual scene; and

sending the composite image to the user device.

2. The method of claim 1, further comprising:

receiving, from the user device, information describing one or more of user specified orientation information indicative of an orientation to render the second object identified by the object ID relative to the first object in the virtual scene or user specified scale information indicative of a scale at which to render the second object relative to the first object in the virtual scene; and

wherein the arranging a visual representation of the second object relative to the visual representation of the first object in the virtual scene is further based on one or more of the user specified orientation information or the user specified scale information.

3. The method of claim 1, further comprising:

determining a color of at least one of the first plurality of polygons, based at least partly on color information included in the one or more images of the first object; and

wherein the generating of the composite image is further based at least partly on the color of the at least one of the first plurality of polygons.

4. The method of claim 1, further comprising:

determining a reflectance of the at least one of the second plurality of polygons, based at least partly on at least one material that comprises at least the portion of the surface of the second object; and

wherein the generating of the composite image is further based at least partly on the reflectance of the at least one of the second plurality of polygons.

5. A system, comprising:

at least one memory storing computer-executable instructions; and

at least one processor in communication with the at least one memory, the at least one processor configured to access the at least one memory and execute the computer-executable instructions to;

access one or more images of a first object;

access depth information indicating distance from a user device to at least a portion of the first object;

receive, from the user device, a composite image request including an object identifier (ID) identifying a second object to be rendered with the first object and user specified position information indicating a preferred position at which to display the second object relative to the first object in a composite image;

processing the depth information to determine a first plurality of polygons representing at least a portion of a surface of the first object;

access a three-dimensional model of the second object using the object ID identifying the second object including three-dimensional coordinates of points associated with a second plurality of polygons representing at least a portion of a surface of the second object identified by the object ID;

arrange the second plurality of polygons representing at least the portion of the surface of the second object relative to the first plurality of polygons representing at least the portion of the surface of the first object in a virtual scene based on the user specified position information;

generate the composite image of the virtual scene, wherein the composite image comprises a plurality of pixels, and wherein values for the plurality of pixels are determined by using ray tracing by directing one or more virtual light rays from at least one virtual light source toward the second plurality of polygons representing at least the portion of the surface of the second object identified by the object ID and the first plurality of polygons representing at least the portion of the surface of the first object in the virtual scene; and

send the composite image to the user device.

6. The system of claim 5, wherein the one or more images are captured using at least one camera included in the user device.

7. The system of claim 5, wherein:

the depth information describes coordinates of a plurality of tracking points associated with at least the portion of the surface of the first object, the plurality of tracking points determined by at least one depth sensor of the user device measuring distance from the user device to at least the portion of the surface of the first object; and

the determining of the first plurality of polygons is based at least partly on the coordinates of the plurality of tracking points.

8. The system of claim 5, wherein:

the at least one processor is further configured to determine one or more colors corresponding to the first plurality of polygons, based at least partly on color information included in the one or more images of the first object; and

the generating of the composite image of the virtual scene is further based at least partly on the one or more colors corresponding to the first plurality of polygons.

9. The system of claim 5, wherein:

the at least one processor is further configured to determine a reflectance of at least one of the second plurality of polygons, based at least partly on at least one material comprising at least the portion of the surface of the second object identified by the object ID; and

the generating of the composite image of the virtual scene is further based at least partly on the reflectance of the at least one of the second plurality of polygons.

10. The system of claim 5, wherein the three-dimensional model of the second object identified by the object ID comprises one or more of:

a plurality of points that describe physical dimensions of the second object;

a plurality of edges that describe a physical extent of the second object;

or

the second plurality of polygons that describe one or more surfaces of the second object.

11. The system of claim 5, wherein:

the first object includes at least a portion of a person; and

the second object identified by the object ID includes at least a portion of a product available for purchase.

12. The system of claim 5, wherein:

the composite image request includes information describing one or more of user specified orientation information indicative of an orientation to render the second object identified by the object ID relative to the first object in the virtual scene or user specified scale information indicative of a scale at which to render the second object relative to the first object in the virtual scene; and

the arranging of the second object identified by the object ID relative to the first object in the virtual scene is further based on one or more of the user specified orientation information or the user specified scale information.

13. The system of claim 5, wherein the ray tracing is of one or more virtual light rays from the at least one virtual light source to points in the virtual scene.

14. The system of claim 13, wherein the at least one processor is further configured to:

receive ambient light information describing ambient light illuminating the first object; and

determine a position of the at least one virtual light source based at least partly on the ambient light information.

15. One or more non-transitory computer-readable media storing instructions which, when executed by at least one processor, instruct the at least one processor to perform actions comprising:

accessing one or more images of a first object including depth information associated with the first object;

receiving a composite image request including an object identifier (ID) identifying a second object to be rendered with the first object and user specified position information indicating a preferred position at which to display the second object identified by the object ID relative to the first object in a composite image;

processing the depth information to determine a surface of the first object;

accessing a three-dimensional model of the second object using the object ID identifying the second object, the three-dimensional model including three-dimensional coordinates of points associated with a surface of the second object;

arranging a visual representation of the second object identified by the object ID relative to a visual representation of the first object in a virtual scene based at least partly on the three-dimensional coordinates of the points associated with the surface of the second object from the three-dimensional model of the second object identified by the object ID and on the user specified position information specified in the composite image request; and

generating the composite image of the virtual scene, wherein the composite image comprises a plurality of pixels, and wherein values for the plurality of pixels are determined by using ray tracing directing at least one virtual light source toward the surface of the second object identified by the object ID and the surface of the first object.

16. The one or more non-transitory computer-readable media of claim 15, the actions further comprising:

including information in the composite image request describing one or more of user specified orientation information indicative of an orientation to render the second object identified by the object ID relative to the first object in the virtual scene or user specified scale information indicative of a scale at which to render the second object relative to the first object in the virtual scene; and

wherein the arranging of the second object identified by the object ID relative to the first object in the virtual scene is further based on one or more of the user specified orientation information or the user specified scale information.

17. The one or more non-transitory computer-readable media of claim 15, wherein:

the arranging the visual representation of the second object identified by the object ID relative to the visual representation of the first object in the virtual scene includes arranging at least a portion of the one or more images of the first object with at least a portion of the model of the second object identified by the object ID;

the ray tracing includes tracing one or more virtual light rays from at least one virtual light source to one or more points in the virtual scene that correspond to the second object identified by the object ID; and

assembling the plurality of pixels to generate the composite image of the virtual scene comprises assembling at least a portion of the one or more images of the first object with the plurality of pixels corresponding to the second object identified by the object ID.

18. The one or more non-transitory computer-readable media of claim 15, wherein the three-dimensional model of the second object identified by the object ID includes one or more of:

a plurality of points that describe physical dimensions of the second object;

a plurality of edges that describe a physical extent of the second object;

or

a plurality of polygons that describe one or more surfaces of the second object.

19. The one or more non-transitory computer-readable media of claim 15, wherein

the first object includes at least a portion of a person;

and

the second object identified by the object ID includes a product available for purchase.

20. The one or more non-transitory computer-readable media of claim 19, wherein:

the arranging of the product with the person in the virtual scene is based at least partly on the user specified position information.

21. The one or more non-transitory computer-readable media of claim 15, wherein the ray tracing includes tracing one or more virtual light rays from at least one virtual light source to one or more points in the virtual scene, and wherein the actions further comprise:

receiving ambient light information describing ambient light illuminating the first object; and

determining a position of the at least one virtual light source based at least partly on the ambient light information.