METHOD AND APPARATUS FOR REPRESENTING A PHYSICAL SCENE

• US 20150062120A1
• Filed: 08/28/2014
• Published: 03/05/2015
• Est. Priority Date: 08/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Siklossy, Istvan, Pan, Qi, Gervautz, Michael, Reisner-Kollmann, Irene, Gee, Andrew
• Granted Patent: US 9,996,974 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems, methods, and devices are described for constructing a digital representation of a physical scene by obtaining information about the physical scene. Based on the information, an initial portion of a planar surface within the physical scene may be identified. In one aspect of the disclosure, constructing a digital representation of a physical scene may include obtaining information about the physical scene, identifying a planar surface within the physical scene, selecting a physical object within the physical scene, placed above the planar surface, detecting properties associated with the physical object, generating a three-dimensional (3D) reconstructed object using the properties associated with the physical object, and representing the planar surface as an augmented reality (AR) plane in an augmented reality environment, wherein the AR plane in the AR environment is capable of supporting 3D reconstructed objects on top of it.

Claims

1. A method for constructing a digital representation of a physical scene comprising:

obtaining information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identifying the planar surface within the physical scene;

determining an AR plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole associated with a first object of the one or more physical objects located on the planar surface;

determining one or more properties associated with the first object based at least in part on information about the AR plane;

generating a three-dimensional (3D) model of the first object using the properties associated with the first object and information associated with the AR plane; and

generating an digital representation of an AR environment comprising the AR plane and the 3D model of the first object.

2. The method of claim 1, determining one or more properties associated with the first object is further based on identifying at least one or more feature points.

3. The method of claim 1, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.

4. The method of claim 1, wherein determining one or more properties associated with the first object comprises:

identifying one or more regions in the planar surface, wherein the one or more regions correspond to location of the first object on the planar surface; and

determining one or more external surfaces of the first object based on location of one or more convex regions and the one or more feature points.

5. The method of claim 1, wherein determining an AR plane corresponding to the planar surface comprises:

analyzing at least two images of the physical scene comprising the first object; and

based upon analysis of the at least two images of the physical scene, creating the AR plane such that the AR plane corresponds to the physical scene and excludes at least one region corresponding to the first object.

6. The method of claim 5, wherein analyzing the at least two images of the physical scene comprises:

capturing, using a camera, a first image of the physical scene at a first location, wherein the first image includes the first object;

determining a distance that the camera has moved from the first location to a second location;

capturing, using the camera, a second image of the physical scene at the second location, wherein the second image includes the first object; and

determining depth information for the first object using the first image and the second image.

7. The method of claim 1, wherein generating the 3D model of the first object comprises:

obtaining one or more feature points of the first object, wherein each feature point of the one or more feature points is indicative of a location, relative to the planar surface, of a point on an external surface of the first object;

estimating, using information associated with the AR plane and the one or more feature points, a shape of a portion of the first object that is not visible; and

constructing the 3D reconstructed object, based on the estimation.

8. The method of claim 7, wherein the estimation comprises at least one of:

a spline representation,

a reflected symmetrical object representation,

a planar representation, or

a curved representation.

9. The method of claim 1, wherein constructing the digital representation of the first object includes extruding object in a direction normal to the planar surface, based on a shape of a portion of the planar surface occluded by the first object.

10. The method of claim 1, wherein determining the AR plane corresponding to the planar surface comprises:

identifying an initial portion of the planar surface within the physical scene;

identifying additional portions of the planar surface, wherein each of the additional portions is identified as being a part of the planar surface based on determining a likeness between at least one visual or geometric property associated with the additional portions with at least one corresponding visual or geometric property associated with one or more portions already identified to be part of the planar surface.

11. The method of claim 10, wherein identifying the additional portions of the planar surface comprises:

establishing a model of a planar region within which the planar surface resides;

partitioning the planar region into a plurality of cells; and

evaluating each cell as a potential additional portion of the planar surface.

12. The method of claim 11, wherein evaluating each cell comprises iterating steps of:

for each cell identified as an additional portion of the planar surface, identifying neighboring cells not already identified as part of the planar surface; and

for each identified neighboring cell, evaluating the neighboring cell as a potential portion of the planar surface.

13. A device for constructing a digital representation of a physical scene, comprising:

a memory;

a camera for obtaining information about the physical scene; and

a processor coupled to the memory and configured to;

obtain information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identify the planar surface within the physical scene;

determine an AR plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole associated with a first object of the one or more physical objects located on the planar surface;

determine one or more properties associated with the first object based at least in part on information about the AR plane;

generate a three-dimensional (3D) model of the first object using the properties associated with the first object and information associated with the AR plane; and

generate an digital representation of an AR environment comprising the AR plane and the 3D model of the first object.

14. The device of claim 13, where in the processor is further configured to determine one or more properties associated with the first object is further based on identifying at least one or more feature points.

15. The device of claim 13, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.

16. The device of claim 13, wherein determining one or more properties associated with the first object comprises the processor further configured to:

identify one or more regions in the planar surface, wherein the one or more regions correspond to location of the first object on the planar surface; and

determine one or more external surfaces of the first object based on location of one or more convex regions and the one or more feature points.

17. The device of claim 13, wherein determining an AR plane corresponding to the planar surface comprises the processor further configured to:

analyze at least two images of the physical scene comprising the first object; and

based upon analysis of the at least two images of the physical scene, create the AR plane such that the AR plane corresponds to the physical scene and excludes at least one region corresponding to the first object.

18. The device of claim 17, wherein analyzing the at least two images of the physical scene comprises the processor further configured to:

capture, using a camera, a first image of the physical scene at a first location, wherein the first image includes the first object;

determine a distance that the camera has moved from the first location to a second location;

capture, using the camera, a second image of the physical scene at the second location, wherein the second image includes the first object; and

determine depth information for the first object using the first image and the second image.

19. The device of claim 13, wherein generating the 3D model of the first object comprises the processor further configured to:

obtain one or more feature points of the first object, wherein each feature point of the one or more feature points is indicative of a location, relative to the planar surface, of a point on an external surface of the first object;

estimate, using information associated with the planar surface and the one or more feature points, a shape of a portion of the first object that is not visible; and

construct the 3D reconstructed object, based on the estimation.

20. The device of claim 19, wherein the estimation comprises at least one of:

a spline representation,

a reflected symmetrical object representation,

a planar representation, or

a curved representation.

21. The device of claim 13, wherein constructing the digital representation of the first object includes the processor configured to extrude object in a direction normal to the planar surface, based on a shape of a portion of the planar surface occluded by the first object.

22. The device of claim 13, wherein determining the AR plane corresponding to the planar surface comprises the processor further configured to:

identify an initial portion of the planar surface within the physical scene;

identify additional portions of the planar surface, wherein each of the additional portions is identified as being a part of the planar surface based on determining a likeness between at least one visual or geometric property associated with the additional portions with at least one corresponding visual or geometric property associated with one or more portions already identified to be part of the planar surface.

23. The device of claim 22, wherein identifying the additional portions of the planar surface comprises the processor further configured to:

establish a model of a planar region within which the planar surface resides;

partition the planar region into a plurality of cells; and

evaluate each cell as a potential additional portion of the planar surface.

24. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a processor, the instructions comprising instructions to:

obtain information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identify the planar surface within the physical scene;

determine an AR plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole associated with a first object of the one or more physical objects located on the planar surface;

determine one or more properties associated with the first object based at least in part on information about the AR plane;

generate a three-dimensional (3D) model of the first object using the properties associated with the first object and information associated with the AR plane; and

generate an digital representation of an AR environment comprising the AR plane and the 3D model of the first object.

25. The non-transitory computer-readable storage medium of claim 24, wherein determining one or more properties associated with the first object is further based on identifying at least one or more feature points.

26. The non-transitory computer-readable storage medium of claim 24, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.

27. The non-transitory computer-readable storage medium of claim 24, wherein representing the planar surface as the AR plane comprises the processor further configured to:

analyze at least two images of the physical scene comprising the first object; and

based upon analysis of the at least two images of the physical scene, create the AR plane such that the AR plane corresponds to the physical scene and excludes at least one region corresponding to the first object.

28. An apparatus, comprising:

means for obtaining information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

means for identifying the planar surface within the physical scene;

means for determining an AR plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole associated with a first object of the one or more physical objects located on the planar surface;

means for determining one or more properties associated with the first object based at least in part on information about the AR plane;

means for generating a three-dimensional (3D) model of the first object using the properties associated with the first object and information associated with the AR plane; and

means for generating an digital representation of an AR environment comprising the AR plane and the 3D model of the first object.

29. The apparatus of claim 28, wherein determining one or more properties associated with the first object is further based on identifying at least one or more feature points.

30. The apparatus of claim 28, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.