Method and apparatus for representing a physical scene

• US 9,996,974 B2
• Filed: 08/28/2014
• Issued: 06/12/2018
• Est. Priority Date: 08/30/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Reisner-Kollmann, Irene, Siklossy, Istvan, Pan, Qi, Gervautz, Michael, Gee, Andrew
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US14/472,278
• Publication Number:
  US 20150062120A1
• Time in Patent Office: 1,384 Days
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Systems, methods, and devices are described for constructing a digital representation of a physical scene by obtaining information about the physical scene. Based on the information, an initial portion of a planar surface within the physical scene may be identified. In one aspect of the disclosure, constructing a digital representation of a physical scene may include obtaining information about the physical scene, identifying a planar surface within the physical scene, selecting a physical object within the physical scene, placed above the planar surface, detecting properties associated with the physical object, generating a three-dimensional (3D) reconstructed object using the properties associated with the physical object, and representing the planar surface as an augmented reality (AR) plane in an augmented reality environment, wherein the AR plane in the AR environment is capable of supporting 3D reconstructed objects on top of it.

Claims

1. A method for constructing a digital representation of a physical scene comprising:

obtaining information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identifying the planar surface within the physical scene;

determining an augmented reality (AR) plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole;

wherein a size of the at least one hole is adjusted based on an occluded portion of the planar surface;

determining one or more properties associated with a first object of the one or more physical objects located on the planar surface based at least in part on information about the at least one hole;

generating a three-dimensional (3D) model using the properties associated with the first object and information associated with the AR plane; and

generating a digital representation of an AR environment comprising the AR plane and the 3D model.

2. The method of claim 1, determining one or more properties associated with the first object is further based on identifying at least one or more feature points.

3. The method of claim 1, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.

4. The method of claim 1, wherein determining one or more properties associated with the first object comprises:

identifying one or more regions in the planar surface, wherein the one or more regions correspond to location of the first object on the planar surface; and

determining one or more external surfaces of the first object based on location of one or more convex regions and the one or more feature points.

5. The method of claim 1, wherein determining an AR plane corresponding to the planar surface comprises:

analyzing at least two images of the physical scene comprising the first object; and

based upon analysis of the at least two images of the physical scene, creating the AR plane such that the AR plane corresponds to the physical scene and excludes at least one region corresponding to the first object.

6. The method of claim 5, wherein analyzing the at least two images of the physical scene comprises:

capturing, using a camera, a first image of the physical scene at a first location, wherein the first image includes the first object;

determining a distance that the camera has moved from the first location to a second location;

capturing, using the camera, a second image of the physical scene at the second location, wherein the second image includes the first object; and

determining depth information for the first object using the first image and the second image.

7. The method of claim 1, wherein generating the 3D model comprises:

obtaining one or more feature points of the first object, wherein each feature point of the one or more feature points is indicative of a location, relative to the planar surface, of a point on an external surface of the first object;

estimating, using information associated with the AR plane and the one or more feature points, a shape of a portion of the first object that is not visible; and

constructing the 3D reconstructed object, based on the estimation.

8. The method of claim 7, wherein the estimation comprises at least one of:

a spline representation,

a reflected symmetrical object representation,

a planar representation, or

a curved representation.

9. The method of claim 1, wherein constructing the digital representation of the first object includes extruding object in a direction normal to the planar surface, based on a shape of a portion of the planar surface occluded by the first object.

10. The method of claim 1, wherein determining the AR plane corresponding to the planar surface comprises:

identifying an initial portion of the planar surface within the physical scene;

identifying additional portions of the planar surface, wherein each of the additional portions is identified as being a part of the planar surface based on determining a likeness between at least one visual or geometric property associated with the additional portions with at least one corresponding visual or geometric property associated with one or more portions already identified to be part of the planar surface.

11. The method of claim 10, wherein identifying the additional portions of the planar surface comprises:

establishing a model of a planar region within which the planar surface resides;

partitioning the planar region into a plurality of cells; and

evaluating each cell as a potential additional portion of the planar surface.

12. The method of claim 11, wherein evaluating each cell comprises iterating steps of:

for each cell identified as an additional portion of the planar surface, identifying neighboring cells not already identified as part of the planar surface; and

for each identified neighboring cell, evaluating the neighboring cell as a potential portion of the planar surface.

13. The method of claim 1, wherein the information about the physical scene is obtained by imaging the physical scene;

and

the occluded portion of the planar surface is caused by imaging at least a portion of the first object and at least a portion of the planar surface.

14. The method of claim 1, wherein the 3D model is a 3D model of the first object.

15. The method of claim 1, wherein the one or more properties associated with the first object include one or more feature points of the first object.

16. The method of claim 15, wherein the one or more properties of the first object are one or more properties indicating a size or shape of the first object.

17. The method of claim 1, wherein the 3D object is generated without using information obtained from imaging the first object.

18. The method of claim 13, further comprising:

identifying a sub-portion of the occluded portion corresponding to a shadow projected by the first object;

wherein the size of the at least one hole is adjusted by removing the sub-portion from the occluded portion.

19. A device for constructing a digital representation of a physical scene, comprising:

a memory;

a camera for obtaining information about the physical scene; and

a processor coupled to the memory and configured to;

obtain information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identify the planar surface within the physical scene;

determine an augmented reality (AR) plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole;

wherein a size of the at least one hole is adjusted based on occluded portion of the planar surface;

determine one or more properties associated with a first object of the one or more physical objects located on the planar surface based at least in part on information about the at least one hole;

generate a three-dimensional (3D) model using the properties associated with the first object and information associated with the AR plane; and

generate a digital representation of an AR environment comprising the AR plane and the 3D model.

20. The device of claim 19, where in the processor is further configured to determine one or more properties associated with the first object is further based on identifying at least one or more feature points.

21. The device of claim 19, wherein the AR plane comprises a plurality of cells, each cell of the plurality of cells representing a region wherein a virtual object can navigate without collision with the first object.

22. The device of claim 19, wherein determining one or more properties associated with the first object comprises the processor further configured to:

identify one or more regions in the planar surface, wherein the one or more regions correspond to location of the first object on the planar surface; and

determine one or more external surfaces of the first object based on location of one or more convex regions and the one or more feature points.

23. The device of claim 19, wherein determining an AR plane corresponding to the planar surface comprises the processor further configured to:

analyze at least two images of the physical scene comprising the first object; and

based upon analysis of the at least two images of the physical scene, create the AR plane such that the AR plane corresponds to the physical scene and excludes at least one region corresponding to the first object.

24. The device of claim 23, wherein analyzing the at least two images of the physical scene comprises the processor further configured to:

capture, using a camera, a first image of the physical scene at a first location, wherein the first image includes the first object;

determine a distance that the camera has moved from the first location to a second location;

capture, using the camera, a second image of the physical scene at the second location, wherein the second image includes the first object; and

determine depth information for the first object using the first image and the second image.

25. The device of claim 19, wherein generating the 3D model comprises the processor further configured to:

obtain one or more feature points of the first object, wherein each feature point of the one or more feature points is indicative of a location, relative to the planar surface, of a point on an external surface of the first object;

estimate, using information associated with the planar surface and the one or more feature points, a shape of a portion of the first object that is not visible; and

construct the 3D reconstructed object, based on the estimation.

26. The device of claim 25, wherein the estimation comprises at least one of:

a spline representation,

a reflected symmetrical object representation,

a planar representation, or

a curved representation.

27. The device of claim 19, wherein constructing the digital representation of the first object includes the processor configured to extrude object in a direction normal to the planar surface, based on a shape of a portion of the planar surface occluded by the first object.

28. The device of claim 19, wherein determining the AR plane corresponding to the planar surface comprises the processor further configured to:

identify an initial portion of the planar surface within the physical scene;

identify additional portions of the planar surface, wherein each of the additional portions is identified as being a part of the planar surface based on determining a likeness between at least one visual or geometric property associated with the additional portions with at least one corresponding visual or geometric property associated with one or more portions already identified to be part of the planar surface.

29. The device of claim 28, wherein identifying the additional portions of the planar surface comprises the processor further configured to:

establish a model of a planar region within which the planar surface resides;

partition the planar region into a plurality of cells; and

evaluate each cell as a potential additional portion of the planar surface.

30. A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises instructions executable by a processor, the instructions comprising instructions to:

obtain information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

identify the planar surface within the physical scene;

determine an augmented reality (AR) plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole;

wherein a size of the at least one hole is adjusted based on an occluded portion of the planar surface;

determine one or more properties associated with a first object of the one or more physical objects located on the planar surface based at least in part on information about the at least one hole;

generate a three-dimensional (3D) model using the properties associated with the first object and information associated with the AR plane; and

generate a digital representation of an AR environment comprising the AR plane and the 3D model.

31. An apparatus, comprising:

means for obtaining information about the physical scene containing at least a planar surface and one or more physical objects located on the planar surface;

means for identifying the planar surface within the physical scene;

means for determining an augmented reality (AR) plane corresponding to the planar surface wherein the AR plane comprises at least a portion of the planar surface and encircles at least one hole;

wherein a size of the at least one hole is adjusted based on occluded portion of the planar surface;

means for determining one or more properties associated with a first object of the one or more physical objects located on the planar surface based at least in part on information about the at least one hole;

means for generating a three-dimensional (3D) model using the properties associated with the first object and information associated with the AR plane; and

means for generating a digital representation of an AR environment comprising the AR plane and the 3D model.