Method and device for illustrating a virtual object in a real environment

• US 9,165,405 B2
• Filed: 09/23/2008
• Issued: 10/20/2015
• Est. Priority Date: 09/25/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Meier, Peter, Holzer, Stefan
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US12/680,117
• Publication Number:
  US 20100289817A1
• Time in Patent Office: 2,583 Days
• Field of Search: 345/632, 345/633
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

The invention relates to a method for representing a virtual object in a real environment, having the following steps: generating a two-dimensional image of a real environment by means of a recording device, ascertaining a position of the recording device relative to at least one component of the real environment, segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data, and merging the virtual object with the two-dimensional image of the real environment with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment. The invention permits any collisions of virtual objects with real objects that occur upon merging with a real environment to be represented in a way largely close to reality.

Claims

1. A method for representing a virtual object in a real environment, having the following steps:

generating at least one two-dimensional image of a real environment by means of a recording device;

ascertaining a position of the recording device relative to at least one component of the real environment, and providing three-dimensional information relating to the real environment based on the position of the recording device;

segmenting at least one area of the real environment within the two-dimensional image to identify at least one segment of the real environment distinct from a remaining part of the real environment, wherein the segmenting is performed using graph cuts, which segmenting produces segmentation data;

merging a virtual object with the two-dimensional image of the real environment or, by means of an optical, semitransparent element, directly with reality, with consideration of the segmentation data such that at least one part of the segmented area of the real environment is removed from the image of the real environment;

identifying texture information within two-dimensional image using the th dimensional information; and

concealing the removed part of the real environment in the image of the real environment using the texture information.

2. The method according to claim 1, wherein the at least one part of the segmented area of the real environment that is removed from the original image of the real environment is reinserted at another place in the representation of the virtual object in the real environment.

3. The method according to claim 1, wherein at least one area of the real environment is selected from which information is obtained for concealing the part of the real environment to be removed.

4. The method according to claim 1, wherein the information with regard to the selected segment is stated spatially, for example by means of a bounding geometry, and can also be reused for other views of the same real object from which the segment has resulted.

5. The method according to claim 1, wherein the texture information is in the surroundings of the part to be removed.

6. The method according to claim 5, wherein the three-dimensional depth information is in the form of an existing floor plane or an existing wall plane.

7. The method according to claim 1, wherein for identifying at least one segment of the real environment, an area of the real environment is identified which goes beyond a boundary between an existing floor plane and an existing wall plane of the real environment.

8. The method according to claim 1, wherein for identifying at least one segment of the real environment, a virtual geometrical figure is positioned at a corresponding place in the real environment.

9. The method according to claim 8, wherein

the virtual geometrical figure is a three-dimensional figure which surrounds the segment of the real environment to be identified;

and

the three-dimensional figure is moved within the space and parallel to the floor plane in a way approximately close to reality on the basis of the spatial description and/or the floor plane known on the basis of an optical marking.

10. The method according to claim 1, wherein spatial information ascertained by means of spatial description or segmentation data restricts the room for movement of virtual objects.

11. The method according to claim 1, wherein spatial information ascertained by means of spatial description or segmentation data influences the shadows cast by virtual objects.

12. The method according to claim 1, wherein

the real environment is optically recorded by means of the recording device from different positions of the recording device, and a respective recording is stored in a corresponding recording image data set;

and

the segment data obtained in one of the recording image data sets are employed in the other ones of the recording image data sets to represent the real environment with the inserted virtual object in respective views corresponding to the recording image data sets.

13. A non-transitory computer readable medium comprising software code sections adapted to be loaded into the internal memory of a digital computer system, which computer system is coupled with at least one of a recording device operable to generate a 2D image, or a memory medium for loading a 2D image therefrom, and by means of which the steps according to claim 1 are performed when running on the computer system.

14. The method according to claim 1, wherein the texture information relates to an existing floor or an existing wall within the two-dimensional image.

15. An apparatus for representing a virtual object in a real environment, having the following features:

a recording device for generating a two-dimensional image of the real environment;

a data processing system which determines the position of the recording device relative to at least one component of the recorded real environment and creates three-dimensional information relating to the real environment based on the determined position of the recording device;

wherein in the data processing system there is implemented a method which is suitable for segmenting at least one area of the real environment within the two-dimensional image to identify at least one segment of the real environment distinct from a remaining part of the real environment, wherein the segmenting is performed using graph cuts, and which segmenting produces segmentation data; and

an indicating device connected to the data processing system, which displays the image of the real environment generated by the recording device merged with the virtual object with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment;

wherein the method is adapted to identify texture information within the two-dimensional image using the three-dimensional information; and

wherein the method is further adapted to conceal the removed part of the real environment in the image of the real environment using the identified texture information.

16. An apparatus for representing a virtual object in a real environment, having the following features:

a recording device for generating a two-dimensional image of the real environment;

a data processing system which determines the position of the recording device relative to at least one component of the recorded real environment and creates three-dimensional information relating to the real environment based on the determined position of the recording, device; and

an optical, semitransparent display apparatus with which the real environment can be seen via a transparency of the display apparatus, and a merging of the real environment with a virtual object is generatable via projection onto the display apparatus;

wherein the data processing system determines the position of the display apparatus relative to at least one component of the real environment;

wherein in the data processing system there is implemented a method which is suitable for segmenting at least one area of the real environment within the two-dimensional image to identify at least one segment of the real environment distinct from a remaining part of the real environment, wherein the segmenting is performed using graph cuts, and which segmenting produces segmentation data;

wherein the method is adapted to identify texture information within the two-dimensional image using the three-dimensional information; and

wherein a representation of the virtual object merged with the real environment with consideration of the segmentation data is displayed on the display apparatus such that at least one part of the segment of the real environment is removed from the representation;

wherein the method is further adapted to conceal the removed part of the real environment in the image of the real environment using the identified texture information.