METHOD AND DEVICE FOR ILLUSTRATING A VIRTUAL OBJECT IN A REAL ENVIRONMENT

• US 20100289817A1
• Filed: 09/23/2008
• Published: 11/18/2010
• Est. Priority Date: 09/25/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Holzer, Stefan, Meier, Peter
• Granted Patent: US 9,165,405 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

The invention relates to a method for representing a virtual object in a real environment, having the following steps: generating a two-dimensional image of a real environment by means of a recording device, ascertaining a position of the recording device relative to at least one component of the real environment, segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data, and merging the virtual object with the two-dimensional image of the real environment with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment. The invention permits any collisions of virtual objects with real objects that occur upon merging with a real environment to be represented in a way largely close to reality.

Claims

1. A method for representing a virtual object in a real environment, having the following steps:

generating at least one two-dimensional image of a real environment by means of a recording device,

ascertaining a position of the recording device relative to at least one component of the real environment,

segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data,

merging the virtual object with the two-dimensional image of the real environment or, by means of an optical, semitransparent element, directly with reality with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment.

4. The method according to claim 1, wherein

the at least one part of the segment of the real environment is removed from the original image of the real environment and reinserted at another place in the representation.

7. The method according to claim 1, wherein

at least one area of the real environment is selected from which information is obtained for concealing the part of the real environment to be removed.

8. The method according to claim 1, wherein

the information with regard to the selected segment is stated spatially, for example by means of a bounding geometry, and can also be reused for other views of the same real object from which the segment has resulted.

9. The method according to claim 1, wherein

a concealment of the part of the real environment to be removed is effected on the basis of texture information in the surroundings of the part to be removed.

10. The method according to claim 9, wherein

three-dimensional depth information for selection of suitable information sources, in particular in the form of an existing floor plane or an existing wall plane, is considered for the choice of the employed texture information.

13. The method according to claim 1, wherein

for identifying at least one segment of the real environment, an area of the real environment is identified which goes beyond a boundary between an existing floor plane and an existing wall plane of the real environment.

15. The method according to claim 1, wherein

for identifying at least one segment of the real environment, a virtual geometrical figure is positioned at a corresponding place in the real environment.

19. The method according to claim 1, wherein

spatial information ascertained by means of spatial description or segmentation data restricts the room for movement of virtual objects.

20. The method according to claim 1, wherein

spatial information ascertained by means of spatial description or segmentation data influences the shadows cast by virtual objects.

26. The method according to claim 1, wherein

the real environment is optically recorded by means of the recording device from different positions of the recording device, and a respective recording is stored in a corresponding recording image data set,

the segment data obtained in one of the recording image data sets are employed in the other ones of the recording image data sets to represent the real environment with the inserted virtual object in respective views corresponding to the recording image data sets.

33. A computer program product which has software code portions which are suitable in a method according to claim 1 for ascertaining the position of the recording device, for segmenting at least one area of the real environment and for merging the virtual object with the real environment with consideration of the segmentation data when they are stored in a system for representing a virtual object in a real environment.

2. (canceled)

3. (canceled)

5. (canceled)

6. (canceled)

11. (canceled)

12. (canceled)

14. (canceled)

16. (canceled)

18. The method according to claim 16, wherein

the three-dimensional figure is moved within the space and parallel to the floor plane in a way approximately close to reality on the basis of the spatial description and/or the floor plane known on the basis of an optical marking.

17. (canceled)

21. (canceled)

22. (canceled)

23. (canceled)

24. (canceled)

25. (canceled)

27. (canceled)

28. (canceled)

29. (canceled)

30. (canceled)

31. An apparatus for representing a virtual object in a real environment, having the following features:

a recording device for generating a two-dimensional image of the real environment,

a data processing system which determines the position of the recording device relative to at least one component of the recorded real environment,

wherein in the data processing system there is implemented a method which is suitable for segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data,

and an indicating device connected to the data processing system, which displays the image of the real environment generated by the recording device merged with the virtual object with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment.

32. An apparatus for representing a virtual object in a real environment, having the following features:

a recording device for generating a two-dimensional image of the real environment,

a data processing system which determines the position of the recording device relative to at least one component of the recorded real environment,

an optical, semitransparent display apparatus with which the real environment can be seen via a transparency of the display apparatus, and a merging of the real environment with a virtual object is generatable via projection onto the display apparatus,

wherein the data processing system determines the position of the display apparatus relative to at least one component of the real environment,

wherein in the data processing system there is implemented a method which is suitable for segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data,

wherein the representation of the virtual model merged with the real environment with consideration of the segmentation data is displayed on the display apparatus such that at least one part of the segment of the real environment is removed from the representation.