Method and device for illustrating a virtual object in a real environment

• US 10,366,538 B2
• Filed: 10/19/2015
• Issued: 07/30/2019
• Est. Priority Date: 09/25/2007
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Meier, Peter, Holzer, Stefan
• Primary Examiner(s): Johnson, Motilewa Good
• Application Number:
  US14/887,280
• Publication Number:
  US 20170109929A1
• Time in Patent Office: 1,380 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

The invention relates to a method for representing a virtual object in a real environment, having the following steps: generating a two-dimensional image of a real environment by means of a recording device, ascertaining a position of the recording device relative to at least one component of the real environment, segmenting at least one area of the real environment unmarked in reality in the two-dimensional image for identifying at least one segment of the real environment in distinction to a remaining part of the real environment while supplying corresponding segmentation data, and merging the virtual object with the two-dimensional image of the real environment with consideration of the segmentation data such that at least one part of the segment of the real environment is removed from the image of the real environment. The invention permits any collisions of virtual objects with real objects that occur upon merging with a real environment to be represented in a way largely close to reality.

Claims

1. A method for representing a virtual object in a real environment, comprising:

capturing, by a recorder, a real environment in two dimensions;

determining position information for the recorder relative to at least one component of the real environment;

obtaining three-dimensional depth information relating to the real environment based on the position information;

segmenting at least one area of the captured real environment to identify at least one segment of the real environment distinct from a remaining part of the real environment;

merging a virtual object with the real environment such that at least one part of a segmented area of the real environment is removed from the real environment;

using the three-dimensional information for an area of the real environment adjacent to the removed part of the real environment to select a texture source from which to identify texture information to conceal the removed part of the real environment, wherein the texture source is selected from a group consisting of a floor plane and a wall plane;

identifying the texture information from the selected texture source; and

concealing the removed part of the real environment using the identified texture information.

2. The method of claim 1, wherein the recorder is a camera.

3. The method of claim 1, wherein the captured real environment in two dimensions is an image.

4. The method of claim 1, wherein merging a virtual object with the real environment comprises merging the virtual object with an image of the real environment.

5. The method of claim 4, wherein the image is displayed on a screen.

6. The method of claim 1, wherein the merging of the virtual object with the real environment is expressed on a semitransparent display apparatus.

7. The method of claim 6, wherein the merging comprises use of an optical mixer.

8. A non-transitory computer readable medium comprising one or more instructions that when executed on a central processing unit (CPU) configure the CPU to:

capture a real environment in two dimensions by actuating a recorder;

determine position information for the recorder relative to at least one component of the real environment;

obtain three-dimensional depth information relating to the real environment based on the position information;

segment at least one area of the captured real environment to identify at least one segment of the real environment distinct from a remaining part of the real environment;

merge a virtual object with the real environment such that at least one part of a segmented area of the real environment is removed from the real environment;

use the three-dimensional information for an area of the real environment adjacent to the removed part of the real environment to select a texture source from which to identify texture information to conceal the removed part of the real environment, wherein the texture source is selected from a group consisting of a floor plane and a wall plane;

identify the texture information from the selected texture source; and

conceal the removed part of the real environment with the identified texture information.

9. The non-transitory computer readable medium of claim 8, wherein the recorder is a camera.

10. The non-transitory computer readable medium of claim 8, wherein the captured real environment in two dimensions is an image.

11. The non-transitory computer readable medium of claim 8, wherein the instructions that when executed on a CPU configure the CPU to merge a virtual object with the real environment comprise instructions that when executed on the CPU configure the CPU to merge the virtual object with an image of the real environment.

12. The non-transitory computer readable medium of claim 8, wherein the instructions that when executed on a CPU configure the CPU to merge a virtual object with the real environment comprise instructions that when executed on the CPU configure the CPU to use an optical, semitransparent element, to merge the virtual object with reality.

13. The non-transitory computer readable medium of claim 11, wherein the image is displayed on a screen.

14. The non-transitory computer readable medium of claim 8, wherein a merged expression of the virtual object with the real environment is expressed on a semitransparent display apparatus.

15. The non-transitory computer readable medium of claim 14, wherein the instructions that when executed on a CPU configure the CPU to merge comprise instructions that when executed on the CPU configure the CPU to use an optical mixer.

16. A system comprising:

a memory configured to store instructions;

a recorder operatively coupled to the memory;

a computer processor operatively coupled to the memory and the recorder, the computer processor configured to execute the instructions stored in the memory so as to cause the system to;

capture a real environment in two dimensions by actuating the recorder;

determine position information for the recorder relative to at least one component of the real environment;

obtain three-dimensional depth information relating to the real environment based on the position information;

segment at least one area of the captured real environment to identify at least one segment of the real environment distinct from a remaining part of the real environment;

merge a virtual object with the real environment such that at least one part of a segmented area of the real environment is removed from the real environment;

use the three-dimensional information for an area of the real environment adjacent to the removed part of the real environment to select a texture source from which to identify texture information to conceal the removed part of the real environment, wherein the texture source is selected from a group consisting of a floor plane and a wall plane;

identify the texture information from the selected texture source; and

conceal the removed part of the real environment with the identified texture information.

17. The system of claim 16, wherein the computer processor being configured to execute the instructions stored in the memory so as to cause the system to merge a virtual object with the real environment comprises the computer processor being configured to execute the instructions stored in the memory so as to cause the system to merge the virtual object with an image of the real environment.

18. The system of claim 16, wherein the computer processor being configured to execute the instructions stored in the memory so as to cause the system to merge a virtual object with the real environment comprises the computer processor being configured to execute the instructions stored in the memory so as to cause the system to use an optical, semitransparent element, to merge the virtual object with reality.

19. The system of claim 18, wherein the image is expressed on a screen.

20. The system of claim 16, wherein a merged expression of the virtual object with the real environment is expressed on a semitransparent display apparatus.