Mixed reality space image generation method and mixed reality system

• US 20050179617A1
• Filed: 09/29/2004
• Published: 08/18/2005
• Est. Priority Date: 09/30/2003
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Suzuki, Masahiro, Matsui, Taichi
• Granted Patent: US 7,589,747 B2
• Time in Patent Office: Days
• US Class Current: 345/7
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A mixed reality space image generation apparatus for generating a mixed reality space image formed by superimposing virtual space images onto a real space image obtained by capturing a real space, includes an image composition unit (109) which superimposes a virtual space image, which is to be displayed in consideration of occlusion by an object on the real space of the virtual space images, onto the real space image, and an annotation generation unit (108) which further imposes an image to be displayed without considering any occlusion of the virtual space images. In this way, a mixed reality space image which can achieve both natural display and convenient display can be generated.

Claims

1. A mixed reality space image generation method for generating a mixed reality space image formed by superimposing virtual space image onto a real space image obtained by capturing a real space, comprising:

a first image superimposing step of superimposing a first virtual space image onto a real space image based on an occlusion by a real object; and

a second image superimposing step of superimposing a second virtual space image onto the superimposed image generated in the first image superimposing step without considering the occlusion.

2. The method according to claim 1, wherein the first image superimposing step includes:

extracting a predetermined object region from the real space image; and

superimposing the first virtual space image onto the real space image other than the predetermined object region.

3. The method according to claim 1, wherein the first virtual space image is an outer appearance of the real object, and the second virtual space image is an annotation associated with the real object.

4. The method according to claim 3, further comprising

obtaining information of location and orientation of the real object;

and

setting the second virtual image based on the information of location and orientation of the real object.

7. A program for making a computer implement respective steps of a mixed reality space image generation method of claim 1.

8. A computer-readable recording medium storing a program of claim 7.

5. A mixed reality space image generation apparatus for generating a mixed reality space image formed by superimposing virtual space images onto a real space image obtained by capturing a real space, comprising:

first image superimposing unit adapted to superimpose a first virtual space image onto the real space image based on an occlusion by a real object; and

second image superimposing unit adapted to superimpose a second virtual space image onto the superimposed image generated by said first image composition unit without considering the occlusion.

6. A mixed reality system using a mixed reality space image generation apparatus of claim 5.

9. A mixed reality space image generation method for generating a mixed reality space image formed by superimposing virtual space image onto a real space image obtained by capturing a real space, comprising:

obtaining information of location and orientation of an observer;

extracting an object region from the real space image;

generating a first virtual space image based on the information of location and orientation of an observer;

first superimposing the first virtual space image onto the real space image other than the object region; and

second superimposing a second virtual space image onto a region of the superimposed image generated in the first superimposing step based on the information of location and orientation of an observer.

10. The method according to claim 9, wherein the second virtual space image is an annotation.

11. The method according to claim 9, further comprising

obtaining information of location and orientation of a real object, and

wherein the first and second virtual images are generated based on the information of location and orientation of the real object and the information of location and orientation of the observer.

12. The method according to claim 9, wherein the second virtual space image is a text image and further comprising:

determining a region on which the second virtual space image is to be overwritten based on a relation between the first and second virtual space images, the information of location and orientation of the observer and the information of location and orientation of the real object; and

overwritten the text image on the region, determined by the determining step, of the superimposed image generated in the first superimposing step.