Mixed reality space image generation method and mixed reality system

• US 7,589,747 B2
• Filed: 09/29/2004
• Issued: 09/15/2009
• Est. Priority Date: 09/30/2003
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Suzuki, Masahiro, Matsui, Taichi
• Primary Examiner(s): HARRISON, CHANTE E
• Application Number:
  US10/951,684
• Publication Number:
  US 20050179617A1
• Time in Patent Office: 1,812 Days
• Field of Search: 345 7- 9, 345/629, 345/632, 345/633, 345/156, 345/157, 700/9, 700/83
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

A mixed reality space image generation method for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, includes a first image superimposing step of superimposing a first virtual space image onto a real space image based on an occlusion by a real object, wherein the first virtual space image is an outer appearance of the real object, and obtaining information of location and orientation of the real object. In addition, a second virtual image is set based on the information of location and orientation of the real object, and a second image superimposing step superimposes the second virtual space image onto the superimposed image generated in the first image superimposing step without considering the occlusion, wherein the second virtual space image is an annotation associated with the real object.

Claims

1. A mixed reality space image generation method for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, comprising:

obtaining information of location and orientation of an observer;

obtaining information of location and orientation of a real object;

extracting an object region from the real space image;

generating a first virtual space image based on the information of location and orientation of an observer;

first superimposing the first virtual space image onto the real space image other than the object region; and

second superimposing a second virtual space image onto the superimposed image generated in the first superimposing step based on the information of location and orientation of an observer,

wherein the first and second virtual images are generated based on the information of location and orientation of the real object and the information of location and orientation of the observer, and

wherein said steps are performed on a computer.

2. The method according to claim 1, wherein the second virtual space image is an annotation.

3. A mixed reality space image generation method for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, comprising:

obtaining information of location and orientation of an observer;

extracting an object region from the real space image;

generating a first virtual space image based on the information of location and orientation of an observer;

first superimposing the first virtual space image onto the real space image other than the object region; and

second superimposing a second virtual space image onto the superimposed image generated in the first superimposing step,

wherein the second virtual space image is a text image, and the second superimposing step determines a position of the superimposed image on which the second virtual space image is to be overwritten based on a relation between the first and second virtual space images, the information of location and orientation of the observer and the information of location and orientation of the real object, and superimposes the text image on the determined position of the superimposed image generated in the first superimposing step, and

wherein said steps are performed on a computer.

4. The method according to claim 3, wherein the second virtual space image is an annotation.

5. A mixed reality space image generation apparatus for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, comprising:

a first obtaining unit to obtain information of location and orientation of an observer;

a second obtaining unit to obtain information of location and orientation of a real object;

an extracting unit to extract an object region from the real space image;

a generation unit to generate a first virtual space image based on the information of location and orientation of an observer;

a first superimposing unit to superimpose the first virtual space image onto the real space image other than the object region; and

a second superimposing unit to superimpose a second virtual space image onto the superimposed image generated in the first superimposing step based on the information of location and orientation of an observer,

wherein the first and second virtual images are generated based on the information of location and orientation of the real object and the information of location and orientation of the observer.

6. The apparatus according to claim 5, wherein the second virtual space image is an annotation.

7. A mixed reality space generation apparatus for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, comprising:

an obtaining unit to obtain information of location and orientation of an observer;

an extracting unit to extract an object region from the real space image;

a generation unit to generate a first virtual space image based on the information of location and orientation of an observer;

a first superimposing unit to superimpose the first virtual space image onto the real space image other than the object region; and

a second superimposing unit to superimpose a second virtual space image onto a region of the superimposed image generated by the first superimposing unit,

wherein the second virtual space image is a text image, and the second superimposing unit determines a position of the superimposed image on which the second virtual space image is to be overwritten based on a relation between the first and second virtual space images, the information of location and orientation of the observer and the information of location and orientation of the real object, and superimposes the text image on the determined position of the superimposed image generated by the first superimposing unit.

8. The apparatus according to claim 7, wherein the second virtual space image is an annotation.

9. A computer-readable recording medium storing a program to implement a mixed reality space image generation method for generating a mixed reality space image formed by superimposing a virtual space image onto a real space image obtained by capturing a real space, comprising:

obtaining information of location and orientation of an observer;

obtaining information of location and orientation of a real object;

extracting an object region from the real space image;

generating a first virtual space image based on the information of location and orientation of an observer;

first superimposing the first virtual space image onto the real space image other than the object region; and

second superimposing a second virtual space image onto the superimposed image generated in the first superimposing step based on the information of location and orientation of an observer,

wherein the first and second virtual images are generated based on the information of location and orientation of the real object and the information of location and orientation of the observer, and

wherein said steps are performed on a computer.

10. A computer-readable recording medium storing a program to implement a mixed reality space generation method for generating a mixed reality space image formed by superimposing virtual space image onto a real space image obtained by capturing a real space, comprising:

obtaining information of location and orientation of an observer;

extracting an object region from the real space image;

generating a first virtual space image based on the information of location and orientation of an observer;

first superimposing the first virtual space image onto the real space image other than the object region; and

second superimposing a second virtual space image onto the superimposed image generated in the first superimposing step,

wherein the second virtual space image is a text image, and the second superimposing step determines a position of the superimposed image on which the second virtual space image is to be overwritten based on a relation between the first and second virtual space images, the information of location and orientation of the observer and the information of location and orientation of the real object, and superimposes the text image on the determined position of the superimposed image generated in the first superimposing step.