Image processing method and image processing apparatus

• US 20050231532A1
• Filed: 03/24/2005
• Published: 10/20/2005
• Est. Priority Date: 03/31/2004
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Suzuki, Masahiro, Morita, Kenji
• Granted Patent: US 7,728,852 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/62 Semi-transparency

Abstract

A virtual object (402) and stylus model (403) as images to be composited to a real space image are rendered by changing background transparencies according to a mutual positional relationship. When the virtual object image is composited to the real space image, the image of the stylus included in the real space image is observed while reflecting the positional relationship with the virtual object. In this way, in an MR image, the depth ordering between real and virtual objects can be correctly and easily expressed.

Claims

1. An image processing method for generating a mixed reality image by compositing an image of a virtual object having position and orientation information to a real space image, comprising:

a user position and orientation information acquisition step of acquiring viewpoint position and orientation information of a user;

a real object position and orientation information acquisition step of detecting position and orientation information of a predetermined real object;

a positional relationship acquisition step of detecting a positional relationship between the virtual object and a model on the basis of the viewpoint position and orientation information, information of the model of the predetermined real object, the position and orientation information of the real object, and position and orientation information of the virtual object;

a rendering step of rendering an image of the virtual object and the model of the predetermined real object using transparencies according to the positional relationship; and

a composition step of compositing the real space image and the image generated in the rendering step.

2. The method according to claim 1, wherein the predetermined real object is operated by the user.

3. The method according to claim 1, wherein the rendering step includes a step of rendering the model as a transparent object, and setting a background transparency used upon rendering an intersecting region of the virtual object with the model to be higher than a non-intersecting region with the model and to be a value less than 100%.

4. The method according to claim 3, wherein the rendering step includes a step of setting a background transparency of the non-intersecting region of the virtual object with the model to be 0%.

5. The method according to claim 3, wherein the rendering step includes a step of changing the background transparency of the intersecting region of the virtual object with the model in a plurality of steps in accordance with intersecting states between the model and the virtual object.

6. The method according to claim 1, wherein the rendering step includes:

a step of rendering a first image so that the virtual object is rendered as a translucent object and the model is rendered as a transparent object; and

a step of rendering a second image so that the virtual object is rendered as an opaque object, the model is rendered as a transparent object, and the entire model is located in front of the virtual object, and

the composition step includes;

a step of generating the mixed reality image by compositing the first image onto the real space image, and further compositing the second image onto the composite image.

7. The method according to claim 1, further comprising:

a collision checking step of checking whether or not the model and the virtual object collide against each other,

wherein the rendering step includes steps of;

rendering, when it is determined that the model and the virtual object collide against each other, only the virtual object as a translucent object; and

rendering, when it is determined that the model and the virtual object do not collide against each other, the virtual object as an opaque object and the model as a transparent object.

8. The method according to claim 1, further comprising:

a collision checking step of checking whether or not the model and the virtual object collide against each other,

wherein the rendering step includes steps of;

dividing, when it is determined that the model and the virtual object collide against each other, the model by a collision surface with the virtual object, rendering at least a part of the model present in the virtual object as a colored object which is located in front of the virtual object, rendering a part of the model, which is located in front of the virtual model, as a transparent object, and rendering the virtual object as an opaque object; and

rendering, when it is determined that the model and the virtual object do not collide against each other, the virtual object as an opaque object and the model as a transparent object.

10. A program for making a computer implement an image processing method of claim 1.

11. A computer-readable recording medium storing a program of claim 10.

9. An image processing apparatus for generating a mixed reality image by compositing an image of a virtual object having position and orientation information to a real space image, comprising:

user position and orientation information acquisition unit adapted to acquire viewpoint position and orientation information of a user;

real object position and orientation information acquisition unit adapted to detect position and orientation information of a predetermined real object;

positional relationship acquisition unit adapted to detect a positional relationship between the virtual object and a model on the basis of the viewpoint position and orientation information, information of the model of the predetermined real object, the position and orientation information of the real object, and position and orientation information of the virtual object;

rendering unit adapted to render an image of the virtual object and the model of the predetermined real object using transparencies according to the positional relationship; and

composition unit adapted to composite the real space image and the image generated by said rendering unit.