Image processing method and image processing apparatus

• US 7,728,852 B2
• Filed: 03/24/2005
• Issued: 06/01/2010
• Est. Priority Date: 03/31/2004
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Suzuki, Masahiro, Morita, Kenji
• Primary Examiner(s): SAMS, MICHELLE L
• Application Number:
  US11/087,499
• Publication Number:
  US 20050231532A1
• Time in Patent Office: 1,895 Days
• Field of Search: 345/633, 345/629, 345/630, 345/632
• US Class Current: 345/632
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/03542 Light pens for emitting or receiving light
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06T 15/405 using Z-buffer
• CPC Class Codes: G06T 15/503 Blending, e.g. for anti-aliasing
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/62 Semi-transparency

Abstract

A virtual object and stylus model as images to be composited to a real space image are rendered by changing background transparencies according to a mutual positional relationship. When the virtual object image is composited to the real space image, the image of the stylus included in the real space image is observed while reflecting the positional relationship with the virtual object. In this way, in an MR image, the depth ordering between real and virtual objects can be correctly and easily expressed.

Claims

1. An image processing method for generating a mixed reality image by compositing an image of a virtual object having position and orientation information to a real space image and displaying the mixed reality image on an image plane of a display unit, comprising the steps of:

acquiring, by a microprocessor, viewpoint position and orientation information of a user;

detecting, by the microprocessor, position and orientation information of a predetermined real object;

detecting, by the microprocessor, a positional relationship between the virtual object and the predetermined real object;

generating, by the microprocessor, a first image by rendering the virtual object as a translucent object and the predetermined real object according to the positional relationship;

generating, by the microprocessor, a second image by superimposing the first image on the real space image on the basis of the viewpoint position and orientation information;

generating, by the microprocessor, a third image by rendering the predetermined real object as a transparent object which is positioned in front of the virtual object and is not intersecting with the virtual object and rendering the virtual object as an opaque object according to the positional relationship; and

generating, by the microprocessor, the mixed reality image by superimposing the third image on the second image;

wherein in the step of generating the first image, if the predetermined real object intersects with the virtual object, the microprocessor renders the intersecting part of the predetermined real object as the virtual object.

2. The method according to claim 1, wherein the predetermined real object is operated by the user.

4. A computer storage medium containing instructions that implements the image processing method of claim 1.

3. An image processing apparatus for generating a mixed reality image by compositing an image of a virtual object having position and orientation information to a real space image and displaying the mixed reality image on an image plane of a display unit, comprising:

a user position and orientation information acquisition unit adapted to acquire viewpoint position and orientation information of a user;

a real object position and orientation information acquisition unit adapted to detect position and orientation information of a predetermined real object;

a positional relationship acquisition unit adapted to detect a positional relationship between the virtual object and the predetermined real object; and

a microprocessor adapted, (i) to generate a first image by rendering the the virtual object as a translucent object and the predetermined real object according to the positional relationship, (ii) to generate a second image by superimposing the first image on the real space image on the basis of the viewpoint position and orientation information, (iii) to generate a third image by rendering the predetermined real object as a transparent object which is positioned in front of the virtual object and is not intersecting with the virtual object and rendering the virtual object as an opaque object according to the positional relationship, and (iv) to generate the mixed reality image by superimposing the third image on the second image;

wherein, when generating the first image, if the predetermined real object intersects with the virtual object, the microprocessor renders the intersecting part of the predetermined real object as the virtual object.