Image processing apparatus, method and computer-readable storage medium for generating and presenting an image of virtual objects including the operation panel viewed from the position and orientation of the viewpoint of the observer

• US 7,557,816 B2
• Filed: 10/25/2005
• Issued: 07/07/2009
• Est. Priority Date: 10/28/2004
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kuroki, Tsuyoshi
• Primary Examiner(s): Brier; Jeffery A
• Application Number:
  US11/257,838
• Publication Number:
  US 20060092131A1
• Time in Patent Office: 1,351 Days
• Field of Search: 345/632, 345/633
• US Class Current: 345/633
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A technology for enabling an observer to appropriately control an operation panel in accordance with a condition is provided. When an instruction for changing the position of the operation panel is input, the operation panel is disposed at the position of a pen input device. When an instruction for terminating the changing of the position of the operation panel is input, the position and orientation of the operation panel are fixed. An image of virtual objects including the operation panel viewed from a camera is generated, the generated image is superimposed on an image in a real space captured by the camera, and an image obtained by the superimposition is output to a display unit.

Claims

1. An image processing method for controlling an image processing apparatus using a display unit and presenting an image in a virtual space to an observer by displaying an image of virtual objects on a screen of the display unit, the image processing method comprising:

a first input step of inputting a position and an orientation of a viewpoint of the observer;

a second input step of inputting a position of a designation tool used by the observer;

a disposing step of disposing an operation panel on a plane determined based on the position and the orientation of the viewpoint of the observer and the position of the designation tool;

a generation step of generating an image of virtual objects including the operation panel viewed from the position and orientation of the viewpoint of the observer input in the first input step; and

a displaying step of presenting the image of virtual objects including the operation panel to the observer by displaying the image of virtual objects on the screen of the display unit.

2. The image processing method according to claim 1, further comprising:

a third input step of inputting an image in a real space viewed from the viewpoint of the observer; and

an output step of generating a mixed image by superimposing the image generated by the generation step on the image input by the third input step and outputting the mixed image.

3. The image processing method according to claim 1, wherein the predetermined plane region is a face region of a tablet.

4. The image processing method according to claim 1, wherein, when the position of the operation panel disposed in the disposing step is located within a predetermined plane region, the operation panel is disposed in a coordinate system defined by a position and an orientation of the predetermined plane region.

5. The image processing method according to claim 1, wherein, when a distance between the position of the operation panel disposed in the disposing step and the position of the viewpoint input in the first input step is equal to or shorter than a predetermined distance, the operation panel is disposed in a coordinate system defined by the position and the orientation of the viewpoint input in the first input step.

6. An image processing apparatus using a display unit, for presenting an image in a virtual space to an observer by displaying an image of virtual objects on a screen of a display unit, the image processing apparatus comprising:

a first input unit configured to input a position and an orientation of a viewpoint of the observer;

a second input unit configured to input a position of a designation tool used by the observer;

a disposing unit configured to dispose an operation panel on a plane determined based on the position and the orientation of the viewpoint of the observer and the position of the designation tool;

a generation unit configured to generate an image of virtual objects including the operation panel viewed from the position and orientation of the viewpoint of the observer input via the first input unit; and

a displaying unit configured to present the image of virtual objects including the operation panel to the observer by displaying the image of virtual objects on the screen of the display unit.

7. A computer-readable storage medium having computer-executable instructions stored thereon for causing an image processing apparatus using a display unit to perform an image processing method for presenting an image in a virtual space to an observer by displaying an image of virtual objects on a screen of the display unit comprising:

a first input step of inputting a position and an orientation of a viewpoint of the observer;

a second input step of inputting a position of a designation tool used by the observer;

a disposing step of disposing an operation panel on a plane determined based on the position and the orientation of the viewpoint of the observer and the position of the designation tool;

a generation step of generating an image of virtual objects including the operation panel viewed from the position and orientation of the viewpoint of the observer input in the first input step; and

a display step of presenting the image of virtual objects including the operation panel to the observer by displaying the image of virtual objects on the screen of the display unit.