Image composition apparatus and method

• US 7,365,706 B2
• Filed: 09/10/2003
• Issued: 04/29/2008
• Est. Priority Date: 03/13/2001
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Yonezawa, Hiroki, Morita, Kenji, Shimoyama, Tomohiko
• Primary Examiner(s): DINH, DUC Q
• Application Number:
  US10/658,335
• Publication Number:
  US 20040056870A1
• Time in Patent Office: 1,693 Days
• Field of Search: 345 7- 9, 345/633
• US Class Current: 345/8
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums

Abstract

An image composition system includes an HMD (100) having a right-eye camera (110), a left-eye camera (111), a right-eye LCD (130) and a left-eye LCD (131) for displaying a real image, and the like, and an information processing apparatus (300) for generating another image different from the real image. A composite image obtained by superimposing the other image generated by the information processing apparatus (300) on the real image captured by the right-eye camera (110) and left-eye camera (111) is displayed on the right-eye LCD (130) and left-eye LCD (131). The display region of the other image is determined based on the position and posture of the head of the user who wears the HMD (100). The user can observe the other image superimposed on the real image at an appropriate position while wearing the HMD on his or her head.

Claims

1. An image composition system for compositing a real image corresponding to a position of a display unit with an other image, said system comprising:

the display unit, which is wearable on a head of a user, and displays a composite image;

a position sensor for detecting the position of the display unit, and position information;

a determination unit for determining a display region where the other image is to be displayed, in accordance with position information; and

a composition unit for compositing the other image on the determined display region,

wherein the other image is used to display information that helps operations of the user, and

wherein said determination unit comprises;

(a) a setting unit for setting a space region for displaying the other image in the real space; and

(b) a conversion unit for converting the space region set by said setting unit into the display region on said display unit on the basis of a position and posture of the user.

2. The system according to claim 1, wherein said display unit has an optical see through structure, and the user can observe the real space via said display unit.

3. The system according to claim 1, further comprising:

a first image taking device for obtaining a video of the real space observed from a viewpoint of the user, and

wherein said composition unit displays the video obtained by said first image taking device on said display unit, and superimposes the other image on the display region determined by said determination unit.

4. The system according to claim 1, wherein the information is a video obtained by a second image taking device for taking an image from a viewpoint other than a viewpoint of the user.

5. The system according to claim 1, wherein the information is dialog information.

6. The system according to claim 1, wherein the information is an image obtained by taking an image of an action of the user.

7. An image composition system for compositing a real image corresponding to a position of a display unit with an other image, said system comprising:

the display unit, which is wearable on a head of a user, and displays a composite image;

a position sensor for detecting the position of the display unit, and outputting position information;

a determination unit for determining a display region where the other image is to be displayed, in accordance with the position information;

a composition unit for compositing the other image on the determined display region, wherein the other image is used to display information that helps operations of the user, and wherein the information is text information; and

a memory for holding a pair of the text information and time information indicating a display timing of the text information,

wherein said composition unit switches the text information to be displayed on the display region in accordance with the time information held by said memory.

8. An image composition system for compositing a real image corresponding to a position of a display unit with an other image, said system comprising:

the display unit, which is wearable on a head of a user, and displays a composite image;

a position sensor for detecting the position of the display unit, and outputting position information;

a determination unit for determining a display region where the other image is to be displayed, in accordance with the position information;

a composition unit for compositing the other image on the determined display region, wherein the other image is used to display information that helps operations of the user; and

a gesture detection unit capable of detecting a predetermined action of the user,

wherein said composition unit performs turn on/off display control of the other image in response to a predetermined action detected by said gesture detection unit.

9. The system according to claim 8, wherein said composition unit switches contents of the other image to be displayed on the display region in response to a predetermined action detected by said gesture detection unit.

10. An information processing method of displaying a composite image of a real image corresponding to a position of a display unit and an other image on the display unit, which is wearable on a head of a user, said method comprising the steps of:

detecting the position of the display unit, and outputting position information;

determining a display region where the other image is to be displayed, in accordance with the position information;

compositing the other image on the determined display region, wherein the other image is used to display information that helps operations of the user, and wherein the information is text information; and

holding in a memory a pair of the text information and time information indicating a display timing of the text information,

wherein said compositing step switches the text information to be displayed on the display region in accordance with the time information held by the memory.

11. A computer-readable medium that stores a computer-executable program for making a computer execute the information processing method according to claim 10.

12. An information processing method of displaying a composite image of a real image corresponding to a position of a display unit and an other image on the display unit, which is wearable on a head of a user, said method comprising the steps of:

detecting the position of the display unit, and position information;

determining a display region where the other image is to be displayed, in accordance with position information; and

compositing the other image on the determined display region, wherein the other image is used to display information that helps operations of the user, wherein said determining step comprises;

(a) a setting step of setting a space region for displaying the other image in a real space; and

(b) a converting step of converting the space region set by said setting step into the display region on the display unit on the basis of a position and posture of the user.

13. A computer-readable medium that stores a computer-executable program for making a computer execute the information processing method according to claim 12.

14. An information processing method of displaying a composite image of a real image corresponding to a position of a display unit and an other image on the display unit, which is wearable on a head of the user, said method comprising the steps of:

detecting the position of the display unit, and outputting position information;

determining a display region where the other image is to be displayed, in accordance with the position information;

compositing the other image on the determined display region, wherein the other image is used to display information that helps operations of the user; and

a gesture detecting step of detecting a predetermined action of the user,

wherein said compositing step performs turn on/off display control of the other image in response to a predetermined action detected by said gesture detecting step.

15. A computer-readable medium that stores a computer-executable program for making a computer execute the information processing method according to claim 14.