Image composition apparatus and method

• US 20040056870A1
• Filed: 09/10/2003
• Published: 03/25/2004
• Est. Priority Date: 03/13/2001
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morita, Kenji, Shimoyama, Tomohiko, Yonezawa, Hiroki
• Granted Patent: US 7,365,706 B2
• Time in Patent Office: Days
• US Class Current: 345/629
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/765 Interface circuits between an apparatus for recording and another apparatus
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/781 on disks or drums

Abstract

An image composition system includes an HMD (100) having a right-eye camera (110), a left-eye camera (111), a right-eye LCD (130) and a left-eye LCD (131) for displaying a real image, and the like, and an information processing apparatus (300) for generating another image different from the real image. A composite image obtained by superimposing the other image generated by the information processing apparatus (300) on the real image captured by the right-eye camera (110) and left-eye camera (111) is displayed on the right-eye LCD (130) and left-eye LCD (131). The display region of the other image is determined based on the position and posture of the head of the user who wears the HMD (100). The user can observe the other image superimposed on the real image at an appropriate position while wearing the HMD on his or her head.

Claims

1. An image composition system for compositing a real image in a line-of-sight direction of a user with another image, comprising:

a display unit which is wearable on a head of the user, and displays a composite image;

a position sensor for detecting the line-of-sight direction of the user, and outputting line-of-sight information;

a determination unit for determining a display region where the other image is to be displayed, in accordance with the line-of-sight information; and

a composition unit for compositing the other image on the determined display region,

wherein the other image is used to display information that helps operations of the user.

2. The system according to claim 1, wherein said display unit has an optical see-through structure, and the user can observe a real space via said display unit.

3. The system according to claim 1, further comprising:

a first image taking device for obtaining a video of a real space observed from a viewpoint of the user, and

wherein said composition unit displays the video obtained by said first image taking device on said display unit, and superimposes the other image on the display region determined by said determination unit.

4. The system according to claim 1, wherein the other image information is a video obtained by a second image taking device for taking an image from a viewpoint other than a viewpoint of the user.

5. The system according to claim 1, wherein the other image information is text information.

6. The system according to claim 5, further comprising:

a memory for holding a pair of the text information and time information indicating a display timing of the text information, and

wherein said composition unit switches the text information to be displayed on the display region in accordance with the time information held by said memory.

7. The system according to claim 1, wherein said determination unit comprises:

a setting unit for setting a space region for displaying the other image in the real space; and

a conversion unit for converting the space region set by said setting unit into the display region on said display unit on the basis of a position and posture of the user.

8. The system according to claim 1, further comprising:

a gesture detection unit capable of detecting a predetermined action of the user, and

wherein said composition unit controls to turn on/off display of the other image in response to a predetermined action detected by said gesture detection unit.

9. The system according to claim 8, wherein said composition unit switches contents of the other image to be displayed on the display region in response to a predetermined action detected by said gesture detection unit.

10. The system according to claim 1, wherein the information that helps the operations of the user is dialog information.

11. The system according to claim 1, wherein the information that helps the operations of the user is an image obtained by taking an image of an action of the user.

12. An information processing method of displaying a composite image of a real image in a line-of-sight direction of a user and another image on a display unit which is wearable on a head of the user, comprising the steps of:

detecting the line-of-sight direction of the user to acquire line-of-sight information;

determining a display region where the other image is to be displayed, in accordance with the line-of-sight information; and

compositing the other image on the determined display region,

wherein the other image is used to display information that helps operations of the user.

13. The method according to claim 12, wherein the other image information is a video obtained by image taking means for taking an image from a viewpoint other than a viewpoint of the user.

14. The method according to claim 12, wherein the information that helps the operations of the user is word information.

15. A computer readable medium that stores a program for making a computer execute an information processing method of displaying a composite image of a real image in a line-of-sight direction of a user and another image on a display unit which is wearable on a head of the user, said control program making the computer execute a process for:

detecting the line-of-sight direction of the user to acquire line-of-sight information;

determining a display region where the other image is to be displayed, in accordance with the line-of-sight information; and

compositing the other image on the determined display region.