Video combining apparatus and method

• US 20040070611A1
• Filed: 09/29/2003
• Published: 04/15/2004
• Est. Priority Date: 09/30/2002
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Tanaka, Rika, Tomite, Kaname
• Granted Patent: US 7,487,468 B2
• Time in Patent Office: Days
• US Class Current: 345/757
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/42222 Additional components integrated in the remote control device, e.g. timer, speaker, sensors for detecting position, direction or movement of the remote control, microphone or battery charging device
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/42653 for processing graphics
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/44504 Circuit details of the additional information generator, e.g. details of the character or graphics signal generator, overlay mixing circuits

Abstract

A video combining apparatus to superimpose a virtual image such a CG image on a video image of the real world or on a see-through type display device. An area in which the virtual image is not to be displayed can be easily designated by a user. If the user holds a frame with markers in his/her sight, the frame is image-sensed in the video image of the real world. The area designated by the user is detected by detecting the position of the marker in the video image, and the virtual image is not superimposed in this area.

Claims

1. A video combining method for superimposing a virtual image generated by a computer on the real world observed by a user, comprising the steps of:

inputting an image obtained by image sensing the real world;

inputting position and orientation information of the user'"'"'s view point;

generating a virtual image based on said position and orientation information;

extracting a virtual image elimination area of said virtual image; and

combining said virtual image with said image obtained by image sensing based on said virtual image elimination area.

2. A video combining apparatus for superimposing a virtual image generated by a computer on the real world observed by a user, comprising:

image input unit adapted to input an image obtained by image sensing the real world;

position and orientation information input unit adapted to input position and orientation information of the user'"'"'s view point;

virtual image generation unit adapted to generate a virtual image based on said position and orientation information;

elimination area extraction unit adapted to extract a virtual image elimination area of said virtual image; and

combining unit adapted to combine said virtual image with said image obtained by image sensing based on said virtual image elimination area.

3. A video combining method for superimposing a virtual image on a video image of the real world observed by a user, comprising:

an image input step of inputting a video image of the real world observed by the user;

a position and orientation information input step of inputting position and orientation information of the user'"'"'s view point;

a virtual image generation step of generating a virtual image based on said position and orientation information;

a designated area detection step of detecting a predetermined area designated by the user; and

a superimposition step of superimposing said virtual image on said video image except a portion corresponding to the area in the video image detected at said designated area detection step.

4. The video combining method according to claim 3, wherein at said designated area detection step, the area in the video image is detected from the video image obtained by image sensing.

5. The video combining method according to claim 4, wherein at said designated area detection step, a marker provided in area designation means operated by the user, included in the video image obtained by image sensing, is detected, and the area in the video image is detected based on the position of the marker.

6. The video combining method according to claim 3, wherein at said designated area detection step, the area in the video image is detected based on position and orientation information of area designation means operated by the user.

7. The video combining method according to claim 3, wherein at said designated area detection step, the area in the video image is detected based on information on an area surrounded with a particular color.

8. The video combining method according to claim 3, wherein at said designated area detection step, the area in the video image is detected based on information on a closed area formed by the user'"'"'s hand.

9. A computer-readable medium holding program code to realize a video combining method for superimposing a virtual image generated by a computer on the real world observed by a user, by a computer, comprising:

process procedure code for inputting an image of the real world obtained by image sensing;

process procedure code for inputting position and orientation information of the user'"'"'s view point;

process procedure code for generating a virtual image based on the position and orientation information;

process procedure code for extracting a virtual image elimination area; and

process procedure code for combining said virtual image with said image obtained by image sensing based on the information on said virtual image elimination area.

10. A computer-readable medium holding program code to realize a video combining method for superimposing a virtual image on a video image of the real world observed by a user, by a computer, comprising:

process procedure code for inputting a video image of the real world observed by the user obtained by image sensing;

process procedure code for inputting position and orientation information of the user'"'"'s view point;

process procedure code for generating a virtual image based on said position and orientation information;

process procedure code for detecting a predetermined area designated by the user; and

process procedure code for superimposing said virtual image on said video image obtained by image sensing except a portion corresponding to the area in the video image detected at said detection process.