GENERATING IMAGES COMBINING REAL AND VIRTUAL IMAGES

• US 20080024523A1
• Filed: 07/23/2007
• Published: 01/31/2008
• Est. Priority Date: 07/27/2006
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Tomite, Kaname
• Granted Patent: US 8,933,965 B2
• Time in Patent Office: Days
• US Class Current: 345/632
• CPC Class Codes: G06T 15/50 Lighting effects
• CPC Class Codes: G06T 15/60 Shadow generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method for combining a real space image with a virtual image, includes: causing an imaging unit to capture an image of a real space; generating an image covering a predetermined space based on a plurality of real images in the captured real space; extracting position information of a light source based on the generated image; and adding a light source or a shadow on the virtual image based on the extracted position information of the light source.

Claims

1. A method of presenting a combined image in which an image of a real space is combined with a virtual image of a virtual object, the method comprising:

receiving image data for a plurality of captured images, the plurality of captured images including images relating to different respective parts of the real space;

generating modeling image data, representing an image covering at least a predetermined portion of the real space, based on the received image data for the plurality of captured images;

estimating light source information relating to the real space based on the modeling image data;

generating virtual image data, representing the virtual image, including managing light source information relating to the virtual object based on the estimated light source information relating to the real space; and

updating the modeling image data following receipt of image data for a new captured image relating to such a part of the real space, and updating the estimated light source information relating to the real space based on the updated modeling image data, and updating the light source information relating to the virtual object based on the updated light source information relating to the real space, and regenerating the virtual image data.

2. The method according to claim 1, further comprising generating the combined image data, representing the combined image, by generating the virtual image data, and combining real image data, representing an image of a desired part of the real space, with the virtual image data.

3. The method according to claim 1, further including employing a model having a predetermined modeling shape to model the predetermined portion of the real space, the modeling image data representing the plurality of captured images as projected onto the model.

4. The method according to claim 1, further comprising:

employing an image capturing apparatus to obtain the plurality of captured images;

obtaining position and orientation data representing positions and orientations of the image capturing apparatus when the plurality of images are being captured; and

employing the obtained position and orientation data together with the received image data for the plurality of captured images to generate the modeling image data.

5. The method according to claim 4, wherein some or all of the position and orientation data is obtained by a position/orientation sensor connected with the image capturing apparatus.

6. The method according to claim 4, wherein some or all of the position and orientation data is obtained by arranging a marker, having predetermined features, at a reference position in the real space and processing the received image data for the captured images to search for a region in a captured image having features corresponding to the predetermined features of the marker.

7. The method according to claim 1, further comprising:

employing a plurality of image capturing apparatuses, corresponding respectively to different users, to obtain the plurality of captured images; and

for each user, generating such combined image data, representing such a combined image for the user concerned, the light source information relating to the virtual object being used in common for generating the combined image data for all of the plurality of users.

8. The method according to claim 1, wherein a change in the estimated light source information is effected gradually as a series of steps having a predetermined step size.

9. The method according to claim 1, wherein the light source information relating to the virtual object has a limited number of virtual light sources, and the method further comprises:

assigning a priority to each of a plurality of highlight regions in the real space; and

employing the assigned priorities and positions of the highlight regions to set the light source information for each of the limited number of virtual light sources.

10. A computer program which, when executed on a computer or processor, causes the computer or processor to carry out the method of claim 1.

11. An apparatus for presenting a combined image in which an image of a real space is combined with a virtual image of a virtual object, the apparatus comprising:

receiving means for receiving image data for a plurality of captured images, the plurality of captured images including images relating to different respective parts of the real space;

modeling image data generating means for generating modeling image data, representing an image covering at least a predetermined portion of the real space, based on the received image data for the plurality of captured images;

light source information estimating means for estimating light source information relating to the real space based on the modeling image data;

image data generating means for generating virtual image data, representing the virtual image, including managing light source information relating to the virtual object based on the estimated light source information relating to the real space; and

updating means operable to update the modeling image data following receipt of image data for a new captured image relating to such a part of the real space, and to update the estimated light source information relating to the real space based on the updated modeling image data, and to update the light source information relating to the virtual object based on the updated light source information relating to the real space, and to regenerate the virtual image data.

12. A method for combining a real space image with a virtual image, comprising:

inputting an image of a real space captured by an imaging unit;

generating an image covering a predetermined space based on a plurality of real images in the captured real space input;

extracting position information of a light source based on the generated image; and

adding a light source or a shadow on the virtual image based on the extracted position information of the light source.

13. The method according to claim 12, further comprising:

measuring the position and orientation of the imaging unit; and

generating an image covering the predetermined space based on the measured position and orientation.

14. The method according to claim 13, further comprising:

generating a virtual image based on the measured position and orientation; and

adding a shadow to the generated virtual image.

15. A method for generating a combined image of a real space image and a virtual image, comprising:

obtaining an image of a real space captured by an imaging apparatus;

obtaining position and orientation information of the imaging apparatus;

calculating light source information of the real space based on the captured image and the position and orientation information;

updating light source information stored in a storage unit based on the calculated light source information; and

generating a virtual image based on the stored light source information and the position and orientation information.

16. The method according to claim 15, further comprising adding the calculated light source information to the stored light source information.

17. The method according to claim 15, further comprising generating a virtual image reflecting a light source indicated by the stored light source information based on the stored light source information and the position and orientation information.

18. The method according to claim 15, further comprising:

obtaining images of the real space captured by a plurality of imaging units;

estimating light source information based on the captured images;

updating the stored light source information based on the estimated light source information; and

generating virtual images corresponding to the position and orientation information of the plurality of imaging units based on the updated light source information.

19. The method according to claim 15, further comprising:

obtaining geometric information of the real space; and

generating the light source information based on the captured image, the position and orientation information, and the geometric information.

20. The method according to claim 15, further comprising:

generating updated light source information based on the stored light source information and the calculated light source information; and

updating the stored light source information based on the generated updated light source information.

21. The method according to claim 20, further comprising:

generating the updated light source information by applying weighted calculations to the stored light source information and the calculated light source information.

22. An apparatus for combining a real space image with a virtual image, comprising:

an input unit which inputs an image of a real space captured by an imaging unit;

a generating unit which generates an image covering a predetermined space based on a plurality of real images in the captured real space inputted by the input unit;

an extracting unit which extracts position information of a light source based on the generated image; and

an adding unit which adds a light source or a shadow on the virtual image based on the extracted position information of the light source.

23. An apparatus for generating a combined image of a real space image and a virtual image, comprising:

an image obtaining unit which obtains an image of a real space captured by an imaging apparatus;

a position/orientation information obtaining unit which obtains position and orientation information of the imaging apparatus;

a calculating unit which calculates light source information of the real space based on the captured image and the position and orientation information;

an updating unit which updates light source information stored in a storage unit based on the calculated light source information; and

a generating unit which generates a virtual image based on the stored light source information and the position and orientation information.