Image processing apparatus and image processing method

• US 20040109009A1
• Filed: 10/15/2003
• Published: 06/10/2004
• Est. Priority Date: 10/16/2002
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Morita, Kenji, Yonezawa, Hiroki
• Granted Patent: US 7,427,996 B2
• Time in Patent Office: Days
• US Class Current: 345/632
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A graphic card (130) renders a virtual space image on a memory (122) according to the position and orientation of a video camera (140) using data indicating the state of a virtual space, 3DCG data loaded from an HDD (123) onto the memory (122), and data indicating the position and orientation of the video camera (140). A real space image captured by the video camera (140) is then acquired. The graphic card (130) superimposes the acquired real space image on the virtual space image rendered on the memory (122) to generate a mixed reality space image on the memory (122), and then outputs the mixed reality space image to a monitor (150). Upon depression of a shutter (160), the mixed reality space image rendered on the memory (122) at that time is output to a printer (170), which prints the mixed reality space image on a print medium such as a paper sheet, OHP film, or the like.

Claims

1. An image processing apparatus which is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

image sensing unit adapted to capture an image of a real space;

generation unit adapted to generate an image of a virtual space in accordance with a position and orientation of said image sensing unit, which serves as a viewpoint of the virtual space, on a coordinate system shared by the real space and the virtual space;

storage unit adapted to superpose and storing the image of the real space captured by said image sensing unit and the image of the virtual space generated by said generation unit to store a mixed reality space image formed by superposing the two images;

instruction unit adapted to instruct whether or not the mixed reality space image stored in said storage unit is to be printed; and

output unit adapted to output the mixed reality space image stored in said storage unit to the print unit upon reception of a print instruction of the mixed reality image from said instruction unit.

2. The apparatus according to claim 1, further comprising management unit adapted to manage state information of the virtual space, and in that said management unit updates the state information of the virtual space in accordance with input information or at each predetermined timing, and said generation unit generates a virtual space image according to the updated state information of the virtual space.

3. The apparatus according to claim 2, wherein said management unit is an independent apparatus connected to said image processing apparatus via a network, and the state information of the virtual space updated by said management unit can be received by image processing apparatuses which are equivalent to said image processing apparatus, and are connected to the network.

4. The apparatus according to claim 1, further comprising display unit adapted to display the mixed reality space image stored in said storage unit.

5. The apparatus according to claim 1, wherein said image sensing unit is an HMD which comprises a display unit for displaying the mixed reality space image stored in said storage unit,

said apparatus further comprises a sensor for measuring a position and orientation near a viewpoint position of an observer who wears the HMD on the coordinate system, and

said generation unit calculates based on the position and orientation near the viewpoint position on the coordinate system as measurement results of said sensor, and a constant which is measured in advance.

6. The apparatus according to claim 5, wherein the viewpoint includes a first viewpoint and a second viewpoint different from the first viewpoint, and

said generation unit generates an image of the virtual space according to a position and orientation of the first viewpoint on the coordinate system using the position and orientation near the viewpoint position on the coordinate system as the position and orientation of the first viewpoint, and generates an image of the virtual space according to a position and orientation of the second viewpoint on the coordinate system using a position which is shifted a predetermined distance from the position of the first viewpoint and the orientation of the first viewpoint as the position and orientation of the second viewpoint.

7. The apparatus according to claim 1, wherein said image sensing unit is an image sensing device, which comprises a display unit for displaying the mixed reality space image stored in said storage unit, and is attached to a fixed post to be rotatable in a desired direction,

said apparatus further comprises a sensor for measuring a orientation of the attached portion of the image sensing device on the coordinate system, and

said generation unit uses a measurement result of said sensor as a orientation of the viewpoint.

14. A computer program making a computer function as an image processing apparatus of claim 1.

16. A computer readable storage medium storing a computer program of claim 14.

8. An image processing apparatus, which is connected to a plurality of apparatuses each of which generates a mixed reality space image formed by superimposing images of a virtual space and real space, is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

selection unit adapted to select a desired one of the plurality of apparatuses;

instruction unit adapted to instruct whether or not the mixed reality space image obtained from the apparatus selected by said selection unit is to be printed; and

output unit adapted to output the mixed reality space image to the print unit upon reception of a print instruction of the mixed reality image from said instruction unit.

9. The apparatus according to claim 8, further comprising generation unit adapted to generate one image which includes the mixed reality space image obtained from the apparatus selected by said selection unit, and in that said output unit outputs the one image to the print unit upon reception of the print instruction from said instruction unit.

10. The apparatus according to claim 9, wherein when a plurality of mixed reality space images are obtained from the apparatus selected by said selection unit, said generation unit reduces the mixed reality space images and generates one image which includes the reduced mixed reality space images.

11. The apparatus according to claim 9, further comprising display unit adapted to display the one image generated by said generation unit.

12. An image processing method to be executed by an image processing apparatus which is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

an image sensing step of capturing an image of a real space using image sensing unit;

a generation step of generating an image of a virtual space in accordance with a position and orientation of the image sensing unit, which serves as, a viewpoint of the virtual space, on a coordinate system shared by the real space and the virtual space;

a storage step of superposing and storing the image of the real space captured in the image sensing step and the image of the virtual space generated in the generation step to store a mixed reality space image formed by superposing the two images in a memory;

an instruction step of instructing whether or not the mixed reality space image stored in the memory is to be printed; and

an output step of outputting the mixed reality space image stored in the memory to the print unit upon reception of a print instruction of the mixed reality image from the instruction step.

15. A computer program making a computer execute an image processing method of claim 12.

17. A computer readable storage medium storing a computer program of claim 15.

13. An image processing method to be executed by an image processing apparatus, which is connected to a plurality of apparatuses each of which generates a mixed reality space image formed by superimposing images of a virtual space and real space, is connected to print unit for printing an image, and can output an image to be printed to the print unit, comprising:

a selection step of selecting a desired one of the plurality of apparatuses;

an instruction step of instructing whether or not the mixed reality space image obtained from the apparatus selected in the selection step is to be printed; and

an output step of outputting the mixed reality space image to the print unit upon reception of a print instruction of the mixed reality image from the instruction step.

18. An image processing apparatus which is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

image sensing unit adapted to capture a real space image;

generation unit adapted to generate a virtual space image according to a viewpoint position and orientation on a coordinate system shared by real and virtual spaces to have said image sensing unit as a viewpoint on the virtual space;

composition unit adapted to superimpose the real space image captured by said image sensing unit and the virtual space image generated by said generation unit;

display unit adapted to display a mixed reality space image obtained by said composition unit; and

instruction unit adapted to give, to the print unit, an output instruction of the mixed reality space image composited by said composition unit, and

in that said generation unit has a plurality of calculation methods and sequences for generating the virtual space image, and

when the instruction is received from said instruction unit, said generation unit re-generates a virtual space image using a second calculation method and sequence, which are different from a first calculation method and sequence used to generate the virtual space image, said composition unit re-composites a mixed reality space image from the re-generated virtual space image and the real space image, and the print unit prints the re-composited mixed reality space image on a medium.

20. The apparatus according to claim 18, wherein a pixel size of one side of the virtual space image generated by said generation unit assumes an arbitrary value which has, as an upper limit, a product of a length of one side and a resolution of an image that the print unit can output.

21. The apparatus according to claim 18, further comprising management unit adapted to manage state information of the real and virtual spaces, and in that said management unit updates the state information of the virtual space in accordance with input information or at each predetermined timing, and said generation unit successively generates virtual space images in accordance with the updated state information of the real and virtual spaces.

22. The apparatus according to claim 21, wherein said management unit is an independent apparatus connected to said image processing apparatus via a network, and the state information of the virtual space updated by said management unit can be received by image processing apparatuses which are equivalent to said image processing apparatus, and are connected to the network.

23. The apparatus according to claim 18, wherein said apparatus can be simultaneously used by at least one user, and has said image sensing unit, said generation unit, said composition unit, said display unit, and detection unit adapted to detect a viewpoint position and visual axis direction of a user, which are prepared for each user.

24. The apparatus according to claim 23, wherein said display unit, or said display unit and said image sensing unit, forms or form a head-mounted display device to be mounted on a head of the user.

25. A computer program making a computer function as an image processing apparatus of claim 18.

26. A computer readable storage medium storing a computer program of claim 25.

19. An image processing apparatus which is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

image sensing unit adapted to capture a real space image;

generation unit adapted to generate a virtual space image according to a viewpoint position and orientation on a coordinate system shared by real and virtual spaces to have said image sensing unit as a viewpoint on the virtual space;

composition unit adapted to superimpose the real space image captured by said image sensing unit and the virtual space image generated by said generation unit;

display unit adapted to display a mixed reality space image obtained by said composition unit; and

instruction unit adapted to give, to the print unit, an output instruction of the mixed reality space image composited by said composition unit, and

in that said generation unit has a plurality of data sets for generating the virtual space image, and

when the instruction is received from said instruction unit, said generation unit re-generates a virtual space image using a second data set, which is different from a first data set used to generate the virtual space image, said composition unit re-composites a mixed reality space image from the re-generated virtual space image and the real space image, and the print unit prints the re-composited mixed reality space image on a medium.

27. An image processing apparatus which is connected to print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

image sensing unit adapted to capture a real space image;

generation unit adapted to generate a virtual space image according to a viewpoint position and orientation on a coordinate system shared by real and virtual spaces to have said image sensing unit as a viewpoint on the virtual space;

composition unit adapted to superimpose the real space image captured by said image sensing unit and the virtual space image generated by said generation unit;

display unit adapted to display a mixed reality space image obtained by said composition unit;

instruction unit adapted to give, to the print unit, an output instruction of the mixed reality space image composited by said composition unit; and

determination unit adapted to check a positional relationship among said image sensing unit, a real space object, and a virtual space object on the coordinate system, and determining whether or not the real space object is occluded by the virtual space object in the mixed reality space image, and

in that said generation unit re-generates a virtual space image using a second calculation method and sequence, which are different from a first calculation method and sequence used to generate the virtual space image, in accordance with a result of said determination unit, said composition unit re-composites a mixed reality space image from the re-generated virtual space image and the real space image, and the print unit prints the re-composited mixed reality space image on a medium.

28. The apparatus according to claim 27, further comprising management unit adapted to manage state information of the real and virtual spaces, and in that said management unit updates the update information of the virtual space in accordance with input information or at each predetermined timing, and said generation unit successively generates virtual space images in accordance with the updated state information of the real and virtual spaces.

29. The apparatus according to claim 28, wherein said management unit is an independent apparatus connected to said image processing apparatus via a network, and the state information of the virtual space updated by said management unit can be received by image processing apparatuses which are equivalent to said image processing apparatus, and are connected to the network.

30. The apparatus according to claim 27, wherein said apparatus can be simultaneously used by at least one observer, and has said image sensing unit, said generation unit, said composition unit, said display unit, and detection unit adapted to detect a viewpoint position and visual axis direction of an observer, which are prepared for each observer.

31. The apparatus according to claim 30, wherein said display unit, or said display unit and said image sensing unit, forms or form a head-mounted display device to be mounted on a head of the user.

32. A computer program making a computer function as an image processing apparatus of claim 27.

33. A computer readable storage medium storing a computer program of claim 32.