Image processing apparatus and image processing method

• US 7,427,996 B2
• Filed: 10/15/2003
• Issued: 09/23/2008
• Est. Priority Date: 10/16/2002
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ohshima, Toshikazu, Morita, Kenji, Yonezawa, Hiroki
• Primary Examiner(s): WANG, JIN CHENG
• Application Number:
  US10/684,401
• Publication Number:
  US 20040109009A1
• Time in Patent Office: 1,805 Days
• Field of Search: 345629-633
• US Class Current: 345/629
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An image processing apparatus is connected to a print unit adapted to print an image, and can output an image to be printed to the print unit. The apparatus includes an image sensing unit adapted to capture a real space image, a generation unit adapted to generate a virtual space image according to a viewpoint position and orientation of the image sensing unit, and a composition unit adapted to obtain a mixed reality space image by combining the real space image captured by the image sensing unit and the virtual space image generated by the generation unit. In addition, a display unit is adapted to display a mixed reality space image obtained by the composition unit, and an output unit is adapted to output a mixed reality space image, which is obtained by the composition unit, to the print unit in response to an input of a print instruction. The generation unit generates a virtual space image to be displayed by the display unit according to a first procedure, and generates a virtual space image to be outputted by the output unit according to a second procedure which is different from the first procedure.

Claims

1. An image processing apparatus which is connected to a print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

an image sensing unit adapted to capture a real space image;

a generation unit adapted to generate a virtual space image according to a viewpoint position and orientation of said image sensing unit;

a composition unit adapted to obtain a mixed reality space image by combining the real space image captured by said image sensing unit and the virtual space image generated by said generation unit;

a display unit adapted to display a mixed reality space image obtained by said composition unit; and

an output unit adapted to output a mixed reality space image, which is obtained by said composition unit, to the print unit in response to an input of a print instruction,

wherein said generation unit generates a virtual space image to be displayed by said display unit according to a first procedure, and said generation unit generates a virtual space image to be outputted by said output unit according to a second procedure, with image quality of a virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.

2. The apparatus according to claim 1, wherein said generation unit generates a virtual space image to be outputted by said output unit according to the second procedure, if the print instruction is inputted after said generation unit generates a virtual space image to be displayed by said display unit.

3. The apparatus according to claim 1, wherein a pixel size of one side of the virtual space image generated by said generation unit is a value within a product of a length of one side and a resolution of an image that said print unit can output.

4. The apparatus according to claim 1, further comprising a management unit adapted to manage state information of the real and virtual spaces, and in that said management unit updates the state information of the virtual space in accordance with input information or at each predetermined timing, and said generation unit successively generates virtual space images in accordance with the updated state information of the real and virtual spaces.

5. The apparatus according to claim 4, wherein said management unit is an independent apparatus connected to said image processing apparatus via a network, and the state information of the virtual space updated by said management unit can be received by image processing apparatuses which are equivalent to said image processing apparatus, and are connected to the network.

6. The apparatus according to claim 1, wherein said apparatus can be simultaneously used by at least one user, and has said image sensing unit, said generation unit, said composition unit, said display unit, and detection unit adapted to detect a viewpoint position and visual axis direction of a user, which are prepared for each user.

7. The apparatus according to claim 1, wherein said second procedure includes at least one of a procedure for calculating the influences of light reflection in detail, a procedure for calculating by increasing the number of bits of numerical value precision, and a procedure for calculating vertical and horizontal pixel sizes of a virtual space image using arbitrary values within range of product resolution and print size of the print unit.

8. A computer readable storage medium storing a computer program to control an image processing apparatus connected to a print unit adapted to print an image, and capable of outputting an image to be printed to the print unit, said storage medium encoded with a computer program comprising:

code to capture a real space image with an image sensing unit;

code to generate a virtual space image with a generation unit according to a viewpoint position and orientation of the image sensing unit;

code to obtain a mixed reality space image with a composition unit by combining the real space image captured by the image sensing unit and the virtual space image generated by the generation unit;

code to display a mixed reality space image obtained by the composition unit on a display unit; and

code to output on an output unit a mixed reality space image, which is obtained by the composition unit, to the print unit in response to an input of a print instruction,

wherein the generation unit generates a virtual space image to be displayed by the display unit according to a first procedure, and the generation unit generates a virtual space image to be outputted by the output unit according to a second procedure, with image quality of a virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.

9. An image processing apparatus which is connected to a print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

an image sensing unit adapted to capture a real space image;

a generation unit adapted to generate a virtual space image according to a viewpoint position and orientation of said image sensing unit;

a composition unit adapted to obtain a mixed reality space image by combining the real space image captured by said image sensing unit and the virtual space image generated by said generation unit;

a display unit adapted to display a mixed reality space image obtained by said composition unit;

an output unit adapted to output a mixed reality space image, which is obtained by said composition unit, to the print unit in response to an input of a print instruction; and

a determination unit adapted to check a positional relationship among said image sensing unit, an observer observing the mixed reality space image, and a virtual space object and determining whether or not the observer is occluded by the virtual space object in the mixed reality space image,

wherein said generation unit generates a virtual space image to be displayed by said display unit according to a first procedure, and said generation unit generates a virtual space image to be outputted by said output unit, based on a determination result of said determination unit, according to a second procedure, with image quality of a virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.

10. The apparatus according to claim 9, further comprising a management unit adapted to manage state information of the real and virtual spaces, and in that said management unit updates the update information of the virtual space in accordance with input information or at each predetermined timing, and said generation unit successively generates virtual space images in accordance with the updated state information of the real and virtual spaces.

11. The apparatus according to claim 10, wherein said management unit is an independent apparatus connected to said image processing apparatus via a network, and the state information of the virtual space updated by said management unit can be received by image processing apparatuses which are equivalent to said image processing apparatus, and are connected to the network.

12. The apparatus according to claim 9, wherein said apparatus can be simultaneously used by at least one observer, and has said image sensing unit, said generation unit, said composition unit, said display unit, and a detection unit adapted to detect a viewpoint position and visual axis direction of an observer, which are prepared for each observer.

13. The apparatus according to claim 9, wherein said second procedure includes at least one of a procedure for calculating the influences of light reflection in detail, a procedure for calculating by increasing the number of bits of numerical value precision, and a procedure for calculating vertical and horizontal pixel sizes of a virtual space image using arbitrary values within range of product resolution and print size of the print unit.

17. The apparatus according to claim 9, wherein said determination unit sets a column to cover the observer, and determines whether or not the observer is occluded by the virtual space object in the mixed reality space image based on the contact of the column and the virtual space object.

14. A computer readable storage medium storing a computer program to control an image processing apparatus connected to a print unit adapted to print an image, and capable of outputting an image to be printed to the print unit, comprising:

code to capture a real space image with an image sensing unit;

code to generate a virtual space image with a generation unit according to a viewpoint position and orientation of the image sensing unit;

code to obtain a mixed reality space image with a composition unit by combining the real space image captured by the image sensing unit and the virtual space image generated by the generation unit;

code to display a mixed reality space image obtained by the composition unit on a display unit;

code to output on an output unit a mixed reality space image, which is obtained by the composition unit, to the print unit in response to an input of a print instruction;

code to check, with a determination unit, a positional relationship among the image sensing unit, an observer observing the mixed reality space image, and a virtual space object, and determining whether or not the observer is occluded by the virtual space object in the mixed reality space image,

wherein the generation unit generates a virtual space image to be displayed by the display unit according to a first procedure, and the generation unit generates a virtual space image to be outputted by the output unit, based on determination result of the determination unit, according to a second procedure, with image quality of virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.

15. An image processing method to be executed by an image processing apparatus which is connected to a print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

an image sensing step of capturing a real space image using an image sensing unit;

a generation step of generating a virtual space image according to a viewpoint position and orientation of the image sensing unit;

a composition step of obtaining a mixed reality space image by combining the captured real space image and the generated virtual space image;

a displaying step of displaying the obtained mixed reality space image; and

an output step of outputting the obtained mixed reality space image to the print unit in response to an input of a print instruction, wherein

a virtual space image to be displayed is generated according to a first procedure, and a virtual space image to be outputted is generated according to a second procedure, with image quality of a virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.

16. An image processing method to be executed by an image processing apparatus which is connected to a print unit adapted to print an image, and can output an image to be printed to the print unit, comprising:

an image sensing step of capturing a real space image using an image sensing unit;

a generation step of generating a virtual space image according to a viewpoint position and orientation of the image sensing unit;

a composition step of obtaining a mixed reality space image by combining the captured real space image and the generated virtual space image;

a displaying step of displaying the obtained mixed reality space image;

an output step of outputting the obtained mixed reality space image to the print unit in response to an input of a print instruction, and

a determination step to check a positional relationship among the image sensing unit, an observer observing the mixed reality space image, and a virtual space object, and determining whether or not the observer is occluded by the virtual space object in the mixed reality space image, wherein

a virtual space image to be displayed is generated according to a first procedure, and a virtual space image to be outputted is generated according to a second procedure, with image quality of a virtual space image generated according to the second procedure being higher than that of a virtual space image generated according to the first procedure.