IMAGE PROCESSING SYSTEM, IMAGE PROCESSING APPARATUS, IMAGE SENSING APPARATUS, AND CONTROL METHOD THEREOF

• US 20070242086A1
• Filed: 04/11/2007
• Published: 10/18/2007
• Est. Priority Date: 04/14/2006
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Shimoyama, Tomohiko, Tsujimoto, Takuya, Yamamoto, Hiroyuki, Katayama, Akihiro
• Granted Patent: US 8,345,952 B2
• Time in Patent Office: Days
• US Class Current: 345/632
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A first process to be executed by an apparatus and a second process to be executed by an apparatus to obtain a composite image by compositing a real space image with a virtual space image are determined by negotiation. The apparatus (101) transmits data necessary for executing the second process to the apparatus. The apparatus executes the second process by using the data and transmits the process result to the apparatus. The apparatus receives the process result by the apparatus and executes the first process. The apparatus displays a composite image obtained on the basis of the result of the first process or a composite image obtained on the basis of the process result and the received process result.

Claims

1. An image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

the image sensing apparatus comprising:

first determination unit adapted to determine, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

first transmission unit adapted to transmit, to the image processing apparatus, data necessary for causing the image processing apparatus to execute the second process;

first reception unit adapted to receive a result of the process executed by the image processing apparatus by using the data transmitted-by said first transmission unit;

first processing unit adapted to execute the first process; and

display unit adapted to display one of the composite image obtained on the basis of the process result by said first processing unit and the composite image obtained on the basis of the process result by said first processing unit and a reception result received by said first reception unit, and

the image processing apparatus comprising;

second determination unit adapted to determine the first process and the second process by the negotiation;

second processing unit adapted to execute the second process by using the data transmitted by said first transmission unit; and

second transmission unit adapted to transmit a process result by said second processing unit to the image sensing apparatus.

2. The system according to claim 1, wherein when the first process includes a first part of a series of processes of obtaining a position and orientation of the viewpoint of the image sensing apparatus and a series of a composition processes of compositing the sensed image with the virtual space image, and the second process includes a second part of the series of processes of obtaining the position and orientation of the viewpoint of the image sensing apparatus,

said first processing unit executes a process corresponding to the first part,

said first transmission unit transmits the process result by said first processing unit,

said first reception unit receives the composite image by the composition process,

said display unit displays the composite image,

said second processing unit executes a process corresponding to the second part by using the process result by said first processing unit and executes the composition process by using a result of the process, and

said second transmission unit transmits the composite image by the composition process.

3. The system according to claim 1, wherein when the first process includes a calculation process of obtaining a position and orientation of the viewpoint of the image sensing apparatus and a composition process of compositing the sensed image with the virtual space image,

said first processing unit executes the calculation process,

said first transmission unit transmits a calculation result of the calculation process,

said first reception unit receives a virtual space image generated by the image processing apparatus on the basis of the calculation result transmitted by said first transmission unit,

said display unit displays a composite image of the sensed image and the virtual space image received by said first reception unit,

said second processing unit generates the virtual space image on the basis of the calculation result transmitted by said first transmission unit, and

said second transmission unit transmits the virtual space image generated by said second processing unit.

4. The system according to claim 1, wherein when the second process includes a calculation process of obtaining a position and orientation of the viewpoint of the image sensing apparatus and a composition process of compositing the sensed image with the virtual space image,

said first transmission unit transmits the sensed image,

said first reception unit receives a composite image of the sensed image and a virtual space image generated by the image processing apparatus on the basis of the sensed image transmitted by said first transmission unit,

said display unit displays the composite image received by said first reception unit,

said second processing unit obtains the position and orientation of the viewpoint on the basis of the sensed image transmitted by said first transmission unit, generates the virtual space image on the basis of the obtained position and orientation, and generates the composite image of the generated virtual space image and the sensed image, and

said second transmission unit transmits the composite image generated by said second processing unit.

5. The system according to claim 1, wherein the sensed image transmitted by said first transmission unit is a reduced image of the sensed image by the image sensing apparatus.

6. The system according to claim 1, wherein the sensed image transmitted by said first transmission unit is a compressed image of the sensed image by the image sensing apparatus.

7. The system according to claim 1, wherein the sensed image transmitted by said first transmission unit is an image obtained by converting color representation and/or the number of tones of the sensed image by the image sensing apparatus.

8. The system according to claim 1, wherein said first determination unit and said second determination unit determine a corresponding mode by the negotiation and determine the first process and the second process to a process corresponding to the determined mode.

9. The system according to claim 1, wherein said first determination unit and said second determination unit determine a corresponding mode by the negotiation on the basis of at least one of a transmission band of communication between the image sensing apparatus and the image processing apparatus, a type of transmission medium, a type and update time of an algorithm to measure the position of the viewpoint, a data process capability of each of the image sensing apparatus and the image processing apparatus, and the number of connected image sensing apparatuses communicating with the image processing apparatus.

10. The system according to claim 1, wherein the position and orientation of the viewpoint are obtained by using one of a marker and a natural feature in an image.

11. An image processing apparatus capable of data communication with an image sensing apparatus for sensing a real space, comprising:

determination unit adapted to determine, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

processing unit adapted to execute the second process by using data that is received from the image sensing apparatus as data necessary for causing the image processing apparatus to execute the second process; and

transmission unit adapted to transmit a process result by said processing unit to the image sensing apparatus.

12. An image sensing apparatus which executes data communication with an image processing apparatus and senses a real space, comprising:

determination unit adapted to determine, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

transmission unit adapted to transmit, to the image processing apparatus, data necessary for causing the image processing apparatus to execute the second process;

reception unit adapted to receive a result of the process executed by the image processing apparatus by using the data transmitted by said transmission unit;

processing unit adapted to execute the first process; and

display unit adapted to display one of the composite image obtained on the basis of the process result by said processing unit and the composite image obtained on the basis of the process result by said processing unit and a reception result received by said reception unit.

13. An image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

the image sensing apparatus comprising:

unit adapted to obtain a position and orientation of a viewpoint of the image sensing apparatus on the basis of a sensed image by the image sensing apparatus;

first transmission unit adapted to transmit the position and orientation of the viewpoint to the image processing apparatus;

first reception unit adapted to receive, from the image processing apparatus, a virtual space image generated by the image processing apparatus on the basis of the position and orientation of the viewpoint;

unit adapted to generate a composite image by compositing the sensed image with the virtual space image received by said first reception unit; and

display unit adapted to display the composite image, and

the image processing apparatus comprising;

second reception unit adapted to receive the position and orientation of the viewpoint transmitted by said first transmission unit;

generation unit adapted to generate the virtual space image on the basis of the position and orientation of the viewpoint received by said second reception unit; and

second transmission unit adapted to transmit the virtual space image generated by said generation unit to the image sensing apparatus.

14. An image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

the image sensing apparatus comprising:

image generation unit adapted to generate a first image by reducing a data amount of a sensed image by the image sensing apparatus;

first transmission unit adapted to transmit the first image to the image processing apparatus;

first reception unit adapted to receive, from the image processing apparatus, a composite image obtained by setting the first image to the same size as a virtual space image and compositing the set first image with the virtual space image;

replacement unit adapted to, if a specific area on the composite image received by said first reception unit corresponds to an area on the first image, replace the specific area with a corresponding area on the sensed image; and

display unit adapted to display one of the composite image received by said first reception unit and the composite image that has undergone the process by said replacement unit, and

the image processing apparatus comprising;

second reception unit adapted to receive the first image transmitted by said first transmission unit;

unit adapted to obtain a position and orientation of a viewpoint of the image sensing apparatus by using the first image received by said second reception unit;

virtual space image generation unit adapted to generate the virtual space image on the basis of the position and orientation of the viewpoint;

composition unit adapted to generate the composite image by setting size of the first image to the size of the virtual space image generated by said virtual space image generation unit and compositing the set first image with the virtual space image; and

second transmission unit adapted to transmit the composite image generated by said composition unit to the image sensing apparatus.

15. The system according to claim 14, wherein said image generation unit generates a reduced image by reducing a size of the sensed image by the image sensing apparatus.

16. A control method of an image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

a control method of the image sensing apparatus comprising:

a first determination step of determining, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

a first transmission step of transmitting, to the image processing apparatus, data necessary for causing the image processing apparatus to execute the second process;

a first reception step of receiving a result of the process executed by the image processing apparatus by using the data transmitted in the first transmission step;

a first processing step of executing the first process; and

a display step of displaying one of the composite image obtained on the basis of the process result in the first processing step and the composite image obtained on the basis of the process result in the first processing step and a reception result received in the first reception step, and

a control method of the image processing apparatus comprising;

a second determination step of determining the first process and the second process by the negotiation;

a second processing step of executing the second process by using the data transmitted in the first transmission step; and

a second transmission step of transmitting a process result in the second processing step to the image sensing apparatus.

17. A control method of an image processing apparatus capable of data communication with an image sensing apparatus for sensing a real space, comprising steps of:

determining, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

executing the second process by using data that is received from the image sensing apparatus as data necessary for causing the image processing apparatus to execute the second process; and

transmitting a process result in the processing step to the image sensing apparatus.

18. A control method of an image sensing apparatus which executes data communication with an image processing apparatus and senses a real space, comprising steps of:

determining, by negotiation with the image processing apparatus, a first process to be executed by the image sensing apparatus and a second process to be executed by the image processing apparatus to obtain a composite image by compositing a sensed image by the image sensing apparatus with an image of a virtual space seen from a viewpoint of the image sensing apparatus;

transmitting, to the image processing apparatus, data necessary for causing the image processing apparatus to execute the second process;

receiving a result of the process executed by the image processing apparatus by using the data transmitted in the transmitting step;

executing the first process; and

displaying one of the composite image obtained on the basis of the process result in the processing step and the composite image obtained on the basis of the process result in the processing step and a reception result received in the receiving step.

19. A control method of an image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

a control method of the image sensing apparatus comprising:

a step of obtaining a position and orientation of a viewpoint of the image sensing apparatus on the basis of a sensed image by the image sensing apparatus;

a first transmission step of transmitting the position and orientation of the viewpoint to the image processing apparatus;

a first reception step of receiving, from the image processing apparatus, a virtual space image generated by the image processing apparatus on the basis of the position and orientation of the viewpoint;

a step of generating a composite image by compositing the sensed image with the virtual space image received in the first reception step; and

a display step of displaying the composite image, and

a control method of the image processing apparatus comprising;

a second reception step of receiving the position and orientation of the viewpoint transmitted in the first transmission step;

a generation step of generating the virtual space image on the basis of the position and orientation of the viewpoint received in the second reception step; and

a second transmission step of transmitting the virtual space image generated in the generation step to the image sensing apparatus.

20. A control method of an image processing system including an image sensing apparatus for sensing a real space, and an image processing apparatus capable of data communication with the image sensing apparatus,

a control method of the image sensing apparatus comprising:

an image generation step of generating a first image by reducing a data amount of a sensed image by the image sensing apparatus;

a first transmission step of transmitting the first image to the image processing apparatus;

a first reception step of receiving, from the image processing apparatus, a composite image obtained by setting size of the first image to the same size as a virtual space image and compositing the set first image with the virtual space image;

a replacement step of, if a specific area on the composite image received in the first reception step corresponds to an area on the first image, replacing the specific area with a corresponding area on the sensed image; and

a display step of displaying one of the composite image received in the first reception step and the composite image that has undergone the process in the replacement step, and

a control method of the image processing apparatus comprising;

a second reception step of receiving the first image transmitted in the first transmission step;

a step of obtaining a position and orientation of a viewpoint of the image sensing apparatus by using the first image received in the second reception step;

a virtual space image generation step of generating the virtual space image on the basis of the position and orientation of the viewpoint;

a composition step of generating the composite image by setting size of the first image to the size of the virtual space image generated by the virtual space image generation step and compositing the set first image with the virtual space image; and

a second transmission step of transmitting the composite image generated in the composition step to the image sensing apparatus.