MIXED REALITY PRESENTATION SYSTEM

• US 20100026714A1
• Filed: 07/22/2009
• Published: 02/04/2010
• Est. Priority Date: 07/31/2008
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Utagawa, Tsutomu
• Granted Patent: US 8,760,470 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An image composition unit outputs a composition image of a physical space and virtual space to a display unit. The image composition unit calculates, as difference information, a half of the difference between an imaging time of the physical space and a generation completion predicted time of the virtual space. The difference information and acquired position and orientation information are transmitted to an image processing apparatus. A line-of-sight position prediction unit updates previous difference information using the received difference information, calculates, as the generation completion predicted time, a time ahead of a receiving time by the updated difference information, and predicts the position and orientation of a viewpoint at the calculated generation completion predicted time using the received position and orientation information. The virtual space based on the predicted position and orientation, and the generation completion predicted time are transmitted to a VHMD.

Claims

1. A mixed reality presentation system including:

a head mounted display having an imaging unit which senses an image of a physical space, and a display unit which displays an image; and

an image processing apparatus which generates an image to be displayed on the display unit,

wherein said head mounted display comprises;

a composition unit which generates a composition image by compositing a physical space image sensed by the imaging unit and a virtual space image transmitted from said image processing apparatus;

a unit which outputs the composition image to the display unit;

a calculation unit which calculates a difference between an imaging time of the physical space image and a generation completion predicted time of the virtual space image transmitted from said image processing apparatus, and calculates, as difference information, a result by multiplying the calculated difference by a ratio which is set in advance;

a unit which acquires position and orientation information indicating a position and orientation of a viewpoint; and

a transmission unit which transmits the difference information and the position and orientation information to said image processing apparatus, and

said image processing apparatus comprises;

a prediction unit which receives an information group transmitted from said transmission unit, updates previous difference information using the received difference information, calculates, as the generation completion predicted time, a time ahead of a receiving time of the information group by a time indicated by the updated difference information, and predicts a position and orientation of the viewpoint at the calculated generation completion predicted time using the received position and orientation information;

a generation unit which generates a virtual space image based on the viewpoint having the position and orientation predicted by said prediction unit; and

a unit which transmits the generation completion predicted time calculated by said prediction unit and the virtual space image generated by said generation unit to said head mounted display.

2. The system according to claim 1, wherein the units included in said image processing apparatus and the units included in said head mounted display repeat processing for respective frames.

3. The system according to claim 1, wherein said calculation unit calculates, as the difference, a result by subtracting the generation completion predicted time from the imaging time of the physical space image, and calculates, as the difference information, the result by multiplying the calculated difference by the ratio which is set in advance.

4. The system according to claim 1, wherein said prediction unit updates the difference information by subtracting the received difference information from the previous difference information.

5. A mixed reality presentation system including:

a head mounted display having an imaging unit which senses an image of a physical space, and a display unit which displays an image; and

an image processing apparatus which generates an image to be displayed on the display unit,

wherein said head mounted display comprises;

a composition unit which generates a composition image by compositing a physical space image sensed by the imaging unit and a virtual space image transmitted from said image processing apparatus;

a unit which outputs the composition image to the display unit;

a calculation unit which calculates, as difference information, a difference between a count value of the physical space image counted by a counter which increments the count value by 1 every time the imaging unit senses an image, and a count value for the virtual space image transmitted from said image processing apparatus;

a correction unit which updates the difference information by correcting the difference information to ½

when a value indicated by the difference information is not less than 1, to −

½

when the value is not more than −

1, and to 0 when the value is larger than −

1 and less than 1;

an acquisition unit which acquires position and orientation information from a sensor which measures a position and orientation of a viewpoint at a rate twice a frame rate of the imaging unit, and acquires a count value of the position and orientation information counted by a counter which increments the count value by ½

every time the sensor performs the measurement; and

a transmission unit which transmits the difference information updated by said correction unit, the position and orientation information, and the count value of the position and orientation information to said image processing apparatus, and

said image processing apparatus comprises;

a reception unit which receives an information group transmitted from said transmission unit;

a generation unit which generates a virtual space image based on the information group received by said reception unit, and which calculates, as a count result, a count value obtained by incrementing a count value by 1 every time said generation unit generates the virtual space image, and adding the difference information received by said reception unit to the incremented count value, specifies position and orientation information received by said reception unit together with a count value having the same value as the calculated count value, and generates a virtual space image based on a viewpoint having a position and orientation indicated by the specified position and orientation information; and

a unit which transmits the virtual space image generated by said generation unit and the count value calculated as the count result by said generation unit with respect to that virtual space image to said head mounted display.

6. A mixed reality presentation system including:

a head mounted display having a display unit which displays an image; and

an image processing apparatus which generates an image to be displayed on the display unit,

wherein said head mounted display comprises;

a unit which outputs a virtual space image transmitted from said image processing apparatus to the display unit;

a calculation unit which calculates a difference between a current time and a generation completion predicted time of the virtual space image transmitted from said image processing apparatus, and calculates, as difference information, a result by multiplying the calculated difference by a ratio which is set in advance;

a unit which acquires position and orientation information indicating a position and orientation of a viewpoint; and

a transmission unit which transmits the difference information and the position and orientation information to said image processing apparatus, and

said image processing apparatus comprises;

a prediction unit which receives an information group transmitted from said transmission unit, updates previous difference information using the received difference information, calculates, as the generation completion predicted time, a time ahead of a receiving time of the information group by a time indicated by the updated difference information, and predicts a position and orientation of the viewpoint at the calculated generation completion predicted time using the received position and orientation information;

a generation unit which generates a virtual space image based on the viewpoint having the position and orientation predicted by said prediction unit; and

a unit which transmits the generation completion predicted time calculated by said prediction unit and the virtual space image generated by said generation unit to said head mounted display.