MIXED REALITY PRESENTATION SYSTEM

• US 20140111546A1
• Filed: 12/20/2013
• Published: 04/24/2014
• Est. Priority Date: 07/31/2008
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: UTAGAWA, Tsutomu
• Granted Patent: US 10,607,412 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An image composition unit outputs a composition image of a physical space and virtual space to a display unit. The image composition unit calculates, as difference information, a half of the difference between an imaging time of the physical space and a generation completion predicted time of the virtual space. The difference information and acquired position and orientation information are transmitted to an image processing apparatus. A line-of-sight position prediction unit updates previous difference information using the received difference information, calculates, as the generation completion predicted time, a time ahead of a receiving time by the updated difference information, and predicts the position and orientation of a viewpoint at the calculated generation completion predicted time using the received position and orientation information. The virtual space based on the predicted position and orientation, and the generation completion predicted time are transmitted to a VHMD.

Claims

1. A system including:

a display device having a display unit which displays an image; and

an image processing apparatus which generates an image to be displayed on the display unit,

wherein said display device comprises;

the display unit configured to display a virtual object which is transmitted from said image processing apparatus;

an acquisition unit configured to acquire position and orientation information indicating a position and orientation of the display device; and

a first transmission unit configured to transmit the acquired position and orientation information to said image processing apparatus, and

said image processing apparatus comprises;

a reception unit configured to receive the position and orientation information transmitted by said first transmission unit;

a generation unit configured to generate the virtual object by using a first position and orientation which is obtained based on the received position and orientation information; and

a second transmission unit configured to transmit the virtual object generated by said generation unit to said display device.

2. The system according to claim 1, wherein the display device is a head mounted display.

3. The system according to claim 1, wherein the virtual object is a computer graphic.

4. The system according to claim 1, wherein the acquisition unit acquires the position and orientation information indicating the position and orientation of the display device using an arbitrary sensor.

5. The system according to claim 1, wherein the first position and orientation is depend on a data amount of the virtual object and the received position and orientation information.

6. The system according to claim 1, wherein said display device is connected with said image processing apparatus wirelessly.

7. The system according to claim 1, further comprises:

a prediction unit configured to predict the first position and orientation of said display device.

8. A display device which is connected with an image processing apparatus comprising:

an acquisition unit configured to acquire position and orientation information indicating a position and orientation of the display device;

a transmission unit configured to transmit the acquired position and orientation information to said image processing apparatus;

a reception unit configured to receive, from said the image processing apparatus, a virtual object which is generated by using a first position and orientation, wherein the first position and orientation is different from the position and orientation indicated by the acquired position and orientation information and is obtained based on the acquired position and orientation information; and

a display unit configured to display the virtual object which is transmitted from said image processing apparatus.

9. The display device according to claim 8, wherein the display device is a head mounted display.

10. The display device according to claim 8, wherein the virtual object is a computer graphic.

11. The display device according to claim 8, wherein the acquisition unit acquires the position and orientation information indicating the position and orientation of the display device using an arbitrary sensor.

12. The display device according to claim 8, wherein the first position and orientation is depend on a data amount of the virtual object and the acquired position and orientation information.

13. The display device according to claim 8, wherein said display device is connected with said image processing apparatus wirelessly.

14. An image processing apparatus which is connected with a display device comprising:

an obtaining unit configured to obtain position and orientation information of the display device;

a generation unit configured to generate a virtual object by using a first position and orientation of said display device, wherein the first position and orientation is different from a position and orientation indicated by the obtained position and orientation information and is obtained based on the obtained position and orientation information; and

a transmission unit configured to transmit the generated virtual object to said display device.

15. The apparatus according to claim 14, wherein the display device is a head mounted display.

16. The apparatus according to claim 14, wherein the virtual object is a computer graphic.

17. The apparatus according to claim 14, further comprises:

a prediction unit configured to predict the first position and orientation of said display device.

18. The apparatus according to claim 14, wherein the first position and orientation is depend on a data amount of the virtual object and the obtained position and orientation information.

19. The apparatus according to claim 14 wherein said display device is connected with said image processing apparatus wirelessly.

20. A method of controlling a system, the system including a display device having a display unit configured to display an image and an image processing apparatus configured to generate an image to be displayed on the display unit, wherein

said display device performs steps comprising:

a display step of displaying a virtual object which is transmitted from said image processing apparatus;

an acquisition step of acquiring position and orientation information indicating a position and orientation of the display device; and

a first transmission step of transmitting the acquired position and orientation information to said image processing apparatus, and

said image processing apparatus performs steps comprising;

a reception step of receiving the position and orientation information transmitted by said first transmission step;

a generation step of generating the virtual object by using a first position and orientation which is obtained based on the received position and orientation information; and

a second transmission step of transmitting the virtual object generated by said generation step to said display device.

21. A method of controlling a display device which is connected with an image processing apparatus, the method comprising:

an acquisition step of acquiring position and orientation information indicating a position and orientation of the display device;

a transmission step of transmitting the acquired position and orientation information to said image processing apparatus;

a reception step of receiving, from said the image processing apparatus, a virtual object which is generated by using a first position and orientation,

wherein the first position and orientation is different from the position and orientation indicated by the acquired position and orientation information and is obtained based on the acquired position and orientation information; and

a display step of displaying the virtual object which is transmitted from said image processing apparatus.

22. A method of controlling an image processing apparatus which is connected with a display device, the method comprising:

an obtaining step of obtaining position and orientation information of the display device;

a generation step of generating a virtual object by using a first position and orientation of said display device,

wherein the first position and orientation is different from a position and orientation indicated by the obtained position and orientation information and is obtained based on the obtained position and orientation information; and

a transmission step of transmitting the generated virtual object to said display device.