HEAD MOUNTED DISPLAY AND CONTROL METHOD THEREFOR

• US 20080297437A1
• Filed: 05/27/2008
• Published: 12/04/2008
• Est. Priority Date: 05/31/2007
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Takahashi, Hiroshi
• Granted Patent: US 8,233,011 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G09G 2340/125 wherein one of the images is motion video
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 2380/02 Flexible displays
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory

Abstract

A viewpoint information calculation unit (710) calculates a difference value (α) between reference position and orientation information and position and orientation information held by a camera viewpoint storage unit (207), and a difference value (β) between the reference position and orientation information and position and orientation information held by a CG viewpoint storage unit (208). If |α|≧|β|, a physical space image is shifted by a shift amount calculated in accordance with the resolution of a display unit (213), a difference amount, and the angle of view of a camera (203), and is composited with a virtual space image. If |α|<|β|, the virtual space image is shifted by the shift amount, and is composited with the physical space image.

Claims

1. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory;

a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory;

a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing;

a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the first memory;

a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; and

a composition unit adapted to composite the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and

wherein the composition unit selects, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and the composition unit executes the selected composition process.

2. The display according to claim 1, wherein the composition unit shifts the physical space image to be composited when an absolute value of the first difference value is equal to or larger than that of the second difference value, and shifts the virtual space image to be composited when the absolute value of the first difference value is smaller than that of the second difference value.

3. The display according to claim 2, wherein the composition unit calculates the shift amount P=M×

Δ

θ

x/Cx, and shifts the physical space image or the virtual space image by the calculated shift amount P, where M represents the resolution of the display device, Δ

θ

x represents the difference amount, and Cx represents the angle of view of the image sensing device.

4. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory;

a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory;

a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around a display timing of the display device and the position and orientation information held in the first memory;

a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory;

a unit adapted to shift the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and

a unit adapted to shift the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and composite the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.

5. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory;

a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory;

a calculation step of calculating a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing;

a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the first memory;

a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; and

a composition step of compositing the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and

wherein the composition step includes,

a step of selecting, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by the shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and

a step of executing the selected composition process.

6. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory;

a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory;

a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around a display timing of the display device and the position and orientation information held in the first memory;

a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory;

a step of shifting the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and

a step of shifting the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and compositing the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.

7. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

a first acquisition unit adapted to acquire a physical space image sensed by the image sensing device;

a second acquisition unit adapted to acquire position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image;

a transmission unit adapted to transmit the position and orientation information acquired by the second acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information;

a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information which has been received by the reception unit and the position and orientation information which has been acquired by the second acquisition unit at the time of receiving the position and orientation information;

a composition unit adapted to, when the virtual space image received by the reception unit is composited on the physical space image acquired by the first acquisition unit at the time of receiving the position and orientation information by the reception unit, shift a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted virtual space image on the physical space image; and

an output unit adapted to output a composite image by the composition unit to the display device.

8. The display according to claim 7, wherein the composition unit calculates the shift amount P=M×

Δ

θ

x/Cx, and shifts the virtual space image by the calculated shift amount P, where M represents the resolution of the display device, Δ

θ

x represents the difference amount, and Cx represents the angle of view of the image sensing device.

9. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:

a first acquisition step of acquiring a physical space image sensed by the image sensing device;

a second acquisition step of acquiring position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image;

a transmission step of transmitting the position and orientation information acquired in the second acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information;

a calculation step of calculating a difference amount of an orientation component between the position and orientation information which has been received in the reception step and the position and orientation information which has been acquired in the second acquisition step at the time of receiving the position and orientation information;

a composition step of, when the virtual space image received in the reception step is composited on the physical space image acquired in the first acquisition step at the time of receiving the position and orientation information in the reception step, shifting a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted virtual space image on the physical space image; and

an output step of outputting a composite image in the composition step to the display device.

10. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory;

a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory;

a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing;

a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the first memory;

a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; and

a composition unit adapted to composite the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and

wherein the composition unit selects, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and the composition unit executes the selected composition process.

11. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory;

a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory;

a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around a display timing of the display device and the position and orientation information held in the first memory;

a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory;

a unit adapted to shift the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and

a unit adapted to shift the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and composite the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.

12. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory;

a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the receiving position and orientation information and the receiving virtual space image in a second memory;

a calculation step of calculating a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing;

a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the first memory;

a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; and

a composition step of compositing the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and

wherein in the composition step includes,

a step of selecting, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by the shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and

a step of executing the selected composition process.

13. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory;

a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory;

a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around a display timing of the display device and the position and orientation information held in the first memory;

a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory;

a step of shifting the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and

a step of shifting the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and compositing the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.

14. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

a first acquisition unit adapted to acquire a physical space image sensed by the image sensing device;

a second acquisition unit adapted to acquire position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image;

a transmission unit adapted to transmit the position and orientation information acquired by the second acquisition unit to an external computer;

a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information;

a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information which has been received by the reception unit and the position and orientation information which has been acquired by the second acquisition unit at the time of receiving the position and orientation information;

a composition unit adapted to, when the virtual space image received by the reception unit is composited on the physical space image acquired by the first acquisition unit at the time of receiving the position and orientation information by the reception unit, shift a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted virtual space image on the physical space image; and

an output unit adapted to output a composite image by the composition unit to the display device.

15. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:

a first acquisition step of acquiring a physical space image sensed by the image sensing device;

a second acquisition step of acquiring position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image;

a transmission step of transmitting the position and orientation information acquired in the second acquisition step to an external computer;

a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information;

a calculation step of calculating a difference amount of an orientation component between the position and orientation information which has been received in the reception step and the position and orientation information which has been acquired in the second acquisition step at the time of receiving the position and orientation information;

a composition step of, when the virtual space image received in the reception step is composited on the physical space image acquired in the first acquisition step at the time of receiving the position and orientation information in the reception step, shifting a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted virtual space image on the physical space image; and

an output step of outputting a composite image in the composition step to the display device.