HEAD MOUNTED DISPLAY AND CONTROL METHOD THEREFOR
First Claim
1. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
- an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory;
a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer;
a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory;
a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing;
a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the first memory;
a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; and
a composition unit adapted to composite the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, andwherein the composition unit selects, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and the composition unit executes the selected composition process.
1 Assignment
0 Petitions
Accused Products
Abstract
A viewpoint information calculation unit (710) calculates a difference value (α) between reference position and orientation information and position and orientation information held by a camera viewpoint storage unit (207), and a difference value (β) between the reference position and orientation information and position and orientation information held by a CG viewpoint storage unit (208). If |α|≧|β|, a physical space image is shifted by a shift amount calculated in accordance with the resolution of a display unit (213), a difference amount, and the angle of view of a camera (203), and is composited with a virtual space image. If |α|<|β|, the virtual space image is shifted by the shift amount, and is composited with the physical space image.
-
Citations
15 Claims
-
1. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory; a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory; a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing; a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the first memory; a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; and a composition unit adapted to composite the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and wherein the composition unit selects, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and the composition unit executes the selected composition process. - View Dependent Claims (2, 3)
-
-
4. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory; a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory; a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around a display timing of the display device and the position and orientation information held in the first memory; a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; a unit adapted to shift the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and a unit adapted to shift the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and composite the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.
-
-
5. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory; a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory; a calculation step of calculating a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing; a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the first memory; a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; and a composition step of compositing the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and wherein the composition step includes, a step of selecting, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by the shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and a step of executing the selected composition process.
-
-
6. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory; a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory; a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around a display timing of the display device and the position and orientation information held in the first memory; a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; a step of shifting the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and a step of shifting the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and compositing the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.
-
-
7. A head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
a first acquisition unit adapted to acquire a physical space image sensed by the image sensing device; a second acquisition unit adapted to acquire position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image; a transmission unit adapted to transmit the position and orientation information acquired by the second acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information; a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information which has been received by the reception unit and the position and orientation information which has been acquired by the second acquisition unit at the time of receiving the position and orientation information; a composition unit adapted to, when the virtual space image received by the reception unit is composited on the physical space image acquired by the first acquisition unit at the time of receiving the position and orientation information by the reception unit, shift a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted virtual space image on the physical space image; and an output unit adapted to output a composite image by the composition unit to the display device. - View Dependent Claims (8)
-
-
9. A control method for a head mounted display including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
a first acquisition step of acquiring a physical space image sensed by the image sensing device; a second acquisition step of acquiring position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image; a transmission step of transmitting the position and orientation information acquired in the second acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information; a calculation step of calculating a difference amount of an orientation component between the position and orientation information which has been received in the reception step and the position and orientation information which has been acquired in the second acquisition step at the time of receiving the position and orientation information; a composition step of, when the virtual space image received in the reception step is composited on the physical space image acquired in the first acquisition step at the time of receiving the position and orientation information in the reception step, shifting a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted virtual space image on the physical space image; and an output step of outputting a composite image in the composition step to the display device.
-
-
10. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory; a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory; a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing; a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the first memory; a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; and a composition unit adapted to composite the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and wherein the composition unit selects, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and the composition unit executes the selected composition process.
-
-
11. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition unit adapted to acquire, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and record the position and orientation information together with the received physical space image in a first memory; a transmission unit adapted to transmit the position and orientation information acquired by the acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information, and record the received position and orientation information and the received virtual space image in a second memory; a unit adapted to calculate a first difference value between the position and orientation information acquired by the acquisition unit at around a display timing of the display device and the position and orientation information held in the first memory; a unit adapted to calculate a second difference value between the position and orientation information acquired by the acquisition unit at around the display timing of the display device and the position and orientation information held in the second memory; a unit adapted to shift the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and a unit adapted to shift the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and composite the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.
-
-
12. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory; a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the receiving position and orientation information and the receiving virtual space image in a second memory; a calculation step of calculating a difference amount of an orientation component between the position and orientation information held in the first memory at a display timing of the display device and the position and orientation information held in the second memory at the display timing; a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the first memory; a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; and a composition step of compositing the physical space image held in the first memory at the display timing of the display device and the virtual space image held in the second memory at the display timing of the display device, and wherein in the composition step includes, a step of selecting, based on a magnitude relationship between the first difference value and the second difference value, either a composition process of shifting a virtual space image to be composited by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device and compositing the shifted virtual space image, or a composition process of shifting a physical space image to be composited by the shift amount calculated in accordance with the resolution information representing the resolution of the display device, the difference amount, and the angle-of-view information representing the angle of view of the image sensing device and compositing the shifted physical space image, and a step of executing the selected composition process.
-
-
13. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
an acquisition step of acquiring, every time receiving a physical space image sensed by the image sensing device, position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the received physical space image, and recording the position and orientation information together with the received physical space image in a first memory; a transmission step of transmitting the position and orientation information acquired in the acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information, and recording the received position and orientation information and the received virtual space image in a second memory; a step of calculating a first difference value between the position and orientation information acquired in the acquisition step at around a display timing of the display device and the position and orientation information held in the first memory; a step of calculating a second difference value between the position and orientation information acquired in the acquisition step at around the display timing of the display device and the position and orientation information held in the second memory; a step of shifting the physical space image held in the first memory at the display timing of the display device by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the first difference value, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted physical space image with the virtual space image held in the second memory at the display timing of the display device; and a step of shifting the virtual space image held in the second memory at the display timing of the display device by a shift amount calculated in accordance with the resolution information representing the resolution of the display device, the second difference value, and the angle-of-view information representing the angle of view of the image sensing device, and compositing the shifted virtual space image with the physical space image held in the first memory at the display timing of the display device.
-
-
14. An apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
a first acquisition unit adapted to acquire a physical space image sensed by the image sensing device; a second acquisition unit adapted to acquire position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image; a transmission unit adapted to transmit the position and orientation information acquired by the second acquisition unit to an external computer; a reception unit adapted to receive, from the computer, the position and orientation information transmitted by the transmission unit and a virtual space image generated by the computer on the basis of the position and orientation information; a calculation unit adapted to calculate a difference amount of an orientation component between the position and orientation information which has been received by the reception unit and the position and orientation information which has been acquired by the second acquisition unit at the time of receiving the position and orientation information; a composition unit adapted to, when the virtual space image received by the reception unit is composited on the physical space image acquired by the first acquisition unit at the time of receiving the position and orientation information by the reception unit, shift a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and composite the shifted virtual space image on the physical space image; and an output unit adapted to output a composite image by the composition unit to the display device.
-
-
15. A control method for an apparatus including an image sensing device, a display device, and a position and orientation sensor, comprising:
-
a first acquisition step of acquiring a physical space image sensed by the image sensing device; a second acquisition step of acquiring position and orientation information representing a position and orientation which has been measured by the position and orientation sensor at the time of sensing the physical space image; a transmission step of transmitting the position and orientation information acquired in the second acquisition step to an external computer; a reception step of receiving, from the computer, the position and orientation information transmitted in the transmission step and a virtual space image generated by the computer on the basis of the position and orientation information; a calculation step of calculating a difference amount of an orientation component between the position and orientation information which has been received in the reception step and the position and orientation information which has been acquired in the second acquisition step at the time of receiving the position and orientation information; a composition step of, when the virtual space image received in the reception step is composited on the physical space image acquired in the first acquisition step at the time of receiving the position and orientation information in the reception step, shifting a composition position of the virtual space image by a shift amount calculated in accordance with resolution information representing a resolution of the display device, the difference amount, and angle-of-view information representing an angle of view of the image sensing device, and compositing the shifted virtual space image on the physical space image; and an output step of outputting a composite image in the composition step to the display device.
-
Specification