INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, COMPUTER PROGRAM, AND IMAGE PROCESSING SYSTEM

• US 20170083084A1
• Filed: 04/21/2015
• Published: 03/23/2017
• Est. Priority Date: 06/30/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Tatsuta, Hirokazu, Kawamoto, Kenta, Koyama, Yuichiro, Kon, Takayasu, Hasegawa, Yuichi, Shimura, Masaru, Yamamoto, Yuki, Tanaka, Eiichi
• Granted Patent: US 10,310,595 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G09G 2320/068 for control of viewing angle adjustment
• CPC Class Codes: G09G 2320/0693 Calibration of display systems
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

Provided is an image display device configured to display a free viewpoint image that tracks a movement of observer'"'"'s head and trunk.

The line-of-sight direction and trunk orientation of the user are individually oriented in a free viewpoint space displayed on a display device 500 based on posture information of the user'"'"'s head and trunk, which are respectively obtained from a head motion tracking device 200 and a trunk motion tracking device 300. A behavior that is natural in real space can be represented by moving a line of sight to a front direction of the trunk on a space in a state where the line of sight does not match the front direction of the trunk. An unnatural UI that goes straight to the line-of-sight direction is prevented regardless of orientation of the trunk.

Claims

1. An information processing apparatus comprising:

a receiver configured to receive first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; and

an image rendering processing unit configured to generate a display image corresponding to a posture of the observer based on the first information and the second information.

2. The information processing apparatus according to claim 1,

wherein the receiver receives at least a posture of a trunk of the observer as the second information, and

the image rendering processing unit generates a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information.

3. The information processing apparatus according to claim 2,

wherein the image rendering processing unit, when receiving a control signal used to instruct the observer to move in the free viewpoint space as an input, determines a point after movement (viewpoint position) by recognizing, as a front direction, an orientation of a body oriented based on a posture of the trunk of the observer obtained from the second information.

4. The information processing apparatus according to claim 1,

wherein the image rendering processing unit generates a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information and by arranging in a fixed position oriented based on the posture obtained from the second information.

5. The information processing apparatus according to claim 1,

wherein the receiver receives at least a posture of the trunk of the observer as the second information, and

the image rendering processing unit generates a free viewpoint image that tracks a posture of the head of the observer by orienting a line-of-sight direction of the observer in a free viewpoint space based on the first information, by orienting an orientation (viewpoint position) of the body of the observer in the free viewpoint space based on a posture of the trunk of the observer obtained from the second information, and by arranging a predetermined image widget in a fixed position oriented based on a posture of the trunk of the observer.

6. The information processing apparatus according to claim 1, further comprising:

a calibration processing unit configured to acquire a calibration parameter,

wherein the image rendering processing unit performs image generation using posture information corrected by the calibration parameter.

7. The information processing apparatus according to claim 6,

wherein the calibration processing unit, based on head posture information obtained from the first information and second posture information obtained from the second information, calculates a calibration parameter used to correct the second posture information and corrects the second posture information using the calibration parameter.

8. The information processing apparatus according to claim 7,

wherein the calibration processing unit calculates a posture conversion parameter used to match the second posture information with the head posture information as the calibration parameter, by using the head posture information and the second posture information.

9. The information processing apparatus according to claim 7,

wherein the posture information is represented by a quaternion,

the calibration processing unit calculates a calibration quaternion used to correct the second posture information by multiplying the head posture information by an inverse of a quaternion of the second posture information from a right side, and

the correction is performed by multiplying the quaternion of the second posture information by the calibration quaternion from a left side.

10. The information processing apparatus according to claim 6,

wherein the calibration processing unit estimates a calibration parameter used to correct the second posture information based on rotational displacement for a certain period of a coordinate system of a head posture obtained from the first information and a coordinate system of a posture obtained from the second information and a property of a human body.

11. The information processing apparatus according to claim 6,

wherein the calibration processing unit sets a time-series average of calibration parameters calculated for a certain period based on head posture information obtained from the first information and second posture information obtained from the second information to be a final calibration parameter.

12. The information processing apparatus according to claim 11,

wherein the posture information is represented by a quaternion, and

the calibration processing unit updates a calibration quaternion by performing spherical linear interpolation between a calibration quaternion newly calculated based on a quaternion of posture information obtained from each of the first information and the second information that have been newly received and a last calibration quaternion determined by the time-series average.

13. The information processing apparatus according to claim 11,

wherein a posture of the trunk of the observer is calculated based on the second information, and

the calibration processing unit determines an average value of an angular difference of a rotation direction, when a gravity direction is set to a rotation axis, of posture information of a head and a trunk obtained respectively from the first information and the second information that have been newly received, and calculates a calibration parameter on a gravity direction and the average value.

14. The information processing apparatus according to claim 1,

wherein the image rendering processing unit processes a display image on the display unit based on a difference between a head posture obtained from the first information and a trunk posture obtained from the second information.

15. The information processing apparatus according to claim 2,

wherein the image rendering processing unit generates a free viewpoint image by correcting a world coordinate system in a manner that a difference between a head posture and a trunk posture in the free viewpoint space decreases under a biosafety support mode.

16. The information processing apparatus according to claim 2,

wherein the image rendering processing unit generates a free viewpoint image by fixing a head posture in a free viewpoint space at a point of time when a biosafety support mode is set and by changing a trunk posture in a free viewpoint space depending on a head posture detected by a head posture detection unit under the biosafety support mode.

17. The information processing apparatus according to claim 15,

wherein the biosafety support mode is set when a state where a difference between a head posture obtained from the first information and a trunk posture obtained from the second information exceeds a first threshold continues for a certain time, and the biosafety support mode is released when the difference is equal to or less than a second threshold that is less than the first threshold.

18. An information processing method comprising:

a receiving step of receiving first information on a posture of a head of an observer and second information on a posture of a body other than the head of the observer; and

; and

an image rendering processing step of generating a display image corresponding to a posture of the observer based on the first information and the second information.

19. A computer program written in a computer readable format to cause a computer to function as:

a head posture operation unit configured to calculate posture information of a head of an observer based on a result obtained by detection in a head posture detection unit configured to detect a posture of the head of the observer;

a second posture operation unit configured to calculate posture information of a second part of a body other than the head of the observer based on a result obtained by detection in a second posture detection unit configured to detect a posture of one or more of the second part; and

an image rendering processing unit configured to process an image to be displayed on a display unit based on a posture of the head of the observer and a posture of the second part, the display unit being fixed to the head or a face of the observer.

20. An image processing system comprising:

a display unit fixed to a head or a face of an observer;

a head posture detection unit configured to detect a posture of the head of the observer;

a second posture detection unit configured to detect a posture of one or more second parts of a body other than the head of the observer; and

an image rendering processing unit configured to process a display image on the display unit based on a posture of the head of the observer and a posture of each of the second parts.