Handling different input signals

• US 10,095,032 B2
• Filed: 06/10/2014
• Issued: 10/09/2018
• Est. Priority Date: 06/11/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Bickerstaff, Ian Henry, Benson, Simon Mark
• Primary Examiner(s): VOLENTINE, REBECCA A
• Application Number:
  US14/300,649
• Publication Number:
  US 20140362180A1
• Time in Patent Office: 1,582 Days
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/356 having separate monoscopic and stereoscopic modes

Abstract

A head mountable display (HMD) system in which the size and/or distance from the viewer of a virtual screen for display of video material depends upon one or more of the following:

• • whether the video material is in a 2-D or a 3-D format;
  • the image resolution of the video material;
  • the compression quality and/or data rate of the video material;
  • the aspect ratio of the video material;
  • the origin of the video material; and
  • metadata associated with the video material.

Claims

1. A head mountable display (HMD) system comprising:

an HMD having stereoscopic display elements for eyes of a wearer of the HMD, the stereoscopic display elements being configured to provide a virtual screen to the HMD wearer at a rendered virtual position within a three-dimensional display environment; and

a video processor configured to process video material for display by the stereoscopic display elements of the HMD;

in which the video processor is configured so that one or both of a size of the virtual screen and a distance from the wearer of the rendered virtual position of the virtual screen within the three-dimensional display environment for display of the video material by the HMD depends upon a spatial frequency content of the video material,

wherein when a maximum spatial frequency is less than a threshold amount, the system is configured to display the video material on a virtual screen that appears smaller and at a selected rendered virtual position which is closer to the wearer within the three-dimensional display environment than a virtual screen on which video with a maximum spatial frequency above the threshold amount is displayed.

2. A system according to claim 1, comprising:

a detector configured to detect the spatial frequency content of the video material;

the video processor being configured to display the video material on a virtual screen having a size and a rendered virtual distance in the three-dimensional display environment from the wearer of the HMD according to the detected spatial frequency content of the video material.

3. A system according to claim 2, in which, in a case where spatial content of the video material indicates a lower capture resolution than a current resolution of the video material, the video processor is configured to display the video material on a virtual screen having a size and a rendered virtual distance in the three-dimensional display environment from the wearer which depend on the capture resolution indicated by the detected spatial frequency content.

4. A system according to claim 1, comprising an audio processor configured to generate audio content in a simulated audio presentation format dependent upon one or both of the size and a rendered virtual distance of the virtual screen in the three-dimensional display environment from the wearer selected by the video processor.

5. A system according to claim 4, in which the HMD comprises a pair of earpieces;

and in which the audio processor is configured to supply the generated audio content to the earpieces.

6. A system according to claim 4, in which the audio processor is configured:

to simulate a single front central monophonic sound source in respect of audio content associated with video content displayed by a virtual screen of less than a first threshold virtual distance in the three-dimensional display environment from the wearer; and

to simulate multiple sound sources in respect of other audio content.

7. A system according to claim 1, in which, in a case of display of 2-D material, one or both of a higher resolution and a higher data rate correspond to a virtual screen which has a rendered virtual position within the three-dimensional display environment which is more distant and which appears larger.

8. A system according to claim 1, in which, in a case of 3-D material, the video material is displayed on a virtual screen of no more than 5 m diagonal and having a rendered virtual position within the three-dimensional display environment which is no more than 8 m from the user.

9. A system according to claim 1, in which, in a case of 3-D material, the video material is displayed on a virtual screen of no more than 2 m diagonal and having a rendered virtual position within the three-dimensional display environment which is between 2 m and 3 m from the viewer.

10. A system according to claim 1, in which, if metadata defining precautions against window violations is associated with the video material, a window violation occurring when an object in a 3-D environment approaches an edge of a display screen and, by virtue of its position in 3-D space and particular camera arrangements in use, the object appears in only one of a left or a right image but not the other, such metadata defining temporary changes to a lateral extent of the left and right images so as to mask objects which would otherwise give rise to window violations, then the metadata is used in rendering of 3-D video material.

11. A system according to claim 1, in which, if metadata defining precautions against window violations is not associated with the video material, a window violation occurring when an object in a 3-D environment approaches an edge of a display screen and, by virtue of its position in 3-D space and particular camera arrangements in use, the object appears in only one of a left or a right image but not the other, such metadata defining temporary changes to a lateral extent of the left and right images so as to mask objects which would otherwise give rise to window violations, then the system is configured to analyse the video material and to change rendering of the video material according to the analysis.

12. A system according to claim 1, in which the system is configured to detect head movements of the HMD in use, and in which the detection of translation as part of a head tracking detection is not used to vary a virtual viewpoint for video display.

13. A system according to claim 1, the system being configured to transition gradually between different virtual screen sizes and different virtual screen distances from the viewer.

14. A video display system comprising:

a frame;

at least one a pair of stereoscopic display elements coupled to the frame to provide a three-dimensional display environment; and

a video processor arranged in operative communication with the pair of stereoscopic display elements,

in which the video processor is configured to generate video material for display, one or both of an image size and a rendered image position within the three-dimensional display environment at which the video material is displayed depending upon an image resolution of the video material and spatial frequency content of the video material, wherein if a maximum spatial frequency is less than a threshold amount, then the video material is configured by the video processor for display on a virtual screen associated with the pair of stereoscopic display elements, the virtual screen configured to appear smaller and at a selected rendered position within the three-dimensional display environment which is closer to a user than a virtual screen on which video with a maximum spatial frequency above the threshold amount is configured for display.

15. A method of operation of a stereoscopic head mountable display (HMD) system operable to provide a three dimensional display environment, comprising:

rendering a virtual screen for display of video material to a viewer of the HMD system, in which one or both of a size and rendered virtual distance from the viewer of the virtual screen within the three-dimensional display environment depends upon spatial frequency content of the video material;

determining, by one or more processors, whether a maximum spatial frequency is less than a threshold amount; and

when the maximum spatial frequency is determined to be less than the threshold amount, displaying the video material on a virtual screen that appears smaller and at a selected rendered virtual position within the three-dimensional display environment which is closer to the viewer than a virtual screen on which video with a maximum spatial frequency above the threshold amount is displayed.

17. A non-transitory machine-readable storage medium which stores computer software which, when executed by a computer, causes the computer to carry out the method of claim 15.

16. A method of operation of a stereoscopic video display operable to provide a three-dimensional display environment, in which one or both of an image size and a rendered image position within the three-dimensional display environment at which video material is displayed depends upon an image resolution of the video material and spatial frequency content of the video material, the method comprising:

determining, by one or more processors, whether a maximum spatial frequency is less than a threshold amount; and

upon determination that the maximum spatial frequency is less than the threshold amount, preparing the video material for display on a virtual screen that appears smaller and at a selected rendered position within the three-dimensional display environment which is closer to a user of the video display than a virtual screen on which video with a maximum spatial frequency above the threshold amount is displayed.