Wireless head mounted display with differential rendering and sound localization

• US 10,585,472 B2
• Filed: 10/28/2016
• Issued: 03/10/2020
• Est. Priority Date: 08/12/2011
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Tokubo, Todd, Osman, Steven, Stafford, Jeffrey Roger, Mao, Crusoe Xiadong, Black, Glenn
• Primary Examiner(s): Lin, Hang
• Application Number:
  US15/337,469
• Publication Number:
  US 20170045941A1
• Time in Patent Office: 1,229 Days
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/32 using local area network [LAN] connections
• CPC Class Codes: A63F 13/53 involving additional visual information provided to the game scene, e.g. by overlay to simulate a head-up display [HUD] or displaying a laser sight in a shooting game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/6615 using models with different levels of detail [LOD]
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

A method is provided, including the following method operations: receiving captured images of an interactive environment in which a head-mounted display (HMD) is disposed; receiving inertial data processed from at least one inertial sensor of the HMD; analyzing the captured images and the inertial data to determine a current and predicted future location of the HMD; using the predicted future location of the HMD to adjust a beamforming direction of an RF transceiver towards the predicted future location of the HMD; tracking a gaze of a user of the HMD; generating image data depicting a view of a virtual environment for the HMD, wherein regions of the view are differentially rendered; generating audio data depicting sounds from the virtual environment, the audio data being configured to enable localization of the sounds by the user; transmitting the image data and the audio data via the RF transceiver to the HMD.

Claims

1. A method, comprising:

receiving captured images of an interactive environment in which a head-mounted display (HMD) is disposed;

receiving inertial data processed from at least one inertial sensor of the HMD;

analyzing the captured images of the interactive environment and the inertial data to determine a current location of the HMD and a predicted future location of the HMD;

using the predicted future location of the HMD to adjust a beamforming direction of an RF transceiver in a direction that is towards the predicted future location of the HMD;

tracking a gaze of a user of the HMD, wherein tracking the gaze of the user includes capturing images of an eye of the user by a gaze tracking camera in the HMD;

generating video depicting a view of a virtual environment for the HMD, wherein regions of the view are differentially rendered based on the tracked gaze of the user;

tracking a trajectory of the gaze of the user over a predetermined period of time, wherein tracking the trajectory of the gaze uses the captured images of the eye of the user;

tracking a trajectory of the HMD over the predetermined period of time;

predicting, while tracking the trajectory of the gaze, a movement of the gaze of the user to a predicted future region where the user will look next in the virtual environment based on analyzing a trend in the tracked trajectory of the gaze of the user and based on analyzing a trend in the tracked trajectory of the HMD;

wherein the regions of the view are differentially rendered based on the predicted movement of the gaze of the user, wherein the predicted future region starts to render before the gaze of the user is at the predicted future region;

generating audio data depicting sounds from the virtual environment, the audio data being configured to enable localization of the sounds by the user when rendered to headphones that are connected to the HMD;

wirelessly transmitting the video and the audio data via the RF transceiver to the HMD using the adjusted beamforming direction.

2. The method of claim 1, wherein a region of the view towards which the gaze of the user is directed is rendered at a higher image quality setting than other regions of the view, the other regions of the view being rendered at a lower image quality setting to reduce a size of the video.

3. The method of claim 2, wherein the image quality setting includes one or more of an update frequency, resolution, complexity of imagery, or a rendering order value that determines an order for rendering the regions of the view.

4. The method of claim 1, wherein generating the audio data includes determining one or more emanating locations in the virtual environment for the sounds, wherein the audio data is configured to simulate the sounds as originating from the one or more emanating locations when rendered to the headphones.

5. The method of claim 1, wherein generating the audio data uses an HRTF that is identified for the user.

6. The method of claim 1, wherein generating the audio data is based on the current and/or predicted future location of the HMD.

7. The method of claim 1, wherein analyzing the captured images and the inertial data includes identifying movement of the HMD, the predicted future location of the HMD being determined using the identified movement of the HMD.

8. The method of claim 7,

wherein identifying movement of the HMD includes determining a motion vector of the HMD, the predicted future location of the HMD being determined by applying the motion vector of the HMD to a current location of the HMD;

wherein a magnitude of the motion vector identifies a speed of the movement of the HMD, and wherein a direction of the motion vector identifies a direction of the movement of the HMD.

9. The method of claim 8, further comprising:

adjusting an angular spread of the RF transceiver based on the speed of the movement of the HMD;

wherein the angular spread increases with increasing speed of the movement of the HMD.

10. The method of claim 1, wherein analyzing the captured images and the inertial data includes identifying movement of the HMD, the predicted future location of the HMD being determined using the identified movement of the HMD.

11. A method, comprising:

receiving captured images of an interactive environment in which a head-mounted display (HMD) is disposed;

receiving inertial data processed from at least one inertial sensor of the HMD;

analyzing the captured images of the interactive environment and the inertial data to determine a current location of the HMD and a predicted future location of the HMD;

using the predicted future location of the HMD to adjust a beamforming direction of an RF transceiver in a direction that is towards the predicted future location of the HMD;

tracking a gaze of a user of the HMD, wherein tracking the gaze of the user includes capturing images of an eye of the user by a gaze tracking camera in the HMD;

generating video depicting a view of a virtual environment for the HMD, wherein regions of the view are differentially rendered based on the tracked gaze of the user;

tracking a trajectory of the gaze of the user over a predetermined period of time, wherein tracking the trajectory of the gaze uses the captured images of the eye of the user;

tracking a trajectory of the HMD over the predetermined period of time;

predicting, while tracking the trajectory of the gaze, a movement of the gaze of the user to a predicted future region where the user will look next in the virtual environment based on analyzing a trend in the tracked trajectory of the gaze of the user and based on analyzing a trend in the tracked trajectory of the HMD;

wherein the regions of the view are differentially rendered based on the predicted movement of the gaze of the user, wherein the predicted future region starts to render before the gaze of the user is at the predicted future region;

transmitting the video via the RF transceiver to the HMD using the adjusted beamforming direction.

12. The method of claim 11, wherein a region of the view towards which the gaze of the user is directed is rendered at a higher image quality setting than other regions of the view, the other regions of the view being rendered at a lower image quality setting to reduce a size of the video.

13. The method of claim 12, wherein the image quality setting includes one or more of an update frequency, resolution, complexity of imagery, or a rendering order value that determines an order for rendering the regions of the view.

14. The method of claim 11, wherein analyzing the captured images and the inertial data includes identifying movement of the HMD, the predicted future location of the HMD being determined using the identified movement of the HMD.

15. A method, comprising:

receiving captured images of an interactive environment in which a head-mounted display (HMD) is disposed;

receiving inertial data processed from at least one inertial sensor of the HMD;

analyzing the captured images of the interactive environment and the inertial data to determine a current location of the HMD and a predicted future location of the HMD;

using the predicted future location of the HMD to adjust a beamforming direction of an RF transceiver in a direction that is towards the predicted future location of the HMD;

generating audio data depicting sounds from the virtual environment, the audio data being configured to enable localization of the sounds by a user when rendered to headphones that are connected to the HMD;

transmitting the audio data via the RF transceiver to the HMD using the adjusted beamforming direction;

tracking a trajectory of a gaze of the user over a predetermined period of time, wherein tracking the trajectory of the gaze uses captured images of an eye of the user captured by a gaze tracking camera in the HMD;

predicting, while tracking the trajectory of the gaze, a movement of the gaze of the user to a predicted future region where the user will look next in the virtual environment based on analyzing a trend in the tracked trajectory of the gaze of the user;

wherein regions of a view of a virtual environment for the HMD are differentially rendered based on the predicted movement of the gaze of the user, wherein the predicted future region starts to render before the gaze of the user is at the predicted future region.

16. The method of claim 15, wherein generating the audio data includes determining one or more emanating locations in the virtual environment for the sounds, wherein the audio data is configured to simulate the sounds as originating from the one or more emanating locations when rendered to the headphones.

17. The method of claim 15, wherein generating the audio data uses an HRTF that is identified for the user.

18. The method of claim 15, wherein generating the audio data is based on the current and/or predicted future location of the HMD.