Game streaming with spatial audio

• US 10,123,150 B2
• Filed: 01/31/2017
• Issued: 11/06/2018
• Est. Priority Date: 01/31/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kim, Ryan Y., Manion, Todd R., Heitkamp, Robert N., Wilssens, Steven, Edry, Philip A.
• Primary Examiner(s): Yu, Norman
• Application Number:
  US15/420,840
• Publication Number:
  US 20180220255A1
• Time in Patent Office: 644 Days
• Field of Search: 381 11, 381 12, 381300, 381303, 381306-307, 381309, 381 56, 381 58- 59, 381 81, 381 85, 381 99-100, 345633, 463 33
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 19/167 Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes
• CPC Class Codes: G10L 19/173 Transcoding, i.e. converting between two coded representations avoiding cascaded coding-decoding
• CPC Class Codes: H04N 21/437 Interfacing the upstream path of the transmission network, e.g. for transmitting client requests to a VOD server flow control in data networks H04L47/10; streaming protocols, e.g. RTP or RTCP, H04L65/65; scheduling or organising the servicing of application requests in data packet switching networks H04L67/60
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/4398 involving reformatting operations of audio signals details of audio signal transcoding G10L19/173
• CPC Class Codes: H04N 21/4781 Games
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/11 Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/308 Electronic adaptation dependent on speaker or headphone connection

Abstract

A game engine may generate video and audio content on a per-frame basis. Audio data corresponding to a current frame may be generated to comprise sound-field information independent of a speaker configuration or spatialization technology that may be used to play the associated audio. The sound-field may be generated based on monaural audio data corresponding to a sound produced by an in-game object at the object'"'"'s position as of the current frame. The sound-field information may be transmitted to a remote computing device for reproduction using a selected, available speaker configuration and spatialization technology.

Claims

1. A computer-implemented method, comprising:

generating a monaural audio stream data associated with a game object, the game object associated with a frame of content;

generating audio data associated with the frame of content, comprising sound-field information based at least in part on the monaural audio stream and a position of the object in the frame of content;

processing, at a first computing device, the audio data to convert the audio data to a first audio signal in a first spatialization format compatible with a first audio device; and

sending the audio data to a remote computing device over a network, wherein the remote computing device processes and converts the audio data to a second audio signal in a second spatialization format compatible with a second audio device.

2. The computer-implemented method of claim 1, wherein the sound-field information comprises audio at a location in the sound-field corresponding to the position of the object in the frame of content.

3. The computer-implemented method of claim 1, wherein the sound-field information is speaker-independent.

4. The computer-implemented method of claim 1, wherein the sound-field information comprises directional information.

5. The computer-implemented method of claim 1, wherein the first spatialization format is channel-based and the second spatialization format is object-based.

6. The computer-implemented method of claim 1, further comprising:

receiving information indicative of a speaker configuration of the first audio device; and

selecting the first spatialization format based at least in part on the speaker configuration.

7. The computer-implemented method of claim 1, wherein the audio data is generated independently of a speaker configuration of the second audio device.

8. The computer-implemented method of claim 1, further comprising:

generating the sound-field information based at least in part on at least one of direction, elevation, or distance of the game object associated with the frame of content.

9. The computer-implemented method of claim 1, wherein audio playback by the second audio device is based at least in part on a head-related transfer function.

10. A system comprising:

at least one processor;

a memory having stored thereon instructions that, when executed by the at least one processor, cause the system to;

generate a monaural audio stream data associated with a game object, the game object associated with a frame of content;

generate audio data associated with the frame of content, comprising sound-field information based at least in part on the monaural audio stream and a position of the object in the frame of content;

process the audio data to convert the audio data to a first audio signal in a first spatialization format compatible with a first audio device; and

send the audio data to a remote computing device over a network, wherein the remote computing device processes and converts the audio data to a second audio signal in a second spatialization format compatible with a second audio device.

11. The system of claim 10, wherein the sound-field information comprises audio at a location in the sound-field corresponding to the position of the object in the frame of content.

12. The system of claim 10, wherein the sound-field information is speaker-independent.

13. The system of claim 10, wherein the sound-field information comprises directional information.

14. The system of claim 10, wherein the first spatialization format is channel-based and the second spatialization format is object-based.

15. The system of claim 10, wherein the memory has further instructions stored thereon to:

receive information indicative of a speaker configuration of the first audio device; and

select the first spatialization format based at least in part on the speaker configuration.

16. The system of claim 10, wherein the audio data is generated independently of a speaker configuration of the second audio device.

17. The system of claim 10, wherein the memory has further instructions stored thereon to:

generate the sound-field information based at least in part on at least one of direction, elevation, or distance of the game object associated with the frame of content.

18. system of claim 10, wherein audio playback by the second audio device is based at least in part on a head-related transfer function.

19. A non-transitory computer readable storage medium having stored thereon instructions that, upon execution by a computing device, cause the computing device to at least:

generate a monaural audio stream data associated with a game object, the game object associated with a frame of content;

generate audio data associated with the frame of content, comprising sound-field information based at least in part on the monaural audio stream and a position of the object in the frame of content;

process the audio data to convert the audio data to a first audio signal in a first spatialization format compatible with a first audio device; and

send the audio data to a remote computing device over a network, wherein the remote computing device processes and converts the audio data to a second audio signal in a second spatialization format compatible with a second audio device.

20. The non-transitory computer readable storage medium of claim 19, wherein the sound-field information comprises audio at a location in the sound-field corresponding to the position of the object in the frame of content.