SPECTATOR AUDIO AND VIDEO REPOSITIONING

• US 20180220252A1
• Filed: 06/07/2017
• Published: 08/02/2018
• Est. Priority Date: 01/31/2017
• Status: Active Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: EDRY, Philip Andrew, MANION, Todd R., HEITKAMP, Robert Norman, WILSSENS, Steven
• Application Number:
  US15/616,883
• Publication Number:
  US 20180220252A1
• Time in Patent Office: Days
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/11 Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

Participants can control a number of aspects of a virtual reality session. A participant of the session can control the position of an object, such as an avatar. Spectators do not have control over aspects of a session. For instance, spectators cannot control the position of objects or change properties of objects within a virtual environment. In some configurations, the position of a spectator'"'"'s viewing area is based on the position of an object that is controlled by a participant. In some embodiments, a spectator'"'"'s viewing area can follow a participant'"'"'s position but the spectator can look in any direction from that position. By following the participant'"'"'s position, spectators can follow the action of a session yet have the freedom to control the direction of their viewing area to enhance their viewing experience. Customized spatial audio is also generated for the spectator based on the direction of their viewing area.

Claims

1. A computing device, comprising:

a processor;

a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing device to

receive session data defining a virtual reality environment comprising a participant object, the session data allowing a participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,

generate a first view for display to the participant, the first view originating from the location of the participant object controlled by the participant, wherein a direction of the first view is based on the direction of the participant object,

generate a spectator view for display on a computing device associated with a spectator, the spectator view originating from the location of the participant object controlled by the participant, the session data allowing the spectator to provide a spectator input for controlling a direction of the spectator view, and

generate a spectator audio output signal of a stream, wherein the spectator audio output signal causes an output device to emanate an audio output of the stream from a speaker object location positioned relative to the spectator, the speaker object location based on the direction of the spectator view and the location of an audio object relative to the location of the participant object.

2. The system of claim 1, wherein the direction of the spectator view is independent of the direction of the participant object, wherein the instructions further cause the system to generate a participant audio output signal of the stream based on the location of the participant object, the direction of the participant object, and the location of the audio object.

3. The system of claim 1, wherein generating the spectator audio output signal comprising configuring the spectator audio output in accordance with an Ambisonics-based technology, wherein the spectator output signal defines at least one sound field modeling the location of the audio object associated with the stream, wherein the data defining the sound field can be interpreted by the computing device associated with the spectator for causing the output device to emanate the audio output of the stream from the speaker object location.

4. The system of claim 1, wherein generating the spectator audio output signal comprising configuring the spectator audio output in accordance with a channel-based audio technology, wherein the spectator audio output signal causes the stream to render to an output device in a channel-based audio format.

5. The system of claim 1, wherein generating the spectator audio output signal comprising configuring the spectator audio output in accordance with an object-based technology, wherein the spectator output signal defines the location of the audio object associated with the stream, the location defined in a three-dimensional coordinate system.

6. The system of claim 1, wherein the output device comprises one or more speakers in communication with the computing device, wherein the instructions further cause the system to transmit the spectator audio output to the computing device, wherein the spectator audio output causes the output device in communication with the computing device to emanate the audio output of the stream from the speaker object location positioned relative to the spectator, the speaker object location modeling the direction of the spectator view and the location of the audio object relative to the location of the participant object.

7. The system of claim 1, wherein the session data is received from a participant computing device, wherein the session data comprises a 360 canvas, and data indicating a direction of the stream, wherein one or more effects applied to the audio output is based on the data indicating the direction of the stream.

8. A system, comprising:

a processor;

a memory having computer-executable instructions stored thereupon which, when executed by the processor, cause the system to

receive session data defining a virtual reality environment comprising a participant object, the session data allowing a participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,

generate a first view for display to the participant, the first view originating from the location of the participant object controlled by the participant, wherein a direction of the first view is based on the direction of the participant object,

generate a spectator view for display on a computing device associated with a spectator, the spectator view originating from the location of the participant object controlled by the participant, the session data allowing the spectator to provide a spectator input for controlling a direction of the spectator view, and

generate a spectator audio output signal of a stream based on the direction of the spectator view, a location of an audio object associated with the stream, and the location of the participant object.

9. The system of claim 8, wherein the direction of the spectator view is independent of the direction of the participant object, where instructions further cause the system to generate a participant audio output signal of the stream based on the location of the participant object, the direction of the participant object, and the location of the audio object.

10. The system of claim 8, wherein generating the spectator audio output signal comprises generating the audio output signal defining an Ambisonics representation.

11. The system of claim 8, wherein generating the spectator audio output signal is processed in accordance with a channel-based audio technology.

12. The system of claim 8, wherein generating the spectator audio output signal is processed in accordance with an object-based technology.

13. The system of claim 8, wherein the output device comprises one or more speakers in communication with the computing device, wherein the instructions further cause the system to transmit the spectator audio output to the computing device, wherein the spectator audio output causes the output device in communication with the computing device to emanate the audio output of the stream from the speaker object location positioned relative to the spectator, the speaker object location modeling the direction of the spectator view and the location of the audio object relative to the location of the participant object.

14. The system of claim 8, wherein the session data is received from a participant computing device, wherein the session data comprises a 360 canvas, and data indicating a direction of the stream, wherein one or more effects applied to the audio output is based on the data indicating the direction of the stream.

15. A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by one or more processors of a system, cause the one or more processors of the system to:

receive session data defining a virtual reality environment comprising a participant object, the session data allowing the participant to provide a participant input for controlling a location of the participant object and a direction of the participant object,

generate a first view for display to the participant, the first view originating from the location of the participant object controlled by the participant, wherein a direction of the first view is based on the direction of the participant object,

generate a spectator view for display on a computing device associated with a spectator, the spectator view originating from the location of the participant object controlled by the participant, the session data allowing the spectator to provide a spectator input for controlling a direction of the spectator view, and

generate a spectator audio output signal of a stream, wherein the spectator audio output signal causes an output device to emanate an audio output of the stream from a speaker object location positioned relative to the spectator, the speaker object location modeling the direction of the spectator view and the location of an audio object relative to the location of the participant object.

16. The computer-readable storage medium of claim 15, wherein the direction of the spectator view is independent of the direction of the participant object, wherein the instructions further cause the system to generate a participant audio output signal of the stream based on the location of the participant object, the direction of the participant object, and the location of the audio object.

17. The computer-readable storage medium of claim 15, wherein generating the spectator audio output signal comprises configuring the spectator audio output in accordance with an Ambisonics-based technology, wherein the spectator output signal defines at least one sound field modeling the location of the audio object associated with the stream, wherein the data defining the sound field can be interpreted by the computing device associated with the spectator for causing the output device to emanate the audio output of the stream from the speaker object location.

18. The computer-readable storage medium of claim 15, wherein generating the spectator audio output signal comprises configuring the spectator audio output in accordance with a channel-based audio technology, wherein the spectator audio output signal causes the stream to render to an output device in a channel-based audio format.

19. The computer-readable storage medium of claim 15, wherein generating the spectator audio output signal comprises configuring the spectator audio output in accordance with an object-based technology, wherein the spectator output signal defines the location of the audio object associated with the stream, the location defined in a three-dimensional coordinate system.

20. The computer-readable storage medium of claim 15, wherein the output device comprises one or more speakers in communication with the computing device, wherein the instructions further cause the system to transmit the spectator audio output to the computing device, wherein the spectator audio output causes the output device in communication with the computing device to emanate the audio output of the stream from the speaker object location positioned relative to the spectator, the speaker object location modeling the direction of the spectator view and the location of the audio object relative to the location of the participant object.