Interactive spatalized audiovisual system

• US 20060212147A1
• Filed: 01/09/2003
• Published: 09/21/2006
• Est. Priority Date: 01/09/2002
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: McGrath, David S., Bennett, Stephen James, Heckenberg, Daniel
• Granted Patent: US 7,533,346 B2
• Time in Patent Office: Days
• US Class Current: 700/94
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04M 3/56 Arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities video conference systems H04N7/15
• CPC Class Codes: H04M 3/564 whereby the feature is a sub-conference
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02

Abstract

An interactive spatialized audiovisual system links a plurality of remote used terminals. The system comprises a networked computer having an associated user database including user status information. Input means are provided at the computer for receiving a plurality of audio streams and associated locating data from the remote user terminals for, virtually locating the users relative to one another within a virtual user environment such as a chat room environment Selection means are provided for enabling selection of at least the first group of the audio streams in a first selection process based on status information in the user database. Output means output the selected group of audio streams and associated locating data for spatialization of the audio streams relative to a first listener-based audio reference frame which is substantially coherent with visual representations of the audio sources defined by the locating data at the first user terminal. Merging means are provided for merging at least some of the audio streams into a merged audio stream for transmittal to the first and other user terminal, with the merged audio stream being spatialized so as to provide for a spatialized background audio effect in the audio reference frame at the user terminal.

Claims

1. An interactive spatialized audiovisual system for linking a plurality of remote user terminals, the system comprising:

a networked computer;

an associated user database including user status information;

input means for receiving at the computer a plurality of audio streams and associated locating data from the remote user terminals for virtually locating the users relative to one another within a virtual user environment;

selection means for enabling selection of at least a first group of the audio streams in a first selection process based on status information in the user database;

output means for outputting the selected group of audio streams and associated locating data for spatialization of the selected group of audio streams relative to a first listener-based audio reference frame which is substantially coherent with visual representations of the audio sources defined by the locating data at a first user terminal.

2. A system according to claim 1, further comprising first spatialization means for spatializing the selected group of audio streams.

3. A system according to claim 1, further comprising merging means for merging at least some of the audio streams into a merged audio stream for transmittal to the user terminal, and second spatializing means for spatializing the merged stream so as to provide for a spatialized background audio effect in the audio reference frame at the user terminal.

4. A system according to claim 1, wherein the selection means are arranged to select different groups of audio streams according to different selection processes based on the user status information in the user database, for transmission to the corresponding user terminals.

5. A system according to claim 1, wherein the user status information is chosen from a group including user location data which serves to locate the user in the virtual environment, user orientation data which serves to orientate the user both with respect to the other users and to the virtual environment, user listening status information, and user talking status information.

6. A system according to claim 5, wherein the user listening status information is arranged to allow the user to listen to other selected users or groups in the environment.

7. A system according to claim 5, wherein the selection means includes a selector for selecting M closest audio sources from N audio sources.

8. A system according to claim 5, wherein the selection means includes a selector for enabling the selection of M loudest sources based on at least one of the following, namely the amplitude of the source signal and the distance of the source from the listener.

9. A system according to claim 5, wherein the selection means includes a selector for enabling a user-driven selection process determined by the subject user or other users.

10. A system according to claim 5, wherein the selection means includes a selector for enabling a moderator-driven selection process in which a “

moderator”

in the environment is able to control the talking and listening status of the other users.

11. A system according to claim 5, in which the selection means includes a selector for enabling a selection based on the geography or topology of the virtual environment, in which features of the environment are arranged realistically to affect the listening capability of users in the environment, so as to provide a coherent visual and sonic landscapes.

12. A system according to claim 11, wherein the features include barriers defining one or more chat rooms, with at least some of the audio streams in a particular room being summed or merged and spatialized to achieve a background reverberation effect characteristic of that particular room.

13. A system according to claim 11, further comprising means for merging and spatialising the audio streams in adjoining rooms or areas to create “

threshold”

effects at entrance/exit points representations of the combined noise emanating from said adjoining room or area.

14. A system according claim 11, further comprising means for generating “

dry”

room signals using summed non-reverberated audio sources.

15. A system according to claim 11, further comprising means for generating “

wet”

room signals using summed reverberated audio sources.

16. A system according to claim 2, wherein the first spatialization means is arranged to process selected groups of audio streams at the networked computer to derive spatialized audio streams for onward transmission to at least the first selected user terminal.

17. A system according to claim 2, wherein the first spatialization means are provided at each of the user terminals for processing of selected groups of audio streams from the networked computer.

18. A system according to claim 2, wherein the first spatialization means includes a binaural processor.

19. A system according to claim 3, wherein the second spatializing means is arranged to process the merged group of audio streams at the networked computer to derive a spatialized merged audio stream for onward transmission to at least the first selected user terminal.

20. A system according to claim 3, wherein the second spatialization means are provided at each of the user terminals for spatializing merged groups of audio streams at each user terminal.

21. A system according to claim 3, wherein the second spatialization means includes a binaural reverberation processor.

32. A system according to claim 1, wherein the virtual user environment is a chat room environment.

39. A system according to claim 5, wherein the selection means includes at least one selector from the group consisting of:

a first selector for selecting M closest audio sources from N audio sources,

a second selector for enabling the selection of M loudest sources based on at least one of the following, namely the amplitude of the source signal and the distance of the source from the listener,

a third selector for enabling a user-driven selection process determined by the subject user or other users,

a fourth selector for enabling a moderator-driven selection process in which a “

moderator”

in the environment is able to control the talking and listening status of the other users, and

a fifth selector for enabling a selection based on the geography or topology of the virtual environment, in which features of the environment are arranged realistically to affect the listening capability of users in the environment, so as to provide a coherent visual and sonic landscapes.

22. A method of providing an interactive spatialized audiovisual facility comprising:

receiving from a plurality of user-based audio sources a plurality of corresponding audio streams and associated locating data capable of virtually locating the audio sources relative to one another within a virtual environment;

determining user status data;

selecting at least some of the audio streams based on the user status data;

transmitting the locating data and selected audio streams to a first listener destination for enabling the display of visual representations of the virtual locations of at least some of the audio sources within the virtual environment; and

spatializing the selected audio streams relative to a first listener-based audio reference frame which is substantially coherent with the visual representations of the audio sources either before or after the audio streams are transmitted to the first listener destination.

23. A method according to claim 22, further comprising:

enabling the user status data to be altered,

reading the altered user status data, and

selecting at least one of the audio streams based on the altered user status data,

wherein at least one of the audio streams selected using the altered user status data is different to the prior selected streams.

24. A method according to claim 22, which includes the steps of:

merging at least some of the audio streams into at least one merged audio stream,

transmitting the merged audio stream to the first listener destination, and

spatializing the merged audio stream either before or after transmitting it so as to provide a background audio effect within the virtual environment.

25. A method according to claim 24, in which the merged audio stream includes audio streams which have not been individually selected.

26. A method of providing an interactive spatialized audiovisual facility comprising:

receiving from a plurality of user-based audio sources a plurality of corresponding audio streams and associated locating data capable of virtually locating the audio sources relative to one another within a virtual environment;

determining user status data;

selecting at least some of the audio streams based on the user status data in a first selection process;

transmitting the selected audio streams and associated locating data to a first listener destination for enabling the display of visual representations of the virtual locations of at least some of the selected audio sources within the virtual environment;

spatializing the selected audio streams relative to a first listener-based audio reference frame which is substantially coherent with the visual representations of the audio sources either before or after the transmitting said streams;

selecting at least some of the audio streams in a second selection process; and

transmitting the selected audio streams and associated locating information to a second listener destination for enabling the display of visual representations of the locations of at least the selected audio sources, and spatializing at the second listener destination the selected audio streams in an audio reference frame which is substantially coherent with the visual representations of the audio sources, either before or after transmitting said streams.

27. A method according to claim 26, wherein multiple selection processes are used to select the audio streams according to at least one predetermined algorithm, the selected audio streams and associated locating information are transmitted to multiple listener destinations, and visible representations of the locations of at least the selected audio streams are displayed at the multiple listener destinations, with each of the selected audio streams being spatialized at the multiple listener destinations or before they are transmitted thereto in audio reference frames which are substantially coherent with the visible representations of the audio sources.

28. A system for providing for spatialized conversation over a network environment, the system comprising:

a plurality of user terminals;

a computer network capable of streaming audio streams to the user terminals, each of the audio streams including associated spatialization information;

a rendering system for rendering the audio streams to predetermined virtual locations around a user using the associated spatialization information; and

a user interface for virtually spatially locating a user amongst the audio streams;

wherein the rendering system spatializes the audio streams so as to maintain a substantially spatially coherent audio reference frame around the user, the user interface includes a visual indicator of the spatial position of each of the audio streams around a listener and the rendering system maintains a substantially spatially coherent audio reference frame relative to the visual indicator.

29. A system according to claim 28, wherein each stream includes user ownership information and the system includes an audio stream access interface for granting access to the audio streams.

30. A system according to claim 28, wherein the rendering system includes an attenuator to attenuate audio sources located virtually remotely from a current user and a merger to merge audio sources located virtually remotely from a current user.

31. A system according to claim 30, wherein the rendering system is located adjacent a user and the audio sources are streamed over a computer network.

33. A computer-readable medium having stored thereon executable instructions for causing a computer to provide an interactive spatialized audiovisual facility, the instructions being arranged to:

receive from a plurality of user-based audio sources a plurality of corresponding audio streams and associated locating data capable of virtually locating the audio sources relative to one another within a virtual environment;

determine user status data;

select at least some of the audio streams based on the user status data;

transmit the locating data and selected audio streams and associated to a first listener destination for enabling the display of visual representations of the virtual locations of at least some of the audio sources within the virtual environment, and

spatialize the selected audio streams relative to a first listener-based audio reference frame which is substantially coherent with the visual representations of the audio sources.

34. A computer-readable medium having stored thereon executable instructions for causing a computer to provide an interactive spatialized audiovisual facility, the instructions being arranged to:

receive from a plurality of user-based audio sources a plurality of corresponding audio streams and associated locating data capable of virtually locating the audio sources relative to one another within a virtual environment;

determine user status data;

select at least some of the audio streams based on the user status data in a first selection process;

transmit the selected audio streams and associated locating data to a first listener destination for enabling the display of visual representations of the virtual locations of at least some of the selected audio sources within the virtual environment;

spatialize the selected audio streams relative to a first listener-based audio reference frame which is substantially coherent with the visual representations of the audio sources;

select at least some of the audio streams in a second selection process; and

transmit the selected audio streams and associated locating information to a second listener destination for enabling the display at the second listener destination of visual representations of the locations of at least the selected audio sources, and for spatializing at the second listener destination the selected audio streams in an audio reference frame which is substantially coherent with the visual representations of the audio sources.

35. A method of operating an interactive spatialized audio facility including a networked computer and a plurality of user terminals linked to the networked computer, the method comprising:

transmitting from a user terminal to the networked computer an audio stream generated by the user and associated locating data capable of virtually locating the audio stream generated by the user within a virtual environment for selective combination with corresponding audio streams, associated locating data and user status data at the networked computer;

receiving at the user terminal a plurality of audio streams selected on the basis of the user status data and associated locating data for virtually locating the users relative to one another within a virtual user environment;

generating at the user terminal visual representations of the locating data, and

spatializing the selected group of audio streams relative to a user based audio reference frame which is substantially coherent with the visual representations of the audio sources of the users as defined by the locating data for playback to the user.

36. A method according to claim 35, further comprising receiving at the user terminal a merged audio stream which is spatialized before or after receipt thereof to provide a spatialized background audio effect in the audio reference frame at the user terminal for playback to the user.

37. A computer-readable medium having stored thereon executable instructions for causing a computer to provide or operate an interactive spatialized audiovisual facility, the instructions including program segments arranged to implement a method, the method comprising:

receiving from a plurality of user-based audio sources a plurality of corresponding audio streams and associated locating data capable of virtually locating the audio sources relative to one another within a virtual environment;

determining user status data;

selecting at least some of the audio streams based on the user status data;

transmitting the locating data and selected audio streams to a first listener destination for enabling the display of visual representations of the virtual locations of at least some of the audio sources within the virtual environment; and

spatializing the selected audio streams relative to a first listener-based audio reference frame which is substantially coherent with the visual representations of the audio sources either before or after the audio streams are transmitted to the first listener destination.

38. A computer-readable medium having stored thereon executable instructions for causing a computer to provide or operate an interactive spatialized audiovisual facility, the instructions including program segments arranged to implement a method, the method comprising:

transmitting from a user terminal to the networked computer an audio stream generated by the user and associated locating data capable of virtually locating the audio stream generated by the user within a virtual environment for selective combination with corresponding audio streams, associated locating data and user status data at the networked computer;

receiving at the user terminal a plurality of audio streams selected on the basis of the user status data and associated locating data for virtually locating the users relative to one another within a virtual user environment;

generating at the user terminal visual representations of the locating data, and

spatializing the selected group of audio streams relative to a user based audio reference frame which is substantially coherent with the visual representations of the audio sources of the users as defined by the locating data for playback to the user.