Processing audio data

• US 7,876,914 B2
• Filed: 05/23/2005
• Issued: 01/25/2011
• Est. Priority Date: 05/21/2004
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Adams, Guy de Warrenne Bruce, Grosvenor, David Arthur
• Primary Examiner(s): Chin; Vivian
• Assistant Examiner(s): MONIKANG, GEORGE C
• Application Number:
  US11/135,556
• Publication Number:
  US 20050281410A1
• Time in Patent Office: 2,073 Days
• Field of Search: 381/56, 381/58, 381/61, 381/77, 381/122, 381/310, 381/26, 381/92, 381/95, 345/418
• US Class Current: 381/92
• CPC Class Codes: H04H 60/04 Studio equipment; Interconnection of studios
• CPC Class Codes: H04H 60/47 for recognising genres

Abstract

An exemplary embodiment is a method of processing audio data comprising: characterising an audio data representative of a recorded sound scene into a set of sound sources occupying positions within a time and space reference frame; analysing the sound sources; and generating a modified audio data representing sound captured from at least one virtual microphone configured for moving about the recorded sound scene, wherein the virtual microphone is controlled in accordance with a result of the analysis of said audio data, to conduct a virtual tour of the recorded sound scene.

Claims

1. A method of processing audio data, said method comprising:

characterizing, using a processor, an audio data representative of a recorded sound scene into a set of sound sources occupying positions within a time and space reference frame;

analyzing said set of sound sources of the audio data;

selecting a subset of sound sources from the set of sound sources of the audio data based on a result of the analysis;

determining at least one virtual microphone trajectory using the selected subset of sound sources; and

generating a modified audio data representing sound captured from at least one virtual microphone configured for moving about said recorded sound scene, wherein said virtual microphone is controlled in accordance with the at least one virtual microphone trajectory and the selected subset of sound sources, to conduct a virtual tour of said recorded sound scene.

2. The method as claimed in claim 1, comprising:

identifying characteristic sounds associated with said sound sources; and

controlling said virtual microphone in accordance with said identified characteristic sounds associated with said sound sources.

3. The method as claimed in claim 1, comprising:

normalising said sound signals captured from the at least one virtual microphone by referencing each of said sound signals to a common maximum signal level; and

mapping said sound sources of the audio data to said normalised sound signals.

4. The method as claimed in claim 1, wherein said analysis comprises selecting sound sources which are grouped together within said reference frame.

5. The method as claimed in claim 1, wherein said analysis comprises determining a causality of said sound sources.

6. The method as claimed in claim 1, wherein said analysis comprises recognizing sound sources representing sounds of a similar classification type.

7. The method as claimed in claim 1, wherein said analysis comprises identifying new sounds which first appear in said recorded sound scene and which were not present at an initial beginning time position of said recorded sound scene.

8. The method as claimed in claim 1, wherein said analysis comprises recognizing sound sources which accompany self reference point within said reference frame.

9. The method as claimed in claim 1, wherein said analysis comprises recognizing a plurality of pre-classified types of sounds by comparing a waveform of a said sound source against a plurality of stored waveforms that are characteristic of said pre-classified types.

10. The method as claimed in claim 1, wherein said analysis comprises classifying sounds into sounds of people and non-people sounds.

11. The method as claimed in claim 1, wherein said analysis comprises grouping said sound sources according to at least one criterion selected from the set of:

physical proximity of said sound sources; and

similarity of said sound sources.

12. The method as claimed in claim 1, wherein said generating modified audio data comprises executing an algorithm for determining a trajectory of said virtual microphone followed with respect to said sound sources, during said virtual tour.

13. The method as claimed in claim 1, wherein said generating a modified audio data comprises executing an algorithm for determining a field of reception of said virtual microphone with respect to said sound sources.

14. The method as claimed in claim 1, wherein said generating a modified audio data comprises executing a search algorithm comprising a search procedure for establishing a saliency of said sound sources.

15. The method as claimed in claim 1, wherein said generating a modified audio data comprises a search procedure, based at least partly on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories.

16. The method as claimed in claim 1, wherein said generating a modified audio data comprises a search procedure, based on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories, said search being constrained by at least an allowable duration of a sound source signal output by said generated virtual microphone.

17. The method as claimed in claim 1, wherein said generating a modified audio data comprises a search procedure, based on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories, said search procedure comprising a calculation of:

an intrinsic saliency of said sound sources; and

at least one selected from the set comprising;

a feature-based saliency of said sources; and

a group saliency of a group of said sound sources.

18. The method as claimed in claim 1, wherein said analysis further comprises:

identifying a predefined sound scene class wherein, in that sound scene class, sub-parts of the sound scene have predefined characteristics; and

establishing index audio clips based on recognised sound sources or groups of sound sources.

19. The method as claimed in claim 1, wherein said generating modified audio data comprises executing an algorithm for determining a trajectory and field of listening of said virtual microphone from one sound source or group of sound sources to the next.

20. The method as claimed in claim 1, wherein said analysis further comprises:

identifying a predefined sound scene class wherein, in that sound scene class, sub-parts of the sound scene have predefined characteristics; and

establishing index audio clips based on recognised sound sources or groups of sound sources; and

said process of generating a modified audio data comprises executing an algorithm for determining a trajectory and field of view of said virtual microphone from one sound source or group of sound sources to the next, said algorithm further determining at least one parameter selected from the set comprising;

the order of the index audio clips to be played;

the amount of time for which each index audio clip is to be played; and

the nature of the transition between each of said index audio clips.

21. The method as claimed in claim 1, wherein said generating a modified audio data comprises use of a psychological model of saliency of said sound sources.

22. The method as claimed in claim 1, comprising an additional process of performing a selective editing of said recorded sound scene to generate a modified recorded sound scene, said at least one virtual microphone being configurable to move about in said modified recorded sound scene.

23. The method as claimed in claim 1, wherein generating said virtual microphone comprises a rendering process of placing said virtual microphone in said soundscape and synthesising the sounds that it would capture in accordance with a model of sound propagation in a three dimensional environment.

24. The method as claimed in claim 1, wherein said audio data is associated with an image data and generating said virtual microphone comprises synchronising said virtual microphone with an image content of said image data.

25. The method as claimed in claim 1, wherein said audio data is associated with image data and generating said virtual microphone comprises synchronising said virtual microphone with an image content of said image data, said modified audio data representing said virtual microphone being used to modify the image content for display in conjunction with said generated virtual microphone.

26. The method as claimed in claim 1, wherein said audio data is associated with an image data and generating said virtual microphone comprises synchronising said virtual microphone with identified characteristics of an image content of said image data.

27. The method as claimed in claim 1, further comprising acquiring said audio data representative of said recorded sound scene.

28. The method as claimed in claim 1, wherein said time and space reference frame is moveable with respect to said recorded sound scene.

29. The method as claimed in claim 1, wherein said characterising of audio data comprises determining a style parameter for conducting a search process of said audio data for identifying said set of sound sources.

30. The method as claimed in claim 1, wherein said characterising comprises:

selecting said time and space reference frame from;

a reference frame fixed with respect to said sound scene; and

a reference flame which is moveable with respect to said recorded sound scene.

31. The method as claimed in claim 1, wherein said virtual microphone is controlled to tour said recorded sound scene following a path which is determined as a path which a virtual listener would traverse within said recorded sound scene;

and

wherein said modified audio data represents sound captured from said virtual microphone from a perspective of said virtual listener.

32. The method as claimed in claim 1, wherein said virtual microphone is controlled to conduct a virtual tour of said recorded sound scene, in which a path followed by said virtual microphone is determined from an analysis of sound sources which draw an attention of a virtual listener;

and

said generated modified audio data comprises said sound sources which draw the attention of said virtual listener.

33. The method as claimed in claim 1, wherein the modified audio data includes additional stock sound sources.

34. The method as claimed in claim 1, wherein said virtual microphone is controlled to follow a virtual tour of said recorded sound scene following a path which is determined as a result of aesthetic considerations of viewable objects in an environment coincident with said recorded sound scene;

and

wherein said generated modified audio data represents sounds which would be heard by virtual listener following said path.

35. A method of processing audio data representative of a recorded sound scene, said audio data comprising a set of sound sources each referenced within a spatial reference frame, said method comprising:

identifying, using a processor, characteristic sounds associated with each of said sound sources of the audio data;

selecting individual sound sources according to their from the identified characteristic sounds;

determining at least one virtual microphone trajectory using the selected individual sound sources;

navigating said sound scene to sample said selected individual sound sources based on the virtual microphone trajectory; and

generating a modified audio data comprising said sampled sounds originating from said selected sound sources.

36. The method as claimed in claim 35, wherein said navigating comprises following a multi-dimensional trajectory within said sound scene.

37. The method as claimed in claim 35, wherein:

said selecting comprises determining which individual said sound sources exhibits features which are of interest to a human listener in the context of said sound scene; and

said navigating said sound scene comprises visiting individual said sound sources which exhibit said features which are of interest to a human listener.

38. A method of processing audio data, the method comprising:

resolving, using a processor, an audio signal into a plurality of constituent sound elements, wherein each of said sound elements is referenced to a spatial reference flame;

defining an observation position within said spatial reference frame;

selecting a set of sound elements from said constituent sound elements in accordance, with the observation position;

determining at least one virtual microphone trajectory using the selected set of sound elements; and

generating from said selected sound elements and the at least one virtual microphone trajectory, an edited version of the audio signal representative of sounds experienced by a virtual observer at said observer position within said spatial reference frame.

39. The method as claimed in claim 38, wherein said observer position is moveable within said spatial reference frame.

40. The method as claimed in claim 38, wherein said observer position follows a three dimensional trajectory with respect to said spatial reference frame.

41. A method of processing audio data, said method comprising:

resolving, using a processor, an audio signal into constituent sound elements, wherein each of said constituent sound elements comprises (a) a characteristic sound quality, and (b) a position within a spatial reference frame;

selecting a set of sound elements from the constituent sound elements;

defining a virtual microphone trajectory through said spatial reference frame using the selected set of sound elements; and

generating from the selected set of sound elements and the defined virtual microphone trajectory, an output audio signal which varies in time.

42. A method of processing audio data, said method comprising:

acquiring a set of audio data representative of a recorded sound scene;

characterizing, using a processor, said audio data into a set of sound sources occupying positions within a time and space reference frame;

identifying characteristic sounds with said of the sound sources;

selecting a subset of sound sources from the set of sound sources based on the identified characteristic sounds of the sound sources;

determining at least one virtual microphone trajectory using the selected subset of sound sources; and

generating a modified audio data representing sound captured from at least one virtual microphone configured for moving around said recorded sound scene, wherein said virtual microphone is controlled in accordance with associated with said the at least one virtual microphone trajectory and the selected subset of the sound sources, to conduct a virtual tour of said recorded sound scene.

43. A computer system comprising an audio data processing means, a data input port and an audio data output port, said audio data processing means being arranged to:

receive from said data input port, a set of audio data representative of a recorded sound scene, said audio data characterised into a set of sound sources positioned within a time-space reference frame;

perform an analysis of said audio data to identify characteristic sounds of the said sound sources;

select a subset of sound sources from the set of sound sources based on the identified characteristic sounds of the sound sources;

determine at least one virtual microphone trajectory using the selected subset of sound sources;

generate a set of modified audio data, said modified audio data representing sound captured from at least one virtual microphone configurable to move about said recorded sound scene; and

output said modified audio data to said data output port,

wherein said virtual microphone is generated in accordance with, and is controlled by the at least one virtual microphone trajectory and the selected subset of the sound sources.

44. A computer system as claimed in claim 43, wherein said performing an analysis of said audio data comprises recognizing a plurality of pre-classified types of sounds by comparing a waveform of a said sound source against a plurality of stored waveforms that are characteristic of said pre-classified types.

45. A computer system as claimed in claim 43, wherein said performing an analysis of said audio data comprises classifying sounds into sounds of people and non-people sounds.

46. A computer system as claimed in claim 43, wherein said analysis of said sound sources comprises grouping said sound sources according to at least one criterion selected from the set of:

physical proximity of said sound sources; and

similarity of said sound sources.

47. A computer system as claimed in claim 43, comprising an algorithm for determining a trajectory of said virtual microphone with respect to said sound sources.

48. A computer system as claimed in claim 43, comprising an algorithm for determining a field of view of said virtual microphone with respect to said sound sources.

49. A computer system as claimed in claim 43, a search algorithm for performing a search procedure for establishing the saliency of said sound sources.

50. A computer system as claimed in claim 43, comprising a search algorithm for performing a search procedure, based at least partly on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories.

51. A computer system as claimed in claim 43, comprising an algorithm for performing a search procedure, based on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories, said search being constrained by at least the allowable duration of a sound source signal output by said generated virtual microphone.

52. A computer system as claimed in claim 43, wherein said generating said modified audio data comprises a search procedure, based on the saliency of said sound sources, to determine a set of possible virtual microphone trajectories, said search procedure comprising a calculation of:

an intrinsic saliency of said sound sources; and

at least one selected from the set comprising;

a feature based saliency of said sources; and

a group saliency of a group of said sound sources.

53. A computer system as claimed in claim 43, wherein said performing an analysis of said audio data further comprises:

identifying a predefined sound scene class wherein, in that sound scene class, sub-parts of the sound scene have predefined characteristics; and

establishing index audio clips based on recognised sound sources or groups of sound sources, and said generating said modified audio data comprises executing an algorithm for determining a trajectory and field of view of said virtual microphone from one sound source or group of sound sources to another sound source or group of sound sources.

54. A computer system as claimed in claim 43, wherein performing an analysis of said audio data further comprises:

identifying a predefined sound scene class wherein, in that sound scene class, sub-parts of the sound scene have predefined characteristics; and

establishing index audio clips based on recognised sound sources or groups of sound sources, said generating modified audio data comprising executing an algorithm for determining a trajectory and field of view of said virtual microphone from one sound source or group of sound sources to the next, said algorithm further determining at least one parameter from the set comprising;

an order of the index audio clips to be played;

an amount of time for which each index audio clip is to be played; and

a nature of a transition between each of said index audio clips.

55. A computer system as claimed in claim 43, wherein said generating modified audio comprises use of a psychological model of saliency of said sound sources.

56. A computer system as claimed in claim 43, wherein said audio data processing means is configured to perform a selective editing of said recorded sound scene to generate a modified recorded sound scene, said at least one virtual microphone being configurable to move about therein.

57. A computer system as claimed in claim 43, wherein generating said virtual microphone comprises a rendering process of placing said virtual microphone in said soundscape and synthesising the sounds that it would capture in accordance with a model of sound propagation in a three dimensional environment.

58. A computer system as claimed in claim 43, wherein said audio data is associated with image data and generating said virtual microphone comprises synchronising said virtual microphone with an image content of said image data, said modified audio data representing said virtual microphone being used to modify said image content for display in conjunction with said generated virtual microphone.

59. A computer system as claimed in claim 43, wherein said audio data is associated with an image data and said generating audio data comprises synchronising said virtual microphone with identified characteristics of an image content of said image data.

60. A non-transitory computer readable medium upon which a computer program is stored, said computer program comprising:

acquiring a set of audio data representative of a recorded sound scene, said audio data characterized into a set of sound sources within a time-space reference frame;

using an audio data processing means to perform an analysis of said audio data to identify characteristic sounds of the sound sources;

selecting a subset of sound sources from the set of sound sources based on the identified characteristic sounds of the sound sources;

determining at least one virtual microphone trajectory using the selected subset of sound sources; and

generating, in said audio data processing means, a set of modified audio data for output to an audio player, said modified audio data representing sound captured from at least one virtual microphone configurable to move about said recorded sound scene, wherein said virtual microphone is generated in accordance with, and thereby controlled by, said the at least one virtual microphone trajectory and the selected subset of the sound sources.

61. Audio data processing apparatus for processing data representative of a recorded sound scene, said audio data comprising a set of sound sources each referenced within a spatial reference frame, said apparatus comprising:

means for identifying characteristic sounds associated with each of said sound sources;

means for selecting individual sound sources from the identified characteristic sounds;

means for determining at least one virtual microphone trajectory using the selected individual sound sources;

means for navigating said sound scene to sample said selected individual sound sources based on the at least one virtual microphone trajectory; and

means for generating a modified audio data comprising said sampled sounds.

62. The apparatus as claimed in claim 61, wherein said navigating means is operable for following a multi-dimensional trajectory within said sound scene.

63. The apparatus as claimed in claim 61, wherein:

said selecting means comprises means for determining which individual said sound sources exhibit features which are of interest to a human listener in the context of said sound scene; and

said navigating means is operable for visiting individual said sound sources which exhibit said features which are of interest to a human listener.

64. Audio data processing apparatus comprising:

a memory, said memory storing code for a sound source characterization component for characterizing an audio data into a set of sound sources occupying positions within a time and space reference frame;

a sound analyzer for performing an analysis of said audio data to identify characteristic sounds of the sound sources;

a sound selecting component for selecting a subset of sound sources from the set of sound sources of the audio data based on the identified characteristic sounds of the sound sources;

a trajectory determining component for determining at least one virtual microphone trajectory using the selected subset of sound sources;

at least one virtual microphone component, configurable to move about said recorded sound scene; and

a modified audio generator component for generating a set of modified audio data representing sound captured from said virtual microphone component;

a processor, wherein the processor is configured to control movement of said virtual microphone component in said sound scene associated with said the at least one virtual microphone trajectory and the selected subset of sound sources.

65. The audio data processing apparatus of claim 64, further comprising a data acquisition component for acquiring said audio data representative of a recorded sound scene.

66. A method of processing an audio visual data representing a recorded audio-visual scene, said method comprising:

characterizing, using a processor, said audio data into a set of sound sources, occupying positions within a time and space reference frame;

analyzing said audio-visual data to obtain visual cues;

selecting a subset of sound sources from the set of sound sources based on the visual cues;

determining at least one virtual microphone trajectory using the selected subset of sound sources; and

generating a modified audio data representing sound captured from at least one virtual microphone configured for moving around said recorded audio-visual scene, wherein said virtual microphone is controlled in accordance with the at least one virtual microphone trajectory and the selected subset of sound sources to conduct a virtual tour of said recorded audio-visual scene.

67. An audio-visual data processing apparatus for processing an audio-visual data representing a recorded audio-visual data representing a recorded audio visual scene, said apparatus comprising:

a memory, said memory storing code for a sound source characterizer for characterizing audio data into a set of sound sources occupying positions within a time and space reference frame;

an analysis component for analyzing said audio-visual to obtain visual cues;

a sound selecting component for selecting a subset of sound sources from the set of sound sources of the audio data based on the visual cues;

a trajectory determining component for determining at least one virtual microphone trajectory using the selected subset of sound sources;

at least one virtual microphone component, configurable to navigate said audio-visual scene; and

an audio generator component for generating a set of modified audio data representing sound captured from said virtual microphone component;

a processor, wherein the processor is configured to control navigation of said virtual microphone component in said audio-visual scene in accordance with said visual the at least one virtual microphone trajectory and the selected subset of sound sources.

68. The data processing apparatus as claimed in claim 67, further comprising a data acquisition component for acquiring audio-visual data representative of a recorded audio-visual scene.