SOUND-OBJECT ORIENTED ANALYSIS AND NOTE-OBJECT ORIENTED PROCESSING OF POLYPHONIC SOUND RECORDINGS

• US 20090241758A1
• Filed: 03/05/2009
• Published: 10/01/2009
• Est. Priority Date: 03/07/2008
• Status: Active Grant

• Current Assignee: Celemony Software GmbH
• Original Assignee: Peter Neubacker
• Inventors: Neubacker, Peter
• Granted Patent: US 8,022,286 B2
• Time in Patent Office: Days
• US Class Current: 84/609
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/06 Circuits for establishing the harmonic content of tones , or other arrangements for changing the tone colour
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2220/116 for graphical editing of sound parameters or waveforms, e.g. by graphical interactive control of timbre, partials or envelope

Abstract

A method of sound-object oriented analysis and of note-object oriented processing a polyphonic digitized sound recording present in the form of a time signal F(A, t) includes the following analytical and processing steps: portion-wise readouts of the time signal F(A, t) using a window function and overlapping windows; Fourier-transforming the readout signal into frequency space, in particular by applying a discrete Fourier transform; calculating an energy value E at each bin from the frequency amplitude resulting from the Fourier transformation, in particular by squaring the real and imaginary parts or forming energy values derived from them; generating a function F(t, f, E); identifying event objects; identifying event objects; identifying note objects; comparing the temporal occurrence of event objects and note objects and associating event objects to note objects in the case of plausible time occurrences; calculating spectral proportion factors for each note object.

Claims

1. A method for sound-object oriented analysis and for note-object oriented processing of a polyphonic, digitized sound recording which is present in the form of a time signal F(A, t), comprising the following analytical and processing steps:

Portion-wise readout of the time signal F(A, t) using a window function and overlapping windows,

Fourier transformation of the readout signal into the frequency space, in particular by carrying out a discrete Fourier transformation,

Calculating from the frequency amplitude, an energy value E for each bin, said amplitude resulting from the Fourier transformation, in particular by squaring the real and imaginary parts or an energy value derived therefrom,

Generating a three-dimensional function F(t, f, E) from the frequency amplitude),

Identifying event objects,

Identifying note objects,

Comparing the occurrence in time of event objects and note objects and associating event objects with note objects in the case of plausible occurrences in time,

Calculating spectral proportion factors for each note object,

Associating signal portions of the frequency signal F(f, t, E) with detected note objects using the calculated proportion factors,

Inverse transformation of the frequency signal portions associated with a note object into a time signal,

Graphically represent the note objects and/or event objects as a time/frequency display on a monitor,

User controlled or automated processing of one or more note objects, Storing the time signals of processed note objects,

Reproducing the stored time signals of processed note objects jointly with the time signal which is reduced by a time signal associated with a note object.

2. Method as claimed in claim 1, wherein a function F′

(t, f, E′

) is calculated from F(t, f, E), the energy values E′

being the sum of all energy values E at a time t at a fundamental frequency f and all its multiples/harmonics.

3. Method as claimed in claim 2, wherein the energy values of the multiples/harmonics of the fundamental frequency are added following weighting by a factor differing from 1.

4. Method as claimed claim 2, wherein the following method stages are carried out to identify the note objects:

Ascertaining an energy maximum in the function F′

(f, t, E′

),

Ascertaining a field of values related to the maximum,

Associating the detected field of values each time with a note object.

5. Method as claimed in claim 4, wherein the energy values E of the field of values associated with the note object is subtracted from the function F(t, f, E).

6. Method as claimed in claim 5, wherein the energy values E are subtracted only at the level G*E, where G is a factor such that 0<

G<

1.

7. Method as claimed in claim 6, wherein the factor G is a parameter that may be varied by user.

8. Method as claimed in claim 5, wherein the search for a maximum is continued on a function from which the energy values were deducted or on a function calculated therefrom.

9. Method as claimed in claim 4, wherein the search for a maximum is carried out iteratively until a termination criterion has been reached.

10. Method as claimed in claim 9, wherein a total energy value Etot relating to the function F(t, f, E) is calculated and the iteration is terminated as soon as a given proportion H*Etot of this total value has been associated with the detected note objects, when said proportion rises above 90%.

11. Method as claimed in claim 10, wherein the H factor is a user-variable parameter.

12. Method as claimed in claim 1, wherein an identified object during a subsequent automated purification shall be discarded in the presence of one or more of the following criteria:

The detected note object'"'"'s energy is minute compared with the total energy

The evolution of pitch and amplitude of the note object substantially coincides with that of another note object of lower frequency, in particular when the lower frequency may be the fundamental,

There is a very large gap in frequency between one note object and the other note objects.

13. Method as claimed in claim 1, wherein, in a post-processing stage, a user separates, links and/or erases automatically identified note objects.

14. Method as claimed in claim 1, wherein for each bin, an instantaneous frequency is calculated from the phase difference of neighboring bins said instantaneous frequency being used as the bin frequency in the function F(t, f, E) or F′

(t, f, E′

).

15. Method as claimed in claim 1, wherein a tonality value and/or noisiness value is calculated in each bin to detect the event objects.

16. Method as claimed in claim 15, wherein each bin energy value is weighted by the tonality value.

17. Method as claimed in claim 1, wherein the identification of note objects resorts to available/stored notes.

18. Method as claimed in claim 1, wherein the detected event objects are extracted from the time signal F(A, t) and the sound resolution is carried out on the residual signal.

19. Method as claimed in claim 18, wherein the event objects are stored separately.

20. Method as claimed in claim 1, wherein the spectral proportion factors of a note object are calculated from a stored/available proportion function.

21. Method as claimed in claim 20, wherein the stored/available spectral proportion function is the mathematical mapping of a note object'"'"'s sound model, this model comprising one or more of the following postulates:

A fundamental tone with a fundamental frequency that is accompanied by spectral components at whole-number multiples of the said fundamental, namely harmonics at harmonic frequencies,

The amplitude evolution of the harmonics of a fundamental obeys a law based on a harmonics model or an empirically determined harmonics amplitude evolution,

The harmonics'"'"' amplitudes are in a fixed relationship to the evolution of the fundamental'"'"'s energy.

22. Method as claimed in claim 20, wherein the stored/available spectral proportion function is the mathematical mapping of the sound of a note played on a given instrument.

23. Method as claimed in claim 20, wherein several spectral proportion functions are stored/available in particular different proportion functions for several instruments.

24. Method as claimed in claim 23, wherein the user selects one of the several proportion functions.

25. Method as claimed in claim 23, wherein one of the several proportion functions is automatically associated to an note object upon input by a user or by automated action, upon detecting on which instrument the note was played.

26. Method as claimed in claim 20, wherein the spectral proportion function by means of a window function carries out weighting at a predetermined frequency width.

27. Method as claimed in claim 1, wherein a residual signal is calculated by subtracting all time signals associated with the note objects and with the event objects from the original time signal.

28. Method as claimed in claim 27, wherein the residual signal is subjected to a further identification of note or event objects.

29. Method as claimed in claim 1, wherein with respect to sound reproductions and following processing a note object, the sound portion of the note object is removed from the total sound and the differential signal produced in this manner shall be played jointly with the note object'"'"'s sound portion.

30. Method as claimed in claim 27, wherein with respect to sound reproduction and following processing a note object by the user, the residual signal is jointly reproduced.

31. A computer program comprising a programming code to carry out the method claimed in claim 1 when the computer program is run on a computer.