Sound alignment user interface

• US 9,201,580 B2
• Filed: 11/13/2012
• Issued: 12/01/2015
• Est. Priority Date: 11/13/2012
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: King, Brian John, Mysore, Gautham J., Smaragdis, Paris
• Primary Examiner(s): Kuntz, Curtis
• Assistant Examiner(s): MAUNG, THOMAS H
• Application Number:
  US13/675,807
• Publication Number:
  US 20140136976A1
• Time in Patent Office: 1,113 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

Sound alignment user interface techniques are described. In one or more implementations, a user interface is output having a first representation of sound data generated from a first sound signal and a second representation of sound data generated from a second sound signal. One or more inputs are received, via interaction with the user interface, that indicate that a first point in time in the first representation corresponds to a second point in time in the second representation. Aligned sound data is generated from the sound data from the first and second sound signals based at least in part on correspondence of the first point in time in the sound data generated from the first sound signal to the second point in time in the sound data generated from the second sound signal.

Claims

1. A method implemented by one or more computing devices, the method comprising:

outputting a user interface having a first representation of sound data generated from a first sound signal and a second representation of sound data generated from a second sound signal;

identifying, automatically and without user input, a first portion of the first representation that contains one or more features that correspond to one or more features in a second portion of the second representation;

receiving one or more manual inputs via interaction with the user interface, the manual inputs being selection of a first point in time within the first representation and selection of a corresponding second point in time within the second representation; and

generating aligned sound data from the sound data from the first and second sound signals by at least;

aligning the first point in time in the sound data generated from the first sound signal to the second point in time in the sound data generated from the second sound signal, the aligning effective to anchor the first point and the second point to each other in the aligned sound data; and

adjusting, automatically and without user input, the second portion of the second representation containing the one or more features to have a slower speed or a faster speed, the adjustment effective to align in time the one or more features in the second portion with the one or more features in the first portion without changing the alignment of the first point with the second point.

2. The method as described in claim 1, wherein the one or more manual inputs define corresponding intervals in the sound data generated from the first and second sound signals, respectively.

3. The method as described in claim 2, wherein a plurality of said intervals are indicated, via manual interaction with the user interface, as corresponding between the sound data of the first and second sound signals, respectively, and the generating of the aligned sound data is performed individually for corresponding said intervals to generate the aligned sound data.

4. The method as described in claim 1, wherein the one or more features are spectral basis vectors, speech weights, or noise weights.

5. The method as described in claim 1, wherein the one or more features are computed using one or more of:

probabilistic latent component analysis, non-negative matrix factorization, non-negative hidden Markov model, or non-negative factorial hidden Markov model.

6. The method as described in claim 1, wherein the first and second representations describe time and frequency of the sound data generated from the first and second sound signals, respectively.

7. The method as described in claim 6, wherein the sound data from the first and second sound signals are computed using short time Fourier transforms.

8. The method as described in claim 1, wherein the adjusting is accomplished by using one or more of:

global offset, sampling factor, compression, or stretching.

9. The method as described in claim 1, wherein the first sound signal is an overdub sound signal and the second sound signal is a reference sound signal.

10. A system comprising:

at least one module implemented at least partially in hardware and configured to output a user interface that is usable to manually select points within a plurality of representations of sound data generated from a plurality of sound signals, each selected point in a representation corresponding to a selected point in each other representation and the selected points within each representation defining a plurality of corresponding intervals within the representations of sound data; and

one or more modules implemented at least partially in hardware and configured to generate aligned sound data from the sound data generated from the plurality of sound signals using the defined plurality of corresponding intervals by;

aligning each selected point within a representation to the corresponding selected point within each other representation, the aligning effective to anchor the corresponding points to each other in the aligned sound data;

automatically and without user intervention, dividing an alignment task for aligning the sound data generated from the plurality of sound signals into a plurality of interval alignment tasks, the interval alignment tasks being alignment of each defined interval within each representation with the corresponding defined interval within each other representation to produce aligned intervals without changing the alignment of the corresponding selected points;

combining the aligned intervals from the plurality of representations; and

using the combination of the aligned intervals as the aligned sound data.

11. The system as described in claim 10, wherein a first said interval defined for a first said representation has a length in time that is different than a second said interval defined for a second said representation.

12. The system as described in claim 11, wherein the one or more modules are configured to generate the aligned sound data by adjusting, automatically and without user intervention, the second interval to have a slower speed or a faster speed such that it is aligned in time with the corresponding first interval.

13. The system as described in claim 10, wherein at least two of the plurality of interval alignment tasks are configured to be executed in parallel by the one or more modules.

14. The system as described in claim 12, wherein the adjusting is accomplished by using one or more of:

global offset, sampling factor, compression, or stretching.

15. The system as described in claim 10, wherein the plurality of representations of sound data are computed using short time Fourier transforms.

16. The system as described in claim 10, wherein the representations describe time and frequency of the sound data generated from respective ones of the plurality of sound signals.

17. One or more non-transitory computer-readable storage media having instructions stored thereon that, responsive to execution on a computing device, causes the computing device to perform operations comprising:

outputting a user interface having a first representation of sound data generated from a first sound signal and a second representation of sound data generated from a second sound signal;

identifying, automatically and without user input, a first portion of the first representation that contains one or more features that correspond to one or more features in a second portion of the second representation;

receiving one or more manual inputs via interaction with the user interface, the manual inputs being selection of a first point in time within the first representation and selection of a corresponding second point in time within the second representation; and

generating aligned sound data from the sound data generated from the first and second sound signals at least in part by;

aligning the first manually selected point in time in the sound data generated from the first sound signal to the second manually selected point in time in the sound data generated from the second sound signal, the aligning effective to anchor the first point and the second point to each other in the aligned sound data; and

adjusting, automatically and without user input, the second portion to have a slower speed or a faster speed, the adjustment effective to align in time the one or more features in the second portion with the one or more features in the first portion without changing the alignment of the first and second manually selected points.

18. The one or more non-transitory computer-readable storage media as described in claim 17, wherein the generating of the aligned sound data further comprises:

dividing the first and second representations of sound data into a plurality of intervals, each interval in the first representation corresponding to an interval in the second representation; and

dividing an alignment task for the sound data generated from the first and second sound signals into a plurality of interval alignment tasks that involve the corresponding intervals.

19. The one or more non-transitory computer-readable storage media as described in claim 18, wherein at least two of the plurality of interval alignment tasks are configured to be executed in parallel.

20. The one or more non-transitory computer-readable storage media as described in claim 18, wherein the generating is performed such that the aligned sound data is generated from the sound data of the first and second sound signals from a combination of results of the plurality of interval alignment tasks.