METHOD AND APPARATUS FOR CORRECTING DELAY BETWEEN ACCOMPANIMENT AUDIO AND UNACCOMPANIED AUDIO, AND STORAGE MEDIUM

• US 20200135156A1
• Filed: 11/26/2018
• Published: 04/30/2020
• Est. Priority Date: 06/11/2018
• Status: Active Grant

• Current Assignee: Guangzhou Kugou Computer Technology Co. Ltd. (Tencent Holdings Limited)
• Original Assignee: Guangzhou Kugou Computer Technology Co. Ltd. (Tencent Holdings Limited)
• Inventors: Zhang, Chaogang
• Granted Patent: US 10,964,301 B2
• Time in Patent Office: Days
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/361 Recording/reproducing of accompaniment for use with an external source, e.g. karaoke systems
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 2210/005 Musical accompaniment, i.e. complete instrumental rhythm synthesis added to a performed melody, e.g. as output by drum machines
• CPC Class Codes: G10H 2210/056 for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10H 2240/325 Synchronizing two or more audio tracks or files according to musical features or musical timings
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation

Abstract

A method and apparatus for correcting a delay between accompaniment audio and unaccompanied audio, and a storage medium are provided. The method includes: acquiring original audio of a target song, and extracting original vocal audio from the original audio; determining a first delay between the original vocal audio and the unaccompanied audio, and determining a second delay between the accompaniment audio and the original audio; and correcting a delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay. Thus, the correction efficiency of the delay between accompaniment audio and unaccompanied audio is improved, and correction mistakes possibly caused by human factors are eliminated, thereby improving the accuracy.

Claims

1. A method for correcting a delay between accompaniment audio and unaccompanied audio, comprising:

acquiring original audio of a target song, and extracting original vocal audio from the original audio;

determining a first delay between the original vocal audio and the unaccompanied audio, and determining a second delay between the accompaniment audio and the original audio; and

correcting a delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay.

2. The method according to claim 1, wherein determining a first delay between the original vocal audio and the unaccompanied audio comprises:

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the original vocal audio, and ranking a plurality of acquired pitch values of the original vocal audio according to a sequence of the plurality of audio frames contained in the original vocal audio to obtain a first pitch sequence;

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the unaccompanied audio, and ranking a plurality of acquired pitch values of the unaccompanied audio according to a sequence of the plurality of audio frames contained in the unaccompanied audio to obtain a second pitch sequence;

determining a first correlation function curve based on the first pitch sequence and the second pitch sequence; and

determining the first delay between the original vocal audio and the unaccompanied audio based on a first peak detected on the first correlation function curve.

3. The method according to claim 2, wherein determining a first correlation function curve based on the first pitch sequence and the second pitch sequence comprises:

determining, based on the first pitch sequence and the second pitch sequence, a first correlation function model as illustrated by the following formula;

4. The method according to claim 1, wherein determining a second delay between the accompaniment audio and the original audio comprises:

acquiring a plurality of audio frames contained in the original audio according to a sequence of the plurality of audio frames contained in the original audio to obtain a first audio sequence;

acquiring a plurality of audio frames contained in the accompaniment audio according to a sequence of the plurality of audio frames contained in the accompaniment audio to obtain a second audio sequence;

determining the second correlation function curve based on the first audio sequence and the second audio sequence; and

determining the second delay between the accompaniment audio and the original audio based on a second peak detected on the second correlation function curve.

6. The method according to claim 1, wherein the correcting the delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay comprises:

determining a delay difference between the first delay and the second delay as a delay between the accompaniment audio and the unaccompanied audio;

deleting audio data in a first period in the accompaniment audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is later than the unaccompanied audio, wherein a start moment of the first period is a start moment of the accompaniment audio, and a duration of the first period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio; and

deleting audio data in a second period in the unaccompanied audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is earlier than the unaccompanied audio, wherein a start moment of the second period is a start moment of the unaccompanied audio, and a duration of the second period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio.

11. The apparatus according to claim 6, wherein the correcting module comprises:

a detecting sub-module, configured to detect a first peak on the first correlation function curve, and detect a second peak on the second correlation function curve;

a third determining sub-module, configured to determine a first delay between the original vocal audio and the unaccompanied audio based on the first peak, and determine a second delay between the accompaniment audio and the original audio based on the second peak; and

a correcting sub-module, configured to correct the delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay.

12. The apparatus according to claim 11, wherein the correcting sub-module is configured to:

determine a delay difference between the first delay and the second delay as a delay between the accompaniment audio and the unaccompanied audio;

delete audio data in a second period in the unaccompanied audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is later than the unaccompanied audio, wherein a start moment of the second period is a start moment of the unaccompanied audio, and a duration of the second period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio; and

delete audio data in a second period in the unaccompanied audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is earlier than the unaccompanied audio, wherein a start moment of the second period is a start moment of the unaccompanied audio, and a duration of the second period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio.

14. A non-transitory computer-readable storage medium storing an instruction, wherein the instruction, when being executed by a processor, causes the processor to implement the method according to claim 1.

19. The storage medium according to claim 14, wherein determining a first delay between the original vocal audio and the unaccompanied audio comprises:

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the original vocal audio, and ranking a plurality of acquired pitch values of the original vocal audio according to a sequence of the plurality of audio frames contained in the original vocal audio to obtain a first pitch sequence;

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the unaccompanied audio, and ranking a plurality of acquired pitch values of the unaccompanied audio according to a sequence of the plurality of audio frames contained in the unaccompanied audio to obtain a second pitch sequence;

determining a first correlation function curve based on the first pitch sequence and the second pitch sequence; and

determining the first delay between the original vocal audio and the unaccompanied audio based on a first peak detected on the first correlation function curve.

20. The storage medium according to claim 19, wherein determining a first correlation function curve based on the first pitch sequence and the second pitch sequence comprises:

determining, based on the first pitch sequence and the second pitch sequence, a first correlation function model as illustrated by the following formula;

21. The storage medium according to claim 14, wherein determining a second delay between the accompaniment audio and the original audio comprises:

acquiring a plurality of audio frames contained in the original audio according to a sequence of the plurality of audio frames contained in the original audio to obtain a first audio sequence;

acquiring a plurality of audio frames contained in the accompaniment audio according to a sequence of the plurality of audio frames contained in the accompaniment audio to obtain a second audio sequence;

determining the second correlation function curve based on the first audio sequence and the second audio sequence; and

determining the second delay between the accompaniment audio and the original audio based on a second peak detected on the second correlation function curve.

22. The storage medium according to claim 14, wherein the correcting the delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay comprises:

determining a delay difference between the first delay and the second delay as a delay between the accompaniment audio and the unaccompanied audio;

deleting audio data in a first period in the accompaniment audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is later than the unaccompanied audio, wherein a start moment of the first period is a start moment of the accompaniment audio, and a duration of the first period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio; and

deleting audio data in a second period in the unaccompanied audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is earlier than the unaccompanied audio, wherein a start moment of the second period is a start moment of the unaccompanied audio, and a duration of the second period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio.

5. (canceled)

7. An apparatus for correcting a delay between accompaniment audio and unaccompanied audio, comprising:

an acquiring module, configured to acquire accompaniment audio, unaccompanied audio and original audio of a target song, and extract original vocal audio from the original audio;

a determining module, configured to determine a first correlation function curve based on the original vocal audio and the unaccompanied audio, and determine a second correlation function curve based on the original audio and the accompaniment audio; and

a correcting module, configured to correct a delay between the accompaniment audio and the unaccompanied audio based on the first correlation function curve and the second correlation function curve.

8. The apparatus according to claim 7, wherein the determining module comprises:

a first acquiring sub-module, configured to acquire a pitch value corresponding to each of a plurality of audio frames contained in the original vocal audio, and rank the plurality of acquired pitch values of the original vocal audio according to a sequence of the plurality of audio frames contained in the original vocal audio to obtain a first pitch sequence, wherein

the first acquiring sub-module is further configured to acquire a pitch value corresponding to each of a plurality of audio frames contained in the unaccompanied audio, and rank a plurality of acquired pitch values of the unaccompanied audio according to a sequence of the plurality of audio frames contained in the unaccompanied audio to obtain a second pitch sequence; and

a first determining sub-module, configured to determine the first correlation function curve based on the first pitch sequence and the second pitch sequence.

9. The apparatus according to claim 8, wherein the first determining sub-module is configured to:

determine, based on the first pitch sequence and the second pitch sequence, a first correlation function model as illustrated by the following formula;

10. (canceled)

13. An apparatus for correcting a delay between accompaniment audio and an unaccompanied audio, comprising:

a processor; and

a memory configured to store a processor-executable instruction, wherein

the processor is configured to implement a method when the instruction is executed by the processor-, and the method comprises;

acquiring original audio of a target song, and extracting original vocal audio from the original audio;

determining a first delay between the original vocal audio and the unaccompanied audio, and determining a second delay between the accompaniment audio and the original audio; and

correcting a delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay.

15. The apparatus according to claim 13, wherein determining a first delay between the original vocal audio and the unaccompanied audio comprises:

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the original vocal audio, and ranking a plurality of acquired pitch values of the original vocal audio according to a sequence of the plurality of audio frames contained in the original vocal audio to obtain a first pitch sequence;

acquiring a pitch value corresponding to each of a plurality of audio frames contained in the unaccompanied audio, and ranking a plurality of acquired pitch values of the unaccompanied audio according to a sequence of the plurality of audio frames contained in the unaccompanied audio to obtain a second pitch sequence;

determining a first correlation function curve based on the first pitch sequence and the second pitch sequence; and

determining the first delay between the original vocal audio and the unaccompanied audio based on a first peak detected on the first correlation function curve.

16. The apparatus according to claim 15, wherein determining a first correlation function curve based on the first pitch sequence and the second pitch sequence comprises:

determining, based on the first pitch sequence and the second pitch sequence, a first correlation function model as illustrated by the following formula;

17. The apparatus according to claim 13, wherein determining a second delay between the accompaniment audio and the original audio comprises:

acquiring a plurality of audio frames contained in the original audio according to a sequence of the plurality of audio frames contained in the original audio to obtain a first audio sequence;

acquiring a plurality of audio frames contained in the accompaniment audio according to a sequence of the plurality of audio frames contained in the accompaniment audio to obtain a second audio sequence;

determining the second correlation function curve based on the first audio sequence and the second audio sequence; and

determining the second delay between the accompaniment audio and the original audio based on a second peak detected on the second correlation function curve.

18. The apparatus according to claim 13, wherein the correcting the delay between the accompaniment audio and the unaccompanied audio based on the first delay and the second delay comprises:

determining a delay difference between the first delay and the second delay as a delay between the accompaniment audio and the unaccompanied audio;

deleting audio data in a first period in the accompaniment audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is later than the unaccompanied audio, wherein a start moment of the first period is a start moment of the accompaniment audio, and a duration of the first period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio; and

deleting audio data in a second period in the unaccompanied audio if the delay between the accompaniment audio and the unaccompanied audio indicates that the accompaniment audio is earlier than the unaccompanied audio, wherein a start moment of the second period is a start moment of the unaccompanied audio, and a duration of the second period is equal to a duration of the delay between the accompaniment audio and the unaccompanied audio.