Systems, methods, and devices for synchronization of vehicle data with recorded audio

• US 10,397,516 B2
• Filed: 04/29/2016
• Issued: 08/27/2019
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Mullen, Jonathan Thomas
• Primary Examiner(s): Topgyal, Gelek W
• Application Number:
  US15/142,743
• Publication Number:
  US 20170318257A1
• Time in Patent Office: Days
• CPC Class Codes: G07C 5/08 Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G11B 27/007 reproducing continuously a part of the information, i.e. repeating
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/9305 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
• CPC Class Codes: H04N 9/802 involving processing of the sound signal H04N9/806, H04N9/835 take precedence

Abstract

A method for post-processing to synchronize audio data with vehicle data includes generating an artificial sound data based on time-series vehicle data. The method includes determining an offset that maximizes cross-correlation between the artificial sound data and recorded audio data. The method also includes shifting one or more of the time-series data and the recorded audio data relative to each other in time based on the offset. The shift may be used to generate or render a synchronized set of time-series data and recorded audio data.

Claims

1. A method comprising:

determining an engine speed based on time-series vehicle data;

determining recorded audio data comprising a vehicle noise having non-zero volume;

generating proxy sound data comprising a proxy vehicle noise synthesized based on the engine speed;

determining an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data; and

shifting one or more of the time-series vehicle data or the recorded audio data relative to each other in time based on the offset to generate a synchronized set of time-series vehicle data and recorded audio data.

2. The method of claim 1, wherein generating the proxy sound data comprises modifying an amplitude of a sound based on a value of a portion of the time-series vehicle data.

3. The method of claim 2, wherein modifying the amplitude of the sound comprises modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

4. The method of claim 1, wherein the vehicle noise having non-zero volume comprises one or more of engine noise, tire noise, or wind noise, and wherein generating the proxy sound data comprises generating a proxy sound to simulate the vehicle noise and comprises one or more of:

a proxy engine noise;

a proxy wind noise;

or

a proxy tire noise.

5. The method of claim 1, wherein the time-series vehicle data further comprises data for a vehicle speed and a vehicle accelerator pedal signal, and wherein generating the proxy sound data comprises generating based on the engine speed and one or more of the vehicle speed or the vehicle accelerator pedal signal.

6. The method of claim 1, wherein shifting one or more of the time-series vehicle data and the recorded audio data in time based on the offset comprises shifting during rendering of one or more of:

the recorded audio data;

or

image or video data corresponding to the audio data.

7. The method of claim 1, wherein shifting one or more of the time-series vehicle data and the recorded audio data in time based on the offset comprises one or more of:

modifying a time signature of one or more of the time-series vehicle data and the recorded audio data;

storing the offset with one or more of the time-series vehicle data or the recorded audio data; and

generating a file that correlates portions of the time-series vehicle data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.

8. The method of claim 1, wherein the time-series vehicle data comprises data produced by one or more vehicle sensors or control systems of a vehicle during a driving period and the recorded audio data comprises audio data recorded during the driving period.

9. A system comprising:

a microphone of a vehicle;

one or more vehicle sensors for capturing time-series vehicle data; and

a processor that is programmable to execute instructions stored in non-transitory computer readable storage media, the instructions comprising;

determining an engine speed based on the time-series vehicle data;

receiving recorded audio data from the microphone comprising a vehicle noise having non-zero volume;

generating proxy sound data comprising a proxy vehicle noise synthesized based on the engine speed;

determining an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data; and

shifting one or more of the time-series vehicle data or the recorded audio data relative to each other in time based on the offset to generate a synchronized set of time-series vehicle data and recorded audio data.

10. The system of claim 9, wherein generating the proxy sound data comprises modifying an amplitude of a sound based on a value of a portion of the time-series vehicle data.

11. The system of claim 10, wherein modifying the amplitude of the sound comprises modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

12. The system of claim 9, wherein the vehicle noise having non-zero volume comprises one or more of engine noise, tire noise, or wind noise, and wherein generating the proxy sound data comprises generating a proxy sound to simulate the vehicle noise and comprises one or more of:

a proxy engine noise;

a proxy wind noise;

or

a proxy tire noise.

13. The system of claim 9, wherein the time-series vehicle data further comprises data for vehicle speed and a vehicle accelerator pedal signal, and wherein the proxy sound component is configured to generate the proxy sound data by generating based on the engine speed and one or more of the vehicle speed or the vehicle accelerator pedal signal.

14. The system of claim 9, wherein shifting one or more of the time-series vehicle data and the recorded audio data in time based on the offset comprises shifting during rendering of one or more of:

the recorded audio data;

or

image or video data corresponding to the audio data.

15. The system of claim 9, wherein shifting one or more of the time-series vehicle data and the recorded audio data in time comprises one or more of:

modifying a time signature of one or more of the time-series vehicle data and the recorded audio data;

storing the offset with one or more of the time-series vehicle data or the recorded audio data; and

generating a file that correlates portions of the time-series vehicle data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.

16. Non-transitory computer readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to:

determine an engine speed based on time-series vehicle data;

determine recorded audio data comprises a vehicle noise having non-zero volume;

generate proxy sound data comprises a proxy vehicle noise synthesized based on the engine speed;

determine an offset that maximizes cross-correlation between the proxy sound data and the recorded audio data; and

shift one or more of the time-series vehicle data or the recorded audio data relative to each other in time based on the offset to generate a synchronized set of time-series vehicle data and recorded audio data.

17. The non-transitory computer readable media of claim 16, wherein the instructions cause the one or more processors to generate the proxy sound data by modifying an amplitude of a sound over time based on a value of a portion of the time-series vehicle data over time.

18. The non-transitory computer readable media of claim 17, wherein the instructions cause the one or more processors to modify the amplitude of the sound by modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

19. The non-transitory computer readable media of claim 16, wherein the vehicle noise having non-zero volume comprises one or more of engine noise, tire noise, or wind noise, and wherein generating the proxy sound data comprises a proxy sound to simulate the vehicle noise and comprises one or more of a proxy engine noise, a proxy wind noise, or a proxy tire noise.

20. The non-transitory computer readable media of claim 16, wherein the instructions cause the one or more processors to determine the offset that maximizes cross-correlation by matching one or more features in the proxy sound data with one or more features in the recorded audio data and determining the offset to match the one or more features in the proxy sound data with the one or more features in the recorded audio data in time.