Systems, Methods, and Devices for Synchronization of Vehicle Data with Recorded Audio

• US 20170318257A1
• Filed: 04/29/2016
• Published: 11/02/2017
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Original Assignee: Ford Global Technologies, LLC (Ford Motor Company)
• Inventors: Mullen, Jonathan Thomas
• Granted Patent: US 10,397,516 B2
• Time in Patent Office: Days
• CPC Class Codes: G07C 5/08 Registering or indicating performance data other than driving, working, idle, or waiting time, with or without registering driving, working, idle or waiting time
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G11B 27/007 reproducing continuously a part of the information, i.e. repeating
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/9305 involving the mixing of the reproduced video signal with a non-recorded signal, e.g. a text signal
• CPC Class Codes: H04N 9/802 involving processing of the sound signal H04N9/806, H04N9/835 take precedence

Abstract

A method for post-processing to synchronize audio data with vehicle data includes generating an artificial sound data based on time-series vehicle data. The method includes determining an offset that maximizes cross-correlation between the artificial sound data and recorded audio data. The method also includes shifting one or more of the time-series data and the recorded audio data relative to each other in time based on the offset. The shift may be used to generate or render a synchronized set of time-series data and recorded audio data.

Claims

1. A method for post-processing to synchronize audio data with vehicle data, the method comprising:

generating proxy sound data based on time-series vehicle data;

determining an offset that maximizes cross-correlation between the proxy sound data and recorded audio data; and

shifting one or more of the time-series data and the recorded audio data relative to each other in time based on the offset to generate a synchronized set of time-series data and recorded audio data.

2. The method of claim 1, wherein generating the proxy sound data comprises modifying an amplitude of a sound based on a value of a portion of the time-series data.

3. The method of claim 2, wherein modifying the amplitude of the sound comprises modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

4. The method of claim 1, wherein generating the proxy sound data comprises generating a proxy sound to simulate one or more of:

an engine sound;

a wind sound; and

a tire sound.

5. The method of claim 1, wherein the time-series data comprises data for one or more of an engine speed and a vehicle speed, wherein generating the proxy sound data comprises generating based on one or more of the engine speed and the vehicle speed.

6. The method of claim 1, wherein shifting one or more of the vehicle time series data and the recorded audio data in time based on the offset comprises shifting during rendering of one or more of:

the recorded audio data;

or

image or video data corresponding to the audio data.

7. The method of claim 1, wherein shifting one or more of the vehicle time series data and the recorded audio data in time based on the offset comprises one or more of:

modifying a time signature of one or more of the time-series data and the recorded audio data;

storing the offset with one or more of the time-series data and the recorded audio data; and

generating a file that correlates portions of the time-series data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.

8. The method of claim 1, wherein the time-series data comprises data produced by one or more vehicle sensors or control systems of a vehicle during a driving period and the recorded audio data comprises audio data recorded during the driving period.

9. A system comprising:

a proxy sound component configured to generate proxy sound data based on time-series vehicle data;

an offset component configured to determine an offset that maximizes cross-correlation between the proxy sound data and recorded audio data; and

a shift component configured to shift one or more of the vehicle time series data and the recorded audio data in time based on the offset to generate a synchronized set of time series data and audio data;

wherein the time-series data comprises data produced by one or more vehicle sensors or control systems of the vehicle during a driving period and the recorded audio data comprises audio data recorded during the driving period.

10. The system of claim 9, wherein the proxy sound component is configured to generate the proxy sound data by modifying an amplitude of a sound based on a value of a portion of the time-series data.

11. The system of claim 10, wherein the proxy sound component is configured to modify the amplitude of the sound by modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

12. The system of claim 9, wherein the proxy sound component is configured to generate the proxy sound data by generating a proxy sound to simulate one or more of:

an engine sound;

a wind sound; and

a tire sound.

13. The system of claim 9, wherein the time-series data comprises data for one or more of an engine speed and a vehicle speed, wherein the proxy sound component is configured to generate the proxy sound data by generating based on one or more of the engine speed and the vehicle speed.

14. The system of claim 9, wherein the shift component is configured to shift one or more of the vehicle time series data and the recorded audio data in time based on the offset by shifting during rendering of one or more of:

the recorded audio data;

or

image or video data corresponding to the audio data.

15. The system of claim 9, wherein the shift component is configured to shift one or more of the vehicle time series data and the recorded audio data in time by one or more of:

modifying a time signature of one or more of the time-series data and the recorded audio data;

storing the offset with one or more of the time-series data and the recorded audio data; and

generating a file that correlates portions of the time-series data with a corresponding audio or video data based on the offset, wherein the recorded audio data comprises audio data from the video recording.

16. Computer readable storage media storing instructions that, when executed by one or more processors, cause the processors to:

generate proxy sound data based on time-series vehicle data produced by one or more vehicle sensors or control systems of the vehicle during a driving period;

determine an offset that maximizes cross-correlation between the proxy sound data and recorded audio data, wherein the recorded audio data comprises audio from video data recorded during the driving period;

render at least a portion of the video data; and

during rendering a portion of the video data, determine and display a portion of the time-series data corresponding to the portion of the video data based on the offset.

17. The computer readable media of claim 16, wherein the instructions cause the processors to generate the proxy sound data by modifying an amplitude of a sound over time based on a value of a portion of the time-series data over time.

18. The computer readable media of claim 17, wherein the instructions cause the processors to modify the amplitude of the sound by modifying an amplitude of a looping generic sound, wherein the looping generic sound replicates a sound of one or more of an engine noise, internal cabin noise, external wind noise, and/or tire noise.

19. The computer readable media of claim 16, wherein the time-series data comprises data for one or more of an engine speed and a vehicle speed, wherein the instructions cause the processors to generate the proxy sound data by generating based on one or more of the engine speed and the vehicle speed.

20. The computer readable media of claim 16, wherein the instructions cause the processors to determine the offset that maximizes cross-correlation by matching one or more features in the proxy sound data with one or more features in the recorded audio data and determining the offset to match the one or more features in the proxy sound data with the one or more features in the recorded audio data in time.