Data search, parser, and synchronization of video and telemetry data

• US 8,189,690 B2
• Filed: 10/15/2010
• Issued: 05/29/2012
• Est. Priority Date: 10/19/2009
• Status: Active Grant

• Current Assignee: Intergraph Technologies Company
• Original Assignee: Intergraph Technologies Company
• Inventors: Whitaker, Sheila G., Grindstaff, Gene A., Shelton, Roger K., Howell, William D.
• Primary Examiner(s): Patel, Jay
• Assistant Examiner(s): Pe, Geepy
• Application Number:
  US12/905,751
• Publication Number:
  US 20110090399A1
• Time in Patent Office: 592 Days
• Field of Search: 375/240.26, 375/240.25, 348/143
• US Class Current: 375/240.28
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: H04N 21/2146 located in mass transportation means, e.g. aircraft, train or bus flight-deck installations for entertainment or communications B64D11/0015; arrangements specially adapted for transportation systems in broadcast systems H04H20/62; moving wireless networks H04W84/005
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program

Abstract

A system to parse both telemetry data and corresponding encoded video data wherein the telemetry and video data are subsequently synchronized based upon temporal information, such as a time stamp. The telemetry data and the video data are originally unsynchronized and the data for each is acquired by a separate device. The acquiring devices may be located within or attached to an aerial vehicle. The system receives the telemetry data and the encoded video data and outputs a series of synchronized video images with telemetry data. Thus, telemetry information is associated with each video image. The telemetry data may be acquired at a different rate than the video data. As a result, telemetry data may be interpolated or extrapolated to correspond to each video image. The present system operates in real-time, thus data acquired from aerial vehicles can be displayed on a map.

Claims

1. A system for synchronizing unsynchronized encoded aerial video data and separately acquired telemetry data for generation of a map containing the aerial video data, each sample of the telemetry data having an associated acquisition time, the system comprising:

a telemetry parser for receiving a targeted acquisition time period and a telemetry data stream, the telemetry parser parsing the telemetry data stream to locate a targeted telemetry data, wherein the targeted telemetry data is the telemetry data temporally proximate or within the targeted acquisition time period;

a video frame extraction module for accessing an encoded aerial video data stream, the video frame extraction module decoding a portion of the encoded aerial video data stream into targeted video frames, each frame having an associated video frame time, wherein the targeted video frames are temporally proximate to or within the targeted acquisition time period;

a synchronization module for synchronizing the targeted telemetry data and the targeted video frames based upon the associated acquisition time and video frame time; and

a filter for producing additional telemetry data so that there is telemetry data that corresponds to each video frame within the targeted acquisition time period.

2. The system according to claim 1 further comprising:

a graphical user interface outputting a targeted acquisition time period of video data for display.

3. The system according to claim 1, wherein the encoded aerial video data stream is parsed to locate data based upon a presentation timestamp for a group of video frames temporally proximate to or within the targeted time period.

4. The system according to claim 1, wherein the telemetry data is sampled at a rate that is longer than a sample rate for the video frame data.

5. The system according to claim 1 further comprising:

a rendering module for producing a map wherein the video frame data is spatially transformed from an aerial acquisition plane to a map plane and the transformed video frame data is inserted into the map based upon the telemetry data associated with the video frame data.

6. The system according to claim 1 further comprising:

memory for storing the telemetry data stream into a telemetry file and for storing the encoded aerial video data stream into an aerial video file.

7. The system according to claim 1, wherein the telemetry data is encoded in Key Length Value format.

8. The system according to claim 1, wherein the telemetry parser parses a first known key byte size of data and places a first and second portion of the data into at least a first and a second buffer, the telemetry parser checks telemetry data within the first buffer for a first partial key value and upon the telemetry data representing a valid partial key, the parser checks the telemetry data in the second buffer for a second partial key value.

9. The system according to claim 1 wherein the video frame extraction module will discard any MPEG-encoded data associated with a presentation time stamp that is not within a targeted time range.

10. A method for parsing a targeted portion of telemetry data from a telemetry data file stored in key length value format based upon a targeted temporal period, the method comprising:

receiving a first section of the telemetry data file;

comparing the first section of the telemetry data file to an identified key;

upon the first section of the telemetry data file matching the identified key;

identifying the length associated with the desired key;

comparing a time stamp within the telemetry data to the targeted temporal period;

upon the time stamp being temporally proximate or within the temporal period, saving data of a length equal to the identified length to a database structure based upon the identified key; and

upon the time stamp not being temporally proximate to or within the temporal period, discarding the telemetry data associated with the key and incrementing to a next byte of data from the telemetry data file.

11. The method according to claim 10, wherein the database structure for the data saved equal to the identified length includes indicia of a ground location.

12. The method according to claim 10, wherein the database structure for the data saved equal to the identified length includes an identifier for longitude and latitude of a ground location.

13. A method for synchronizing a targeted portion of a telemetry data file and a targeted portion of a separately acquired encoded video file, the method comprising:

receiving into a processor a data segment of a telemetry data file;

comparing the data segment of the telemetry data file to an identified key;

upon the data segment of the telemetry data file not matching the identified key;

incrementing to a next byte of data and repeating receiving and comparing;

upon the first section of the telemetry data file matching the identified key;

identifying a length identifier associated with the desired key;

identifying a telemetry time stamp within the telemetry data;

upon the telemetry time stamp being temporally proximate or within a user-defined temporal period, saving data from the telemetry data file of a length equal to the identified length to a database structure in memory based upon the identified key;

upon the telemetry time stamp not being temporally proximate to or within the user-defined temporal period, discarding the telemetry data associated with the key and incrementing to a next byte of data from the telemetry data file and repeating receiving, comparing, identifying and saving;

receiving a data segment of an encoded video data file;

for each presentation time stamp identified within the data segment of the encoded video data file, determining if a presentation time is proximate to or within the user-defined temporal period;

upon the presentation time stamp being proximate to or within the user-defined temporal period, decoding video data and identifying frame time stamps for each video frame and saving to memory the video frames within the user-defined temporal period;

synchronizing the telemetry data stored in memory with the video frames stored in memory based upon the user-defined temporal period; and

storing the synchronized telemetry data and corresponding video frames to memory.

14. The method according to claim 13, further comprising:

view space transforming the video frames stored in memory based upon the synchronized telemetry data; and

rendering the view space transformed video frames on display with a map.

15. A method for parsing encoded video data to locate one or more video frames that have a targeted time period, the method comprising:

receiving encoded video data into a processor;

locating an approximate time stamp associated with a presentation time stamp;

comparing the approximate time stamp to the targeted time period;

upon the approximate time stamp being proximate to or within the targeted time period, decoding a group of encoded video frames and identifying a frame time stamp for each decoded video frame; and

upon the frame time stamp being within the targeted time period, storing the associated frame and corresponding frame time stamp to memory for later retrieval.

16. A method according to claim 15, wherein the encoded video data is encoded in an MPEG format.

17. A method according to claim 15 wherein the approximate time stamp is an estimate of playback time and is associated with an identifiable range.

18. A method according to claim 15, further comprising:

receiving the targeted time period from a look-up module wherein the look-up module receives as input a map location and identifies the targeted time period for an encoded video data file based on a flight path for an aerial vehicle acquiring the encoded video data file.

19. A method according to claim 15 wherein the encoded video data is encoded in an MPEG-2 format.