Data Search, Parser, and Synchronization of Video and Telemetry Data

• US 20110090399A1
• Filed: 10/15/2010
• Published: 04/21/2011
• Est. Priority Date: 10/19/2009
• Status: Active Grant

• Current Assignee: Intergraph Technologies Company (Hexagon AB)
• Original Assignee: Intergraph Technologies Company (Hexagon AB)
• Inventors: Howell, William D., Shelton, Roger K., Whitaker, Sheila G., Grindstaff, Gene A.
• Granted Patent: US 8,189,690 B2
• Time in Patent Office: Days
• US Class Current: 348/500
• CPC Class Codes: G01C 11/025 by scanning the object
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/78 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: H04N 21/2146 located in mass transportation means, e.g. aircraft, train or bus flight-deck installations for entertainment or communications B64D11/0015; arrangements specially adapted for transportation systems in broadcast systems H04H20/62; moving wireless networks H04W84/005
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program

Abstract

A system is provided that can parse both telemetry data and corresponding encoded video data wherein the telemetry and video data are subsequently synchronized based upon temporal information, such as a time stamp. The telemetry data and the video data are originally unsynchronized and the data for each is acquired by a separate device. The acquiring devices may be located within or attached to an aerial vehicle. The system receives the telemetry data stream or file and the encoded video data stream or file and outputs a series of synchronized video images with telemetry data. Thus, there is telemetry information associated with each video image. The telemetry data may be acquired at a different rate than the video data. As a result, telemetry data may be interpolated or extrapolated to create telemetry data that corresponds to each video image. The present system operates in real-time, so that data acquired from aerial vehicles can be displayed on a map.

Claims

1. A system for synchronizing unsynchronized encoded aerial video data and separately acquired telemetry data for generation of a map containing the aerial video data, the system comprising:

a telemetry parser for receiving a user-defined acquisition time period and a telemetry data stream where the telemetry data stream includes telemetry data acquired at a sampling rate that has an associated acquisition time and parsing the telemetry data stream to locate telemetry data temporally proximate or within the user-defined acquisition time period;

a video frame extraction module for accessing an encoded aerial video data stream and decoding a portion of the encoded aerial video data stream into video frames having an associated video frame data time wherein the video frames are temporally proximate to or within the user-defined time period;

a synchronization module for synchronizing the telemetry data and the video frame data for the desired acquisition time period based upon the acquisition time and video frame data time; and

a filter for producing additional telemetry data so that there is telemetry data that corresponds to each video frame within the desired acquisition time period.

2. The system according to claim 1 further comprising:

a graphical user interface outputting a desired acquisition time period of video data for display.

3. The system according to claim 1, wherein the encoded aerial video data stream is parsed to locate data based upon a presentation timestamp for a group of video frames temporally proximate to or within the desired time period.

4. A system according to claim 1, wherein the telemetry data is sampled at a rate that is longer than a sample rate for the video frame data.

5. The system according to claim 1 further comprising:

a rendering module for producing a map wherein the video frame data is spatially transformed from an aerial acquisition plane to a map plane and the transformed video frame data is inserted into the map based upon the telemetry data associated with the video frame data.

6. The system according to claim 1 further comprising:

memory for storing the telemetry data stream into a telemetry file and for storing the encoded aerial video data stream into an aerial video file.

7. The system according to claim 1, wherein the telemetry data is encoded in Key Length Value format.

8. The system according to claim 1, wherein the telemetry parser parses a first known key byte size of data and places a first and second portion of the data into at least a first and a second buffer, the telemetry parser checks telemetry data within the first buffer for a first partial key value and if the telemetry data represents a valid partial key, the parser checks the telemetry data in the second buffer for a second partial key value.

9. The system according to claim 1 wherein the video frame extraction module will discard any MPEG-encoded data associated with a presentation time stamp that is not within a desired time range.

10. A method for parsing telemetry data from a telemetry data file stored in key length value format based upon a desired temporal period, the method comprising:

receiving a first section of the telemetry data file;

comparing the first section of the telemetry data file to an identified key;

if the first section of the telemetry data file matches the identified key;

identifying the length associated with the desired key;

comparing a time stamp within the telemetry data to the desired temporal period;

if the time stamp is temporally proximate or within the temporal period,

saving data of a length equal to the identified length to a database structure based upon the identified key;

if the time stamp is not temporally proximate to or within the temporal period discarding the telemetry data associated with the key and incrementing to a next byte of data from the telemetry data file.

11. The method according to claim 10, wherein the database structure for the data saved equal to the identified length includes indicia of a ground location.

12. The method according to claim 10, wherein the database structure for the data saved equal to the identified length includes an identifier for longitude and latitude of a ground location.

13. A method for synchronizing a telemetry data file and a separately acquired encoded video file, the method comprising:

receiving into a processor a data segment of a telemetry data file;

comparing the data segment of the telemetry data file to an identified key;

if the data segment of the telemetry data file does not match the identified key;

incrementing to a next byte of data and repeating receiving and comparing;

if the first section of the telemetry data file matches the identified key;

identifying a length identifier associated with the desired key;

identifying a telemetry time stamp within the telemetry data;

if the telemetry time stamp is temporally proximate or within a user-defined temporal period, saving data from the telemetry data file of a length equal to the identified length to a database structure in memory based upon the identified key;

if the telemetry time stamp is not temporally proximate to or within the user-defined temporal period, discarding the telemetry data associated with the key and incrementing to a next byte of data from the telemetry data file and repeating receiving, comparing, identifying and saving;

receiving a data segment of an encoded video data file;

for each presentation time stamp identified within the data segment of the encoded video data file, determining if a presentation time is proximate to or within the user-define temporal period;

if the presentation time stamp is proximate to or within the user-defined temporal period, decoding video data and identifying frame time stamps for each video frame and saving to memory the video frames within the user-defined temporal period;

synchronizing the telemetry data stored in memory with the video frames stored in memory based upon the user-defined temporal period; and

storing the synchronized telemetry data and corresponding video frames to memory.

14. The method according to claim 13, further comprising:

view space transforming the video frames stored in memory based upon the synchronized telemetry data; and

rendering the view space transformed video frames on display with a map.

15. A method for parsing encoded video data to locate one or more video frames that have a desired time period, the method comprising:

receiving encoded video data into a processor;

locating an approximate time stamp associated with a presentation time stamp;

comparing the approximate time stamp to the desired time period;

if the approximate time stamp is proximate to or within the desired time period, decoding a group of encoded video frames and identifying a frame time stamp for each decoded video frame;

if the frame time stamp is within the desired time period, storing the associated frame and corresponding frame time stamp to memory for later retrieval.

16. A method according to claim 15, wherein the encoded video data is encoded in an MPEG format.

17. A method according to claim 15 wherein the approximate time stamp is an estimate of playback time and is associated with an identifiable range.

18. A method according to claim 15, further comprising:

receiving the desired time period from a look-up module wherein the look-up module receives as input a map location and identifies the desired time period for an encoded video data file based on a flight path for an aerial vehicle acquiring the encoded video data file.

19. A method according to claim 15 wherein the encoded video data is encoded in an MPEG-2 format.