Synchronized video and synthetic visualization system and method

• US 8,944,822 B2
• Filed: 02/18/2011
• Issued: 02/03/2015
• Est. Priority Date: 07/22/2005
• Status: Active Grant

• Current Assignee: Appareo Systems LLC (AGCO Corporation)
• Original Assignee: Appareo Systems LLC (AGCO Corporation)
• Inventors: Gelinske, Joshua N., Allen, Robert M., Batcheller, Barry D., Halvorson, Jacob A., Ohlsen, Tyler C., Schubert, Seth J., Weinmann, Robert V., Wiig, Johan A.
• Primary Examiner(s): Utama, Robert J
• Application Number:
  US13/030,901
• Publication Number:
  US 20110171612A1
• Time in Patent Office: 1,446 Days
• Field of Search: 434/35
• US Class Current: 434/35
• CPC Class Codes: G09B 19/165 Control of aircraft
• CPC Class Codes: G09B 9/08 for teaching control of aircraft, e.g. Link trainer
• CPC Class Codes: G09B 9/30 Simulation of view from aircraft

Abstract

The present invention presents a flight training and synthetic visualization system, which comprises a fully mobile, self-contained data recording unit including a desktop graphics software engine for creating a virtual model of the flight capable of playing back the recorded trip, synchronized with a real-time video or imagery recording of the actual flight with a view from the cockpit of the aircraft as a pilot would actually view the flight, along with ambient audio of the cockpit. This allows for the user of the simulation to view both modeled data of the flight, as well as actual time-sequenced still images or video of the flight. The two sources of data are synched in time so that real video images of the aircraft as it is flying at a specific point in time is displayed in the simulation at the same moment as the rendered visualization of the flight.

Claims

1. A synchronized video and synthetic visualization system for a mobile object comprising an aircraft traversing a flight path, which system comprises:

a mobile video recorder associated with the aircraft and adapted to record video images from different camera angles along the flight path;

a data recording device associated with the aircraft and adapted to record navigational and flight information;

a secondary computer system adapted for creating a 3-D recreation of a flight path of the aircraft based on the navigational and flight information;

a digital terrain model stored onto the data recording device and including an area of the Earth'"'"'s surface including at least a portion of the flight path;

a graphics software engine on the secondary computer system;

a first 3-D display of a 3-D recreation including;

the terrain model;

a representation of the aircraft superimposed on the terrain model; and

a data ribbon representing the flight path superimposed on the terrain model;

altitude readings computed from the navigational and flight information at pre-defined intervals along the flight path;

a second 3-D display comprising a vertical synthetic flight wall extending downwardly from the flight path data ribbon to a ground level on the terrain model computed using the altitude readings and the navigational and flight information;

the flight wall being subdivided graphically into a vertically-oriented checkerboard configuration comprising multiple rectangular segments separated by multiple, horizontally-spaced vertical striations each representing a pre-defined horizontal distance and multiple, vertically-stacked horizontal striations each representing a pre-defined vertical distance, the pre-defined vertical and horizontal distances corresponding to altitude and distance of travel flight along the flight path respectively;

a display device adapted for dynamically displaying in 3-D with the graphics software engine the flight wall including the vertical and horizontal striations below the flight path data ribbon;

wherein the display device is adapted for dynamically displaying in 3-D with the graphics software engine the progress along the flight path of the aircraft on top of the flight wall and over the terrain model; and

dynamically displaying aircraft altitudes at respective rectangular segments along the flight path;

a synchronizer connected to the video recorder and the secondary computer system, the synchronizer being adapted for synchronizing video images from the video recorder with the 3-D recreation of a flight path; and

the simulator adapted for analyzing the vehicle flight path.

2. The system according to claim 1, which includes:

a time-referencing function adapted for time-referencing said video images with said simulated movement of said object along said flight path.

3. The system according to claim 1, which includes:

a display device connected to said synchronizer and adapted for simultaneously displaying video images synchronized with a synthetic visualization associated with the object in motion.

4. The system according to claim 3, which includes:

said synthetic visualization comprising a simulated flight for said object.

5. The system according to claim 1, which includes:

sensors mounted on said object and adapted for sensing data associated with the object as it moves along said flight path; and

said sensor data being synchronized with said video images and said flight path simulation.

6. The system according to claim 1, which includes:

an audio recorder connected to said synchronizer and adapted for synchronizing audio recording data with said video images and said sensor data.

7. The system according to claim 6, which includes:

an audio output device adapted for generating an audio signal synchronized with said display device output and simulating movement of said object along said flight path.

8. The system according to claim 6, which includes:

a navigation device mounted on the object and adapted for determining geo-referenced positions of said object along said flight path; and

said geo-referenced positions of said object along said flight path being synchronized with said playback of recorded video and said visually-modeled simulation.

9. The system according to claim 1, which includes:

a processor connected to said synchronizer; and

a display device connected to said processor and adapted for displaying said video images synchronized with said time-referenced object positions.

10. The system according to claim 9, which includes:

said processor being adapted to synchronize a 3-D flight path for said object with said time-referenced object locations and output a graphical representation of said flight path to said display device.

11. The system of claim 1, wherein the video recording device is a digital camcorder.

12. The system of claim 1, which includes:

said vehicle having a data storage device;

a video recording device installed on the exterior of the vehicle;

said video recording device being directed towards a control element of the vehicle; and

said video recording device recording real-time video of the control element, including time reference, and storing the recorded data on the vehicle data storage device.

13. A method of synchronizing a simulation of a flight path of a mobile object comprising a vehicle with images from object positions along said flight path, which method comprises the steps of:

providing a processor associated with said object;

providing as input to said processor simulation data corresponding to a simulation of said object'"'"'s path along said flight path;

providing the object with an image-recording device;

recording images from different camera angles associated with said object'"'"'s path along said flight path;

providing as input to said processor said recorded images;

synchronizing with said processor said simulation with said images;

providing a display device connected to and receiving output from said processor;

simultaneously displaying on said display device said synchronized simulation and images;

analyzing said flight path with said processor;

wherein said simulation data comprises a digital terrain model for an area of the Earth'"'"'s surface including at least a portion of the flight path;

a representation of the aircraft superimposed on the terrain model;

a data ribbon representing the flight path superimposed on said terrain model; and

a vertical synthetic flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model;

wherein said flight wall is graphically subdivided into a vertically-oriented checkerboard configuration comprising multiple rectangular segments separated by multiple, horizontally-spaced vertical striations each representing a pre-defined horizontal distance and multiple, vertically-stacked horizontal striations each representing a pre-defined vertical distance, said pre-defined vertical and horizontal distances corresponding to altitude and distance of travel flight along said flight path respectively;

wherein said flight wall is displayed dynamically in 3-D on said display device with said graphics software engine; and

wherein the progress along said flight path is displayed dynamically in 3-D on said display device on top of said flight wall and over said terrain model.

14. The method according to claim 13, which includes the additional steps of:

providing said object with a geo-reference positioning device; and

providing geo-referenced positions from said positioning device to said processor; and

synchronizing with said processor said geo-referenced positions with said simulation and said images.

15. The method according to claim 13, which includes the additional steps of:

providing a time-referencing device; and

time-referencing said simulation on said images.

16. The method according to claim 13 wherein said processor comprises one of an on-board processor located on said object and a remote processor receiving downloads of said simulation data and said images from said object.

17. The method according to claim 13 wherein said simulation data include position, speed and heading of said vehicle.

18. The method according to claim 13 wherein said simulation data include position, speed, heading and altitude of said aircraft.

19. The method according to claim 13 wherein said processor creates a 3-D simulation of said object'"'"'s movement along said flight path.

20. The method according to claim 13, which includes the additional step of:

time-stamping points in said simulation corresponding to positions of said object along said flight path; and

examining said simulation data at a time-stamped point along said flight path.

21. The method according to claim 20, which includes the additional step of:

downloading said simulation data and said images to a memory device;

searching said memory device for simulation data and said images corresponding to a respective timestamp; and

displaying simulation data and images corresponding to said timestamp.

22. The method according to claim 13 wherein said image-recording device comprises a video recorder.

23. The method according to claim 13 wherein said simulation data and said images are synchronized in real-time and post-processed.

24. The method according to claim 13 wherein said simulation data and said images are stored on one or more data storage devices on said object.

25. The method according to claim 13 wherein said simulation data and said images are transmitted wirelessly and stored remotely on data storage devices remote from said object.

26. The method according to claim 13 wherein said display device renders a 3-D simulation model of said object'"'"'s movement along said flight path.

27. The method according to claim 13 wherein said synchronization step includes the additional steps of:

comparing all recorded timestamps;

determining if proper synchronization has occurred; and

providing an output signal alerting an operator if said simulation and said images were not properly synchronized.

28. The method according to claim 13, which includes the additional step of:

providing a meta-data stream including periodic synchronization markers;

providing external data consisting of one or more of location data, speed data and heading data; and

correlating external data sync markers and meta-data sync markers; and

synchronizing two or more streams of data recorded by separate devices.

29. The method according to claim 28, which includes the additional steps of:

providing said meta-data with timestamp data;

providing a video datastream; and

correlating said timestamp data to specific positions along said flight path in the video datastream and the audio datastream.

30. The method according to claim 28, which includes the additional steps of:

providing said meta-data with timestamp data;

providing an audio datastream;

correlating said timestamp data to specific positions along said flight path in the audio datastream.

31. The method according to claim 13, which includes the additional steps of:

providing a multimedia container;

providing video data;

providing audio data;

storing said video and audio data in said multimedia container; and

creating a separate sync data file from said video and audio data as the video and audio data are recorded;

periodically time stamping information in the sync data file; and

providing references to both the multimedia container and external data to the sync data file;

syncing specific points in the video data; and

syncing specific points in the external data.

32. The method according to claim 13, which includes the additional steps of:

creating a proprietary data format including one or more of video data, audio data, location data and attitude data;

creating internal sync markers in real-time as the data streams are recorded;

providing automatic synchronization points for the synchronized playback of the recorded data;

storing multiple streams of data in a single multimedia container;

transferring the proprietary multimedia container data contents to a single file;

providing software for reading, interpreting and playing back said recorded data streams in a synchronized manner; and

correlating the data streams with the internal sync markers.