SYNCHRONIZED VIDEO AND SYNTHETIC VISUALIZATION SYSTEM AND METHOD

• US 20110171612A1
• Filed: 02/18/2011
• Published: 07/14/2011
• Est. Priority Date: 07/22/2005
• Status: Active Grant

• Current Assignee: Appareo Systems LLC (AGCO Corporation)
• Original Assignee: Appareo Systems LLC (AGCO Corporation)
• Inventors: Allen, Robert M., Wiig, Johan A., Schubert, Seth J., Halvorson, Jacob A., Batcheller, Barry D., Gelinske, Joshua N., Ohlsen, Tyler C., Weinmann, Robert V.
• Granted Patent: US 8,944,822 B2
• Time in Patent Office: Days
• US Class Current: 434/35
• CPC Class Codes: G09B 19/165 Control of aircraft
• CPC Class Codes: G09B 9/08 for teaching control of aircraft, e.g. Link trainer
• CPC Class Codes: G09B 9/30 Simulation of view from aircraft

Abstract

The present invention presents a flight training and synthetic visualization system, which comprises a fully mobile, self-contained data recording unit including a desktop graphics software engine for creating a virtual model of the flight capable of playing back the recorded trip, synchronized with a real-time video or imagery recording of the actual flight with a view from the cockpit of the aircraft as a pilot would actually view the flight, along with ambient audio of the cockpit. This allows for the user of the simulation to view both modeled data of the flight, as well as actual time-sequenced still images or video of the flight. The two sources of data are synched in time so that real video images of the aircraft as it is flying at a specific point in time is displayed in the simulation at the same moment as the rendered visualization of the flight.

Claims

1. A synchronized video and synthetic visualization system for a mobile object traversing a travel path, which system comprises:

a mobile video recorder associated with the object and adapted to record video images along the travel path;

a simulator adapted for simulating movement of said object along said travel path; and

a synchronizer connected to said video recorder and said simulator, said synchronizer being adapted for synchronizing video images from said video recorder with a simulation of said object'"'"'s movement along said travel path.

2. The system according to claim 1, which includes:

a time-referencing function adapted for time-referencing said video images with said simulated movement of said object along said travel path.

3. The system according to claim 1, which includes:

a display device connected to said synchronizer and adapted for simultaneously displaying video images synchronized with a synthetic visualization associated with the object in motion.

4. The system according to claim 3, which includes:

said synthetic visualization comprising a simulated travel path for said object.

5. The system according to claim 4 wherein said object comprises an aircraft and said travel path comprises a flight path.

6. The system according to claim 1, which includes:

sensors mounted on said object and adapted for sensing data associated with the object as it moves along said travel path; and

said sensor data being synchronized with said video images and said travel path simulation.

7. The system according to claim 1, which includes:

an audio recorder connected to said synchronizer and adapted for synchronizing audio recording data with said video images and said sensor data.

8. The system according to claim 7, which includes:

an audio output device adapted for generating an audio signal synchronized with said display device output and simulating movement of said object along said travel path.

9. The system according to claim 7, which includes:

a navigation device mounted on the object and adapted for determining geo-referenced positions of said object along said travel path; and

said geo-referenced positions of said object along said travel path being synchronized with said playback of recorded video and said visually-modeled simulation.

10. The system according to claim 1, which includes:

a processor connected to said synchronizer; and

a display device connected to said processor and adapted for displaying said video images synchronized with said time-referenced object positions.

11. The system according to claim 10, which includes:

said processor being adapted to synchronize a 3-D travel path for said object with said time-referenced object locations and output a graphical representation of said travel path to said display device.

12. The system of claim 1, wherein the video recording device is a digital camcorder.

13. The system of claim 1, which includes:

said object comprising a vehicle;

said vehicle having a data storage device;

a video recording device installed on the exterior of the vehicle;

said video recording device being directed towards a control element of the vehicle; and

said video recording device recording real-time video of the control element, including time reference, and storing the recorded data on the vehicle data storage device.

14. The system of claim 1, wherein said vehicle is an aircraft, which system includes:

navigational and flight information data stored on said data storage device;

a secondary computer system adapted for creating a 3-D recreation of a flight path of the aircraft based on said navigational and flight information;

a digital terrain model stored onto said data storage device and including an area of the Earth'"'"'s surface including at least a portion of the flight path;

a graphics software engine on said secondary computer system;

a 3-D display of a 3-D recreation including;

said terrain model;

a representation of the aircraft superimposed on the terrain model; and

a data ribbon representing the flight path superimposed on the terrain model;

altitude readings computed from said navigational and flight information at pre-defined intervals along the flight path;

a 3-D display comprising a vertical synthetic flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model computed using said altitude readings and said navigational and flight information;

said flight wall being subdivided graphically into a vertically-oriented checkerboard configuration comprising multiple rectangular segments separated by multiple, horizontally-spaced vertical striations each representing a pre-defined horizontal distance and multiple, vertically-stacked horizontal striations each representing a pre-defined vertical distance, said pre-defined vertical and horizontal distances corresponding to altitude and distance of travel along said flight path respectively;

wherein said display device is adapted for dynamically displaying in 3-D with said graphics software engine said flight wall including said vertical and horizontal striations below said flight path data ribbon; and

wherein said display device is adapted for dynamically displaying in 3-D with said graphics software engine the progress along said flight path of the aircraft on top of said flight wall and over said terrain model; and

dynamically displaying aircraft altitudes at respective rectangular segments along said flight path.

15. A method of synchronizing a simulation of a travel path of a mobile object with images from object positions along said travel path, which method comprises the steps of:

providing a processor associated with said object;

providing as input to said processor data corresponding to a simulation of said object'"'"'s path along said travel path;

providing the object with an image-recording device;

recording images associated with said object'"'"'s path along said travel path;

providing as input to said processor said recorded images;

synchronizing with said processor said simulation with said images;

providing a display device connected to and receiving output from said processor; and

simultaneously displaying on said display device said synchronized simulation and images.

16. The method according to claim 15, which includes the additional steps of:

providing said object with a geo-reference positioning device; and

providing geo-referenced positions from said positioning device to said processor; and

synchronizing with said processor said geo-referenced positions with said simulation and said images.

17. The method according to claim 15, which includes the additional steps of:

providing a time-referencing device; and

time-referencing said simulation on said images.

18. The method according to claim 15 wherein said processor comprises one of an on-board processor located on said object and a remote processor receiving downloads of said simulation data and said images from said object.

19. The method according to claim 15 wherein said object comprises a ground vehicle and said simulation data include position, speed and heading of said ground vehicle.

20. The method according to claim 15 wherein said object comprises an aircraft and said simulation data include position, speed, heading and altitude of said aircraft.

21. The method according to claim 15 wherein said processor creates a 3-D simulation of said object'"'"'s movement along said travel path.

22. The method according to claim 15, which includes the additional step of:

time-stamping points in said simulation corresponding to positions of said object along said travel path; and

examining said simulation data at a time-stamped point along said travel path.

23. The method according to claim 22, which includes the additional step of:

downloading said simulation data and said images to a memory device;

searching said memory device for simulation data and said images corresponding to a respective timestamp; and

displaying simulation data and images corresponding to said timestamp.

24. The method according to claim 15 wherein said image-recording device comprises a video recorder.

25. The method according to claim 15 wherein said simulation data and said images are synchronized in real-time and post-processed.

26. The method according to claim 15 wherein said simulation data and said images are stored on one or more data storage devices on said object.

27. The method according to claim 15 wherein said simulation data and said images are transmitted wirelessly and stored remotely on data storage devices remote from said object.

28. The method according to claim 15 wherein said display device renders a 3-D simulation model of said object'"'"'s movement along said travel path.

29. The method according to claim 15 wherein said synchronization step includes the additional steps of:

comparing all recorded timestamps;

determining if proper synchronization has occurred; and

providing an output signal alerting an operator if said simulation and said images were not properly synchronized.

30. The method according to claim 15, which includes the additional step of:

providing a meta-data stream including periodic synchronization markers;

providing external data consisting of one or more of location data, speed data and heading data; and

correlating external data sync markers and meta-data sync markers; and

synchronizing two or more streams of data recorded by separate devices.

31. The method according to claim 30, which includes the additional steps of:

providing said meta-data with timestamp data;

providing a video datastream;

correlating said timestamp data to specific positions along said travel path in the video datastream and the audio datastream.

32. The method according to claim 30, which includes the additional steps of:

providing said meta-data with timestamp data;

providing an audio datastream;

correlating said timestamp data to specific positions along said travel path in the audio datastream.

33. The method according to claim 15, which includes the additional steps of:

providing a multimedia container;

providing video data;

providing audio data;

storing said video and audio data in said multimedia container; and

creating a separate sync data file from said video and audio data as the video and audio data are recorded;

periodically time stamping information in the sync data file; and

providing references to both the multimedia container and external data to the sync data file.

Syncing specific points in the video data; and

syncing specific points in the external data.

34. The method according to claim 15, which includes the additional steps of:

creating a proprietary data format including one or more of video data, audio data, location data and attitude data;

creating internal sync markers in real-time as the data streams are recorded;

providing automatic synchronization points for the synchronized playback of the recorded data;

storing multiple streams of data in a single multimedia container;

transferring the proprietary multimedia container data contents to a single file;

providing software for reading, interpreting and playing back said recorded data streams in a synchronized manner;

correlating the data streams with the internal sync markers.

35. The method according to claim 15, which includes the additional steps of:

providing a secondary computer system;

providing and downloading navigational and flight information from readable media to said secondary computer system;

computing with said secondary computer system a 3-D recreation of a flight path of the aircraft based on said navigational and flight information;

computing with said secondary computer system a digital terrain model for an area of the Earth'"'"'s surface including at least a portion of the flight path;

installing and executing a graphics software engine on said secondary computer system;

generating with said graphics software engine and displaying with said display device a 3-D display of said 3-D recreation including;

said terrain model;

a representation of the aircraft superimposed on the terrain model; and

a data ribbon representing the flight path superimposed on the terrain model;

computing altitude readings from said navigational and flight information at pre-defined intervals along the flight path;

using said altitude readings and said navigational and flight information to compute a 3-D display comprising a vertical synthetic flight wall extending downwardly from said flight path data ribbon to a ground level on said terrain model;

subdividing said flight wall graphically into a vertically-oriented checkerboard configuration comprising multiple rectangular segments separated by multiple, horizontally-spaced vertical striations each representing a pre-defined horizontal distance and multiple, vertically-stacked horizontal striations each representing a pre-defined vertical distance, said pre-defined vertical and horizontal distances corresponding to altitude and distance of travel along said flight path respectively;

dynamically displaying in 3-D on said display device with said graphics software engine said flight wall including said vertical and horizontal striations below said flight path data ribbon;

dynamically displaying in 3-D on said display device with said graphics software engine the progress along said flight path of the aircraft on top of said flight wall and over said terrain model; and

dynamically displaying aircraft altitudes at respective rectangular segments along said flight path.

36. A method of providing a trip simulation comprising a video playback element and a rendered 3D playback element, the method comprising the steps:

providing a vehicle for creating a simulation recording;

providing a data storage device;

providing a video recording device and equipping said vehicle with the device;

providing a a means of providing a time reference, wherein said time reference is associated with a point in time as said vehicle completes said trip;

recording real-time video with said video recording device and storing said recorded data onto said data storage device;

providing a plurality of sensors and installing said sensors into said vehicle;

providing real-time data with said plurality of sensors and storing said recorded data onto said data storage device;

providing a remote computer comprising a processor, data storage device, and display, said computer adapted for rendering stored sensor data into a 3D simulation playback;

downloading recorded sensor, time reference, and video data onto said computer;

rendering recorded flight sensor data into a 3D simulation playback element;

synching video playback element with the 3D simulation playback element using recorded time references; and

replaying the synched video playback element and simulation playback element on a display device.