SYSTEMS AND METHODS FOR GENERATING IMPROVED ENVIRONMENTAL DISPLAYS FOR VEHICLES

• US 20170345321A1
• Filed: 11/05/2015
• Published: 11/30/2017
• Est. Priority Date: 11/05/2014
• Status: Active Grant

• Current Assignee: Sierra Nevada Co. LLC
• Original Assignee: Sierra Nevada Corporation
• Inventors: Cross, Jack Philip, Brandstetter, III, William Edward, Valigura, Igor, Bode, Justin Michael, Jourdan, Damien Bruno, Howard, David McMartin, Pickens, Rodney Lee, Sweet, Timothy L., Gleason, Joshua D., Hansen, Jr., Joseph H.
• Granted Patent: US 10,410,531 B2
• Time in Patent Office: Days
• CPC Class Codes: B64D 43/00 Arrangements or adaptations of instruments arrangements of cameras B64D47/08; aeronautical measuring instruments per se G01C
• CPC Class Codes: B64D 45/08 optical
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: G01C 23/005 Flight directors indicating arrangements specially adapted for rotary gyroscopes G01C19/32
• CPC Class Codes: G01S 13/953 mounted on aircraft
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G08G 5/0021 located in the aircraft
• CPC Class Codes: G08G 5/0052 for cruising combined instruments indicating more than one navigational value G01C23/00
• CPC Class Codes: G08G 5/0086 for monitoring terrain radar or analogous systems specially adapted for terrain avoidance G01S13/935
• CPC Class Codes: G08G 5/045 Navigation or guidance aids, e.g. determination of anti-collision manoeuvers radar or analogous systems specially adapted for anti-collision between aircraft G01S13/933

Abstract

An imaging system for a moving vehicle aggregates pre-existing data with sensor data to provide an image of the surrounding environment in real-time. The pre-existing data are combined with data from one or more 3-D sensors, and 2-D information from a camera, to create a scene model that is rendered for display. The system accepts data from a 3-D sensor, transforms the data into a 3-D data structure, fuses the pre-existing scene data with the 3-D data structure and 2-D image data from a 2-D sensor to create a combined scene model, and renders the combined scene model for display. The system may also weight aspects of data from first and second sensors to select at least one aspect from the first sensor and another aspect from the second sensor; wherein fusing the pre-existing scene data with the sensor data uses the selected aspect from the first sensor and the selected aspect from the second sensor.

Claims

1. A machine implemented method for rendering a synthetic display of a scene including terrain and obstacles, comprising:

(a) loading pre-existing scene data;

(b) accepting sensor data from at least one environmental sensor;

(c) transforming the sensor data into a 3-D data cloud;

(d) fusing the pre-existing scene data with the transformed sensor data to create a combined scene model comprising a multi-resolution 3-D data structure;

(e) rendering the combined scene model into an image signal; and

(f) displaying an image of a scene in response to the image signal.

2. The method of claim 1, wherein sensor data are accepted from a plurality of environmental sensors, at least one of which is a sensor that provides 3-D point cloud data.

3. The method of claim 2, wherein at least one of the plurality of environmental sensors is a 2-D sensor from which 3-D point cloud data is derived.

4. The method of claim 1, wherein the at least one environmental sensor is selected from the group consisting of one or more of a radar sensor, a lidar sensor, a sonar sensor, and a video camera.

5. The method of claim 2, wherein the sensor data include sensor data from first and second environmental sensors, the method further comprising:

(g) weighting aspects of the sensor data from the first and second environmental sensors to select an aspect of the sensor data from the first environmental sensor and an aspect of the sensor data from the second environmental sensor; and

wherein fusing the pre-existing scene data with the sensor data uses the selected aspect of the sensor data from the first environmental sensor and the selected aspect of the sensor data from the second environmental sensor.

6. The method of claim 1, wherein rendering the combined scene model comprises merging the combined scene model with imagery from at least one camera.

7. The method of claim 6, wherein the at least one camera is selected from the group consisting of one or more of a visible spectrum camera, a stereo camera, an infrared camera, and a spectral imaging camera.

8. The method of claim 1, wherein fusing the pre-existing scene data with the sensor data provides a higher resolution at a distance closer to a generation point of the sensor data than at a distance farther from the generation point.

9. A non-transitory, machine readable storage medium having stored thereon instructions for performing a method for rendering a display of a scene including terrain and obstacles, the instructions comprising machine executable code which, when executed by a machine, causes the machine to:

(a) load pre-existing scene data from database;

(b) accept sensor data in the form of a 3-D point cloud from at least one environmental sensor;

(c) transform the sensor data from the at least one environmental sensor into a multi-resolution 3-D data structure;

(d) fuse the pre-existing scene data with the sensor data to create a combined scene model comprising a multi-resolution 3-D data structure; and

(e) render the combined scene model to fuse with 2-D imagery.

10. The machine-readable storage medium of claim 9, wherein the at least one environmental sensor is selected from the group consisting of at least one of a radar sensor and a lidar sensor.

11. The machine-readable storage medium of claim 10, wherein the sensor data include sensor data from first and second environmental sensors, and wherein the code, when executed by a machine, further causes the machine to:

weigh aspects of the sensor data from the first and second environmental sensors to select an aspect of the sensor data from the first environmental sensor and an aspect of the sensor data from the second environmental sensor; and

fuse the pre-existing scene data with the sensor data using the selected aspect of the sensor from the first environmental sensor and the selected aspect of the sensor data from the second environmental sensor.

12. The machine-readable storage medium of claim 9, wherein the code, when executed by a machine, further causes the machine to merge the combined scene model with imagery from at least one camera.

13. The machine-readable storage medium of claim 12, wherein the at least one camera is selected from the group consisting of one or more of a visible spectrum camera, a stereo camera, an infrared camera, and a spectral imaging camera.

14. The machine-readable storage medium of claim 9, wherein fusing the pre-existing scene data with the sensor data provides a higher resolution at a distance closer to a generation point of the sensor data than at a distance farther from the generation point.

15. A system for rendering a synthetic display of a scene including terrain and obstacles, the system comprising:

a data source comprising pre-existing scene data including terrain data and obstacle data;

an environmental sensor providing real-time sensor data in the form of a 3-D point cloud;

a memory having stored therein a set of machine-executable instructions that are executable to render a synthetic display of a scene including terrain and obstacles from the pre-existing scene data and the real-time sensor data; and

a processor operatively coupled to the memory and configured to execute the machine executable instructions to;

(a) transform the sensor data from the at least one environmental sensor into a multi-resolution 3-D data structure;

(b) fuse the pre-existing scene data with the real-time sensor data to create a combined scene model that comprises a multi-resolution 3-D data structure; and

(c) render the combined scene model to fuse with 2-D video imagery to provide an image signal representing the synthetic display of the scene.

16. The system of claim 15, wherein the environmental sensor is selected from the group consisting of at least one of a radar sensor and a lidar sensor.

17. The system of claim 15, further comprising a second environmental sensor providing real time sensor data, wherein the set of machine-executable instructions includes instructions to replace at least one aspect of the sensor data from the first environmental sensor with sensor data from the second environmental sensor.

18. The system of claim 17, wherein execution of the instructions to replace at least one aspect of the sensor data from the first environmental sensor is at least partially determined by a comparison of a quality of the sensor data from the first environmental sensor relative to a corresponding quality of the sensor data from the second environmental sensor.

19. The system of claim 15, further comprising a camera, wherein the processor is configured to render the combined scene model on a display by merging the combined scene model with images from the camera.

20. The system of claim 19, wherein the camera is selected from the group consisting of one or more of a visible spectrum camera, a stereo camera, an infrared camera, and a spectral imaging camera.

21. An imaging system for use in a moving vehicle to provide a synthetic image of an environment in a vicinity of the vehicle in real-time, comprising:

a first data source comprising pre-existing scene data including terrain data and obstacle data;

a second data source providing 3-D point cloud data including terrain data and obstacle data;

a third data source providing 2-D image data including terrain data and obstacle data;

a scene fusion module configured to receive and combine the pre-existing scene data, the 3-D point cloud data, and the 2-D image data to produce a combined scene model including the pre-existing scene data updated in real time by the 3-D point cloud data and the 2-D image data;

an image rendering module configured to receive and process the combined scene model to produce an image signal representing the combined scene model; and

a visual display configured to display a virtual view of the environment in the vicinity of the vehicle in response to the image signal.

22. The system of claim 21, wherein:

the first data source includes a database containing the pre-existing scene data and a first process and classify module configured to yield the pre-existing scene data as an a priori scene data structure;

the second data source includes a real-time 3-D environmental sensor and a second process and classify module configured to yield the 3-D point cloud data as a 3-D scene data structure;

the third data source includes a camera and a third process and classify module configured to yield the 2-D image data as a video image data structure; and

wherein the scene fusion module is configured to receive and combine the a priori scene data structure, the 3-D scene data structure, and the video image data structure.

23. The system of claim 22, wherein the a priori scene data structure includes 2-D and 3-D a priori scene data structures, and the video image data structure includes at least one of a 2-D scene data structure and a 3-D scene data structure.

24. The system of claim 22, wherein at least one of the a priori scene data structure, the 3-D scene data structure, and the video image data structure includes navigation data.

25. The system of claim 22, wherein the real-time 3-D environmental sensor is selected from the group consisting of one or more of a radar sensor and a lidar sensor.

26. The system of claim 22, wherein the second data source includes first and second real-time 3-D environmental sensors, each providing real time sensor data as 3-D point cloud data, wherein the second process and classify module is configured to yield the 3-D point cloud data from the first and second sensors as first and second 3-D scene data structures, respectively.

27. The system of claim 26, wherein the scene fusion module is configured to replace at least one aspect of the first 3-D scene data structure sensor with a corresponding aspect of the second 3-D scene data structure.

28. The system of claim 27, wherein the scene fusion module is configured to compare a quality of the first 3-D scene data structure to a corresponding quality of the second 3-D scene data structure.

29. The system of claim 21, wherein the first data source is configured to be updatable by the combined scene model.

30. The system of claim 21, wherein the scene fusion module is configured to (a) receive a first scene data point set from one of the first, second, and third data sources, (b) receive a second scene data point set from another of the first, second, and third data sources, (c) validate the data points of the second scene data point set against the data points of the first scene data point set to create an updated scene data point set, and (d) produce the combined scene model from the updated scene data point set.

31. The system of claim 31, wherein the first scene data point set is a set of data points, each of which has been determined to have met a predetermined validity criterion, and wherein the scene fusion module is configured to validate the data points of the second scene data point set by determining whether each of the data points of the second scene data point set meets the predetermined validity criterion.

32. The system of claim 31, wherein the scene fusion module is further configured to use only those data points of the second scene data point set that have been determined to have met the predetermined validity criterion to create the updated scene data point set.