SYSTEMS AND METHODS OF SCENE AND ACTION CAPTURE USING IMAGING SYSTEM INCORPORATING 3D LIDAR

• US 20120044476A1
• Filed: 05/02/2011
• Published: 02/23/2012
• Est. Priority Date: 05/09/2008
• Status: Active Grant

• Current Assignee: BAE Systems Space & Mission Systems Incorporated
• Original Assignee: Ball Aerospace & Technologies Corporation (Ball Corporation)
• Inventors: Earhart, Ronald P., Craig, Rex M., Grund, Christian J., Staple, Bevan, Weimer, Carl S.
• Granted Patent: US 9,041,915 B2
• Time in Patent Office: Days
• US Class Current: 356/4.01
• CPC Class Codes: G01S 17/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/88 Lidar systems specially adapted for specific applications
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/933 of aircraft or spacecraft
• CPC Class Codes: G01S 3/7867 Star trackers navigation using star trackers G01C21/025
• CPC Class Codes: G01S 7/003 Transmission of data between radar, sonar or lidar systems and remote stations
• CPC Class Codes: G01S 7/4813 Housing arrangements
• CPC Class Codes: G01S 7/4814 of transmitters alone
• CPC Class Codes: G01S 7/4816 of receivers alone

Abstract

The present invention pertains to systems and methods for the capture of information regarding scenes using single or multiple three-dimensional LADAR systems. Where multiple systems are included, those systems can be placed in different positions about the imaged scene such that each LADAR system provides different viewing perspectives and/or angles. In accordance with further embodiments, the single or multiple LADAR systems can include two-dimensional focal plane arrays, in addition to three-dimensional focal plane arrays, and associated light sources for obtaining three-dimensional information about a scene, including information regarding the contours of the objects within the scene. Processing of captured image information can be performed in real time, and processed scene information can include data frames that comprise three-dimensional and two-dimensional image data.

Claims

1. A system, comprising:

A) a first LIDAR system, including;

a first 2D sensor, wherein the first aperture provides image information to the first 2D sensor;

a first 3D sensor, wherein the first aperture provides image information to the first 3D sensor;

a first illumination source;

B) a second LIDAR system, including;

a first 2D sensor, wherein the first aperture provides image information to the first 2D sensor;

a first 3D sensor, wherein the first aperture provides image information to the first 3D sensor;

a first illumination source;

C) a processor, wherein image information obtained by the first LIDAR system and image information obtained by the second LIDAR system is combined.

2. The system of claim 1, wherein the first LIDAR system has a first field of view and the second LIDAR system has a second field of view.

3. The system of claim 2, wherein combining the image information includes creating one of a super frame and a new field of view.

4. The system of claim 1, wherein the first illumination source of at least one of the first LIDAR system and the second LIDAR system is operable to provide a different amount of illumination to different areas of a scene within a field of view of the at least one of the first LIDAR system and the second LIDAR system.

5. The system of claim 1, wherein the first 2D sensor of the first LIDAR system includes a full multiple color sensor.

6. The system of claim 5, wherein the first LIDAR system includes a second 2D sensor.

7. The system of claim 6, wherein the second 2D sensor is a high resolution sensor.

8. The system of claim 1, further comprising:

a global positioning system sensor, wherein geolocation information is provided by the global positioning system sensor.

9. The system of claim 1, wherein the first processor outputs a sequence of frames, wherein each of the frames in the sequence contain image information from at least the first and second LIDAR systems, wherein a spacing in time between adjacent frames in such that output frames are provided at a rate at least equal to a rate at which 3D data frames are acquired.

10. The system of claim 1, wherein the first LIDAR system includes a first aperture that provides signals to the first 2D sensor and the first 3D sensor of the first LIDAR system, and wherein the second LIDAR system includes a first aperture that provides signals to the first 2D sensor and the first 3D sensor of the second LIDAR system.

11. A method, comprising:

obtaining a first 2D image of a first portion of a scene from a first view point;

obtaining a first 3D image of the first portion of the scene from the first view point, wherein the first 3D image includes a first point cloud;

determining the location of the first view point;

associating a location to at least some of the points in the first point cloud;

orthorectifying the first 2D image;

associating a location to at least some of the pixels in the 2D image;

fusing pixels in the first 2D image to points in the first point cloud to create a first fused image.

12. The method of claim 11, further comprising:

outputting the first fused image.

13. The method of claim 11, wherein the locations associated with points in the first point cloud are made with respect to an absolute location reference, whereby the points in the first point cloud associated with locations are geolocated.

14. The method of claim 11, further comprising:

obtaining a second 2D image of a second portion of a scene from a secondview point;

obtaining a second 3D image of the second portion of the scene from the second view point, wherein the second 3D image includes a second point cloud;

determining the location of the second view point;

associating a location to at least some of the points in the second point cloud;

orthorectifying the second 2D image;

associating a location to at least some of the pixels in the 2D image;

fusing pixels in the second 2D image to points in the second point cloud to create a second fused image.

15. The method of claim 14, further comprising:

stitching first and second fused images using location info to create a composite image;

outputting the first composite image.

16. The method of claim 15, wherein the first viewpoint is different than the second viewpoint.

17. The method of claim 16, wherein the first 2D and first 3D images are obtained by a first LADAR system, and wherein the second 2D and second 3D images are obtained by a second LADAR system.

18. The method of claim 17, wherein the first 2D and first 3D images are taken from a first perspective, wherein the second 2D and second 3D images are taken from a second perspective, and wherein the first composite image presents a view of the scene from a third perspective.

19. The method of claim 18, wherein the first 2D and first 3D images are taken at a first point in time, wherein the second 2D and second 3D images are taken at a second point in time, and wherein the first composite image is output at a third point in time that is within 3 times the period of time between the first point in time and the second point in time.

20. The method of claim 11, further comprising:

calculating the physical parameters of the scene from at least one of an orthorectified and a fused image of the scene and ancillary data;

choosing a location in the image of the scene to embed an image of a virtual object;

calculating an effect of the local environment physical properties on the physical properties of the virtual object;

modifying the virtual object to conform to a physical response to the environmental forcing according to specified rules and the instant properties of the modified object;

embedding the modified virtual object image in the 3D image scene;

providing the 3D image scene as an electronic virtual image for further processing or display.

21. The method of claim 20, wherein the altered virtual model is used as the virtual image for a subsequent frame in which the virtual object is embedded.

22. The method of claim 11, further comprising:

evaluating objects in the scene to determine the object contours;

draping a virtual skin on the objects conforming to the determined contours.

23. A system, comprising:

a first aperture;

a first 2D sensor, wherein the first 2D sensor is operable to output 2D image data of a first scene, and wherein the first 2D sensor is provided with a first signal by the first aperture;

a first light source;

a beam steering element, wherein the beam steering element is operable to steer a beam output by the first light source;

a first 3D sensor, wherein the first 3D sensor is a flash sensor operable to output 3D image data of the first scene, wherein the first 3D sensor is provided with a second signal by the first aperture, and wherein the second signal includes light output by the first light source and reflected from at least a portion of the first scene;

a location determination device;

a real time processor, wherein a frame of 2D image data is fused with a frame of 3D image data to create a fused data frame, wherein absolute location information output by the location determination device is associated with each frame of fused data, wherein subsequent frames of fused data are generated in real time.

24. The system of claim 23, further comprising:

at least one of a source of artificial image information and a non-3D sensor provides at least one of artificially generated scene image information or information from other non-3D sensors that is fused with a frame of 3D image data to create a fused data frame.

25. The method of claim 24, wherein the real-time fused frame image information in a scene is manipulated with 3 degrees of freedom (DOF) by direct contact with a touch-pad screen viewer for the scene and wherein individual objects in the scene, multiple object in the scene, or the whole scene can be interacted with and manipulated with at least 3 DOF, and wherein, methods to interact and manipulate with individual objects in the scene, multiple object in the scene, or the whole scene or individual objects within the scene with at least 3 DOF using non-contact methods.

26. The system of claim 23, comprising:

wherein the real-time fused frame image data in a scene is manipulated with 3 degrees of freedom (DOF) by direct contact with a touch-pad screen viewer for the scene, wherein at least a first object in the scene can be interacted with and manipulated with 3 DOF using non-contact methods.