Orthoscopic fusion platform

• US 9,948,914 B1
• Filed: 05/02/2016
• Issued: 04/17/2018
• Est. Priority Date: 05/06/2015
• Status: Active Grant

• Current Assignee: US Department of The Air Force (U.S. Department Of Defense)
• Original Assignee: United States Of America As Represented By The Secretary Of The Air Force
• Inventors: Pinkus, Alan R, Dommett, David W
• Primary Examiner(s): Li, Tracy Y
• Application Number:
  US15/143,670
• Time in Patent Office: 715 Days
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/20164 Salient point detection; Corner detection
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof

Abstract

A mobile three-dimensional sensor configuration is provided with a first sensor of a second type mounted proximate to the first sensor of a first type. A second sensor of the second type is mounted proximate to a second sensor of the first type. The first and second sensors are coupled to a first platform operable to vary the azimuth and elevation of the first and second sensors of the first and second types. The first and second platforms are mounted on a translational drive configured to independently translate the first and second platforms with respect to one another. A controller is configured to receive images sensors of and further configured to create a pair fused images of the first sensors of the first and second type and the second sensors of the first and second type. The controller is still further configured to display the pair of fused images.

Claims

1. A mobile three-dimensional sensor configuration, comprising:

a first sensor of a first type;

a first sensor of a second type mounted proximate to the first sensor of the first type;

a second sensor of the first type;

a second sensor of the second type mounted proximate to the second sensor of the first type;

the first sensors of the first and second types coupled to a first platform operable to vary the azimuth and elevation of the first sensors of the first and second types;

the second sensors of the first and second types coupled to a second platform operable to vary the azimuth and elevation of the second sensors of the first and second types;

the first and second platforms mounted on a translational drive, the translational drive configured to independently translate the first and second platforms with respect to one another; and

a controller configured to receive images from the first and second sensor of the first type and the first and second sensor of the second type;

the controller further configured to create a pair fused images of the first sensors of the first and second type and the second sensors of the first and second type; and

a display,

wherein the controller still further configured to display the pair of fused images on the display.

2. The mobile three-dimensional sensor configuration of claim 1, wherein the display is a three-dimensional display.

3. The mobile three-dimensional sensor configuration of claim 1, wherein the controller is further configured to adjust the translation, azimuth, and elevation of the first and second platforms.

4. The mobile three-dimensional sensor configuration of claim 1, wherein the mobile three dimensional sensor configuration is incorporated into a manned vehicle.

5. The mobile three-dimensional sensor configuration of claim 1, wherein the mobile three dimensional sensor configuration is incorporated into an unmanned vehicle.

6. The mobile three-dimensional sensor configuration of claim 5, wherein the unmanned vehicle is an autonomous vehicle.

7. The mobile three-dimensional sensor configuration of claim 1, where the first and second sensors of the first type are selected from a group consisting of:

a visible (VIS) spectrum camera, a low-light-level (LLL) TV camera, a near infrared (NIR) sensor, a short-wave infrared (SWIR) sensor, a medium-wave infrared (MWIR) sensor, a long-wave infrared (LWIR;

thermal) sensor, and combinations thereof.

8. The mobile three-dimensional sensor configuration of claim 7, wherein the first and second sensors of the second type are selected from a group consisting of:

a visible (VIS) spectrum camera, a low-light-level (LLL) TV camera, a near infrared (NIR) sensor, a short-wave infrared (SWIR) sensor, a medium-wave infrared (MWIR) sensor, a long-wave infrared (LWIR;

thermal) sensor, and combinations thereof.

9. The mobile three-dimensional sensor configuration of claim 1, wherein the controller comprises:

a memory;

a processor; and

program code resident in the memory and configured acquire images from the first and second sensors of the first and second types, the program code further configured, when executed by the processor, to;

acquire individual images from the first sensor of the first type;

the first sensor of the second type, the second sensor of the first type, and the second sensor of the second type;

determine a list of matching stereo point pairs between the images from the first sensor of the first type and the second sensor of the first type;

determine a list of matching stereo point pairs between the images from the first sensor of the second type and the second sensor of the second type; and

store a frame of information including the images and the lists of matched point pairs into a set of frames.

10. The mobile three-dimensional sensor configuration of claim 9, wherein the program code is further configured to display acquired images, the program further configured, when executed by the processor, to:

choose a number of frames from the set of frames where there is movement;

build a three-dimensional model of salient points of the images from the first sensor of the first type and the second sensor of the first type using multiple frames of data from the set of frames;

build a three-dimensional model of salient points of the images from the first sensor of the second type and the second sensor of the second type using multiple frames of data from the set of frames;

register the two three-dimensional models using cross correlation, warp the three-dimensional model generated from the sensors of the second type such that salient points are collocated with the salient points of the three-dimensional model generated from the sensors of the first type;

determine a relationship between salient points on the warped three-dimensional model and locations on a two-dimensional image from the sensors of the second type;

register the image from the first sensor of the second type with the image of the first sensor of the first type and fuse the images;

register the image from the second sensor of the second type with the image from the second sensor of the first type and fuse the images; and

display the fused images on the display.

11. The mobile three-dimensional sensor configuration of claim 9, wherein the program code is further configured, when executed by the processor, to determine the list of matching stereo point pairs for the sensors of the first type or the second type by:

selecting the image from the first sensor;

performing a multi-scale Harris Corner Detection on the image from the first sensor and extracting a first list of salient points;

selecting the image from the second sensor;

performing a multi-scale Harris Corner Detection on the image from the second sensor and extracting a second list of salient points;

receiving azimuth and elevation information about the first sensor;

receiving azimuth and elevation information about the second sensor;

receiving separation distance between the first and second sensor;

mapping the first list of salient points with the second list of salient points using azimuth, elevation, and distance information to match salient points between the first list of salient points and the second list of salient points; and

output list of matched salient point pairs.