Low latency pyramid processor for image processing systems

• US 20050265633A1
• Filed: 05/25/2005
• Published: 12/01/2005
• Est. Priority Date: 05/25/2004
• Status: Abandoned Application

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: van der Wal, Gooitzen Siemen, Piacentino, Michael Raymond, Bergen, James Russell, Burt, Peter Jeffrey
• Application Number:
  US11/136,908
• Publication Number:
  US 20050265633A1
• Time in Patent Office: Days
• US Class Current: 382/302
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

A video processor that uses a low latency pyramid processing technique for fusing images from multiple sensors. The imagery from multiple sensors is enhanced, warped into alignment, and then fused with one another in a manner that provides the fusing to occur within a single frame of video, i.e., sub-frame processing. Such sub-frame processing results in a sub-frame delay between a moment of capturing the images to the display of the fused imagery.

Claims

1. A method of processing video from a plurality of sensors, comprising:

creating a Laplacian pyramid for a portion of a frame of a first video signal;

creating a second Laplacian pyramid for a portion of a frame for a second video signal;

combining the first and second Laplacian pyramids at each pyramid level to form composite levels; and

constructing, using the composite levels, a portion of a fused video signal containing information from the first and second video signals.

2. The method of claim 1, wherein the portion of a frame is a plurality of lines of a video signal.

3. The method of claim 1, wherein the combining step further comprises:

determining weights associated with each video signal; and

using weights to control an amount of each video signal to form the fused video signal.

4. The method of claim 3, wherein the determining step further comprises:

performing a statistical analysis of the pyramid levels to determine the weights.

5. The method of claim 4 wherein the using step further comprises:

applying the weights to the pyramid levels to determine the amount of each level to combine to form the composite levels.

6. The method of claim 1, further comprising:

enhancing the first and second video signals before creating the Laplacian pyramid.

7. The method of claim 6, wherein said enhancing step comprises at least one of non-uniformity compensation, Bayer filtering, noise reduction, and scaling.

8. The method of claim 1 further comprising:

warping the first video signal into alignment with the second video signal prior to creating the Laplacian pyramid.

9. The method of claim 1 wherein creating a Laplacian pyramid for the first video signal step comprises:

receiving a plurality of lines of a frame of the first video signal;

filtering the plurality of lines to produce a filtered signal; and

subtracting the plurality of lines from the filtered signal to produce a pyramid level.

10. The method of claim 9 wherein the creating step further comprises:

filtering the filtered signal to produce a second filtered signal;

subtracting the filtered signals from the second filtered signal to produce a second pyramid level;

decimating the second filtered signal prior to filtering the second filtered signals to produce a third filtered signal; and

subtracting the third filtered signal from the decimated second filtered signal to produce a third pyramid level.

11. The method of claim 10 wherein the constructing step further comprising:

delaying at least one composite level by a predefined number of lines;

applying an inverse pyramid transform to the composite levels to construct the portion of the fused video signal.

12. A video processor for fusing video signals from at least two video signal sources:

a warper for aligning a first video signal with a second video signal on a sub-frame and sub-pixel basis;

a first pyramid transform module for creating a first image pyramid containing first levels from a portion of the first video signal, where the portion is less than a frame;

a second pyramid transform module for creating a second image pyramid containing second levels from a portion of the first video signal, where the portion is less than a frame;

a fuser, coupled to the first and second pyramid transform modules, for fusing, on a level-by-level basis, the levels in the first and second pyramids; and

an inverse pyramid transform module, coupled to the fuser, for reconstructing a portion of a fused video signal from the fused levels.

13. The video processor of claim 12 further comprising;

an adaptation module for statistically analyzing the levels of the first and second pyramids to create weights that are used by the fuser to control level fusing.