Method for fusing images and apparatus therefor

• US 5,325,449 A
• Filed: 05/15/1992
• Issued: 06/28/1994
• Est. Priority Date: 05/15/1992
• Status: Expired due to Term

• Current Assignee: Sarnoff Corporation (SRI International, Inc.)
• Original Assignee: David Sarnoff Research Center, Inc.
• Inventors: Hingorani, Rajesh, van der Wal, Gooitzen S., Kolczynski, Raymond J., Burt, Peter J.
• Primary Examiner(s): Boudreau, Leo H.
• Application Number:
  US07/884,098
• Time in Patent Office: 774 Days
• Field of Search: 382/41, 382/22, 382/49, 382/56, 364/715.02, 358/426, 358/133
• US Class Current: 382/240
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A method for fusing two or more source images to form a composite image with extended information content and apparatus for forming the composite image from the source images is disclosed. Each source image is decomposed into a number of source images of varying resolution. The decomposed source images are analyzed using directionally sensitive operators to generate a set of oriented basis functions characteristic of the information content of the original images. The oriented basis functions for the composite image are then selected from those of the different source images and the inverse of the decomposition performed to construct the composite image. Apparatus for fusing two or more source images to form a composite image is also disclosed.

Claims

1. A method for forming a composite image from N source images where N is greater than one comprising the steps of:

(a) decomposing each source image I_N into a plurality L of sets of oriented component patterns P_N (m, L), where m is the number of patterns in each of the L sets for the N^th source image;

(b) computing a saliency measure S_N (m, L) for each component pattern P_N (m, L);

(c) selecting component patterns from the component pattern sets P_N (m, L) using the saliency measures S_N (m, L) for each component pattern to form a set of oriented component patterns P_c (m, L) of the composite image; and

(d) constructing the composite image from the set of oriented component patterns P_c (m, L).

2. The method of claim 1 further comprising the step of aligning the source images prior to step (a).

3. The method of claim 1 wherein the step of decomposing each source image I_N into a plurality L of sets of oriented component patterns P_N (m, L) comprises applying oriented operators to an image feature at each picture element of each source image.

4. The method of claim 3 wherein each oriented component pattern P_N (m, L) is the gradient of the local image intensity at each picture element of each source image in a particular direction.

5. The method of claim 4 wherein the gradient operators are the first derivatives of the image intensity at each picture element in first and second orthogonal directions in each source image.

6. The method of claim 5 wherein the gradient operators also include the first derivatives of the image intensity in third and fourth directions in each source image which are orthogonal two one another and at a forty five degree angle to the first direction.

7. The method of claim 3 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting the component pattern having the greatest saliency.

8. The method of claim 7 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting the component pattern having the amplitude with the greatest absolute magnitude.

9. The method of claim 3 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting a linear combination of the amplitudes of the component patterns for each image.

10. The method of claim 9 wherein the linear combination of the amplitudes of the component patterns for each image is a weighted average of the amplitudes of the component patterns of the source images.

11. The method of claim 10 wherein the weight assigned to a particular component pattern in each of the source images depends upon the normalized saliency of the particular component pattern.

12. The method of claim 10 wherein the weighted average of the amplitudes of the component patterns is the weighted root mean square of the amplitudes of the particular component pattern and other components within a local neighborhood.

13. The method of claim 10 wherein the weight assigned to a particular component pattern in each of the source images is the weighted root means square of the amplitudes of the particular component pattern and other components within a local neighborhood.

14. The method of claim 1

wherein step (a) further comprises the step of decomposing each source image into a plurality K of images I_K of different resolution and then decomposing each of the images I_K into a plurality L of sets of oriented component patterns P_N (m, K, L);

wherein step (b) comprises computing a saliency measure S_N (m, K, L) for each component pattern P_N (m, K L);

wherein step (c) comprises computing component patterns from the source image component pattern sets P_N (m, K, L) using the saliency measures S_N (m, K L) for each component pattern to form a set of oriented component patterns P_c (m, K, L) of the composite image; and

wherein step (c) comprises constructing the composite image from the set of oriented component patterns P_c (m, K, L).

15. The method of claim 14 wherein the step of decomposing each image I_K into a plurality L of sets of oriented component patterns P_N (m, K, L) comprises applying oriented operators to a feature of each picture element of each source image.

16. The method of claim 15 wherein each oriented component pattern P_N (m, K, L) is the gradient of the local image intensity at each picture element of each source image in a particular direction.

17. The method of claim 16 wherein the gradient operators are the first derivatives of the image intensity at each picture element in first and second orthogonal directions in each source image.

18. The method of claim 17 wherein the gradient operators also include the first derivatives of the image intensity in third and fourth directions in each source image which are orthogonal two one another and at a forty five degree angle to the first direction.

19. The method of claim 14 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting a linear combination of the amplitudes of the component patterns for each image.

20. The method of claim 19 wherein the linear combination of the amplitudes of the component patterns for each image is a weighted average of the amplitudes of the component patterns of the source images.

21. The method of claim 20 wherein the weight assigned to a particular component pattern in each of the source images depends upon the normalized saliency of the particular component pattern.

22. The method of claim 20 wherein the weighted average of the amplitudes of the component patterns is the weighted root mean square of the amplitudes of the particular component pattern and other components within a local neighborhood.

23. The method of claim 1 wherein the step of decomposing each source image I_N comprises forming two or more sets of wavelets oriented in different directions in each source image.

24. The method of claim 23 comprising first and second sets of wavelets in first and second orthogonal directions.

25. The method of claim 24 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting the component pattern having the greatest saliency.

26. The method of claim 24 wherein the step of selecting component patterns from the source image pattern sets P_N (m, L) for each value of m and L comprises selecting a linear combination of the amplitudes of the component patterns for each image.

27. The method of claim 1

wherein step (a) further comprises the step of decomposing each source image into a plurality K of images I_K of different resolution and then decomposing each of the images I_K into a plurality L of sets of oriented component patterns P_N (m, K, L);

wherein step (b) comprises computing a saliency measure S_N (m, K, L) for each component pattern P_N (m, K L) of each image N;

wherein step (c) comprises computing component patterns from the source image component pattern sets P_N (m, K, L) using the saliency measures S_N (m, K L) for each component pattern to form a set of oriented component patterns P_c (m, K, L) of the composite image; and

wherein step (c) comprises constructing the composite image from the set of oriented component patterns P_c (m, K, L).

28. A method for forming a composite image from N source images where N is greater than one comprising the steps of:

(a) decomposing each source image into a plurality K of images I_K of different resolution and then decomposing each of the images I_K into a plurality L of sets of oriented component patterns P_N (m, K, L), where m is the number of patterns in each of the L sets for the N^th source image;

(b) computing a saliency measure S_N (m, K, L) for each component pattern P_N (m, K, L) of each image N at each resolution level K;

(c) selecting component patterns from the source image component pattern sets P_N (m, K, L) using the saliency measures S_N (m, K, L) for each component pattern to form a set of oriented component patterns P_c (m, K, L) of the composite image; and

(d) constructing the composite image from the set of oriented component patterns P_c (m, K, L).

29. The method of claim 28 wherein the step of decomposing each source image I_N into a plurality L of sets of oriented component patterns P_N (m, L) comprises applying oriented operators to an image feature at each picture element of each source image.

30. The method of claim 29 wherein each oriented component pattern P_N (m, L) is the gradient of the local image intensity at each picture element of each source image in a particular direction.

31. The method of claim 30 wherein the gradient operators are the first derivatives of the image intensity at each picture element in first and second orthogonal directions in each source image.

32. The method of claim 31 wherein the gradient operators also include the first derivatives of the image intensity in third and fourth directions in each source image which are orthogonal two one another and at a forty five degree angle to the first direction.

33. The method of claim 28 wherein the step of selecting component patterns from the source image pattern sets P_N (m, K, L) for each value of m, K and L comprises selecting the component pattern having the greatest saliency.

34. The method of claim 33 wherein the step of selecting component patterns from the source image pattern sets P_N (m, K, L) for each value of m, K and L comprises selecting the component pattern having the amplitude with the greatest absolute magnitude.

35. The method of claim 33 wherein the step of selecting component patterns from the source image pattern sets P_N (m, K, L) for each value of m, K and L comprises selecting a linear combination of the amplitudes of the component patterns for each image.

36. The method of claim 35 wherein the linear combination of the amplitudes of the component patterns for each image is a weighted average of the amplitudes of the component patterns of the source images.

37. The method of claim 36 wherein the step of decomposing each source image I_N comprises forming two or more sets of wavelets oriented in different directions in each source image.

38. The method of claim 37 comprising first and second sets of wavelets in first and second orthogonal directions.

39. Apparatus for forming a composite image from a plurality of N source images, where N is greater than one, comprising:

means for decomposing each source image I_N into a plurality L of sets of oriented component patterns P_N (m, L), where m is the number of patterns in each of the L sets for the N^th source image;

means for computing a saliency measure S_N (m, L) for each component pattern P_N (m, L) connected to the decomposing means;

means for selecting component patterns from the component pattern sets P_N (m, L) using the saliency measures S_N (m, L) for each component pattern to form a set of oriented component patterns P_c (m, L) of the composite image; and

means for constructing the composite image from the set of oriented component patterns P_c (m, L).

40. The apparatus of claim 39 further comprising means for decomposing each source image into a plurality K of images I_K of different resolution prior to the means for decomposing each source image.

41. The apparatus of claim 40 wherein the means for decomposing each source image I_N comprises a electrical circuit for performing two different gradient operations on each of the source images I_N.

42. The apparatus of claim 39 wherein the means for computing a saliency measure S_N (m, L) for each component pattern P_N (m, L) includes electrical circuitry for calculating the weights for the different component patterns.

43. The apparatus of claim 39 wherein the means for computing a saliency measure S_N (m, L) for each component pattern P_N (m, L) comprises means for calculating the root mean square weight for the amplitudes different component patterns.

44. The apparatus of claim 41 wherein the means for constructing each composite image I_N comprises a electrical circuit for performing the same two gradient operations on each set of composite patterns.