Image data fusion systems and methods

• US 8,824,833 B2
• Filed: 01/30/2009
• Issued: 09/02/2014
• Est. Priority Date: 02/01/2008
• Status: Active Grant

• Current Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Original Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Inventors: Dagher, Joseph C., Ashok, Amit, Tremblay, David, Kubala, Kenneth S.
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): MOYER, ANDREW M
• Application Number:
  US12/865,343
• Publication Number:
  US 20110064327A1
• Time in Patent Office: 2,041 Days
• US Class Current: 382/294
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/75 Unsharp masking

Abstract

Systems and methods for image data fusion include providing first and second sets of image data corresponding to an imaged first and second scene respectively. The scenes at least partially overlap in an overlap region, defining a first collection of overlap image data as part of the first set of image data, and a second collection of overlap image data as part of the second set of image data. The second collection of overlap image data is represented as a plurality of image data subsets such that each of the subsets is based on at least one characteristic of the second collection, and each subset spans the overlap region. A fused set of image data is produced by an image processor, by modifying the first collection of overlap image data based on at least a selected one of, but less than all of, the image data subsets.

Claims

1. A method for image data fusion, comprising:

receiving a first set of image data as at least one first data channel and

corresponding to a first imaged first scene;

receiving a second set of image data as at least one second data channel and

corresponding to a second imaged second scene that at least partially overlaps said first scene in an overlap region,

converting at least a portion of the first set of image data corresponding to the overlap region from at least one of the at least one first data channels formats into a plurality of first frequency subsets within a transform domain by applying a first forward transformation;

converting at least a portion of the second set of image data corresponding to the overlap region from at least one of the at least one second data channels into a plurality of second frequency subsets within the transform domain by applying a second forward transformation; and

producing a fused set of image data by replacing at least one of the plurality of first frequency subsets with a corresponding at least one of the plurality of second frequency subsets of a selected second data channel;

wherein;

each of the plurality of first and second frequency subsets correspond to one of a plurality of frequency sub-bands, and

the converting and receiving steps are performed by an image processor, and

the step of converting the second set includes filtering at least the data from the selected second data channel corresponding to the overlap region, and

the first set of image data is received from a color camera and the second set of image data is received from a grayscale camera.

2. The method of claim 1 wherein

the at least one first data channel includes a first luminance channel and the at least one second data channel includes a second luminance channel;

the steps of converting include applying the first and second forward transformations to the first and second luminance channels to produce a plurality of first and second luminance frequency subsets, respectively; and

the step of producing includes replacing at least one of the plurality of first luminance frequency subsets with a corresponding portion of the plurality of second luminance frequency subsets.

3. The method of claim 2 wherein the second luminance channel is a grayscale channel.

4. The method of claim 1 wherein filtering at least the portion of the second set includes high-pass filtering the portion of the second set such that high-pass filtered data forms the portion of the second frequency subsets from the selected second data channel.

5. The method of claim 1 wherein filtering the portion of the second set includes convolution filtering the second set of image data.

6. The method of claim 1 wherein converting the second set of image data includes scaling said second set of image data.

7. The method of claim 6 wherein

receiving the second set of image data includes providing intensity information as part of the second set of image data, and

scaling said second set of image data includes changing at least some of said intensity information.

8. The method of claim 7 wherein scaling said second set of image data includes applying a gain to said intensity information.

9. The method of claim 1 wherein

receiving said first set of image data includes establishing a first level of zoom with respect to said first scene, and

receiving said second set of image data includes establishing a second level of zoom with respect to said second scene that is greater than said first level of zoom.

10. The method of claim 9 wherein establishing the first and second levels of zoom causes said first set of image data to have first angular frequency information based at least in part on said first level of zoom, and causes said second collection of overlap image data to have second angular frequency information based at least in part on said second level of zoom, and said second angular frequency information represents a higher angular frequency than said first angular frequency information.

11. The method of claim 10 wherein receiving the first set of image data includes

receiving an initial set of image data and

producing said first set of image data from the initial set of image data by upsampling said initial set of image data to increase the angular frequency represented in said first set of image data to a target angular frequency.

12. The method of claim 11 wherein

receiving said initial set of image data generates a group of initial data points, and

upsampling causes the first set of image data to include (i) said group of initial data points and (ii) an additional number of data points produced by interpolating between the initial data points for assigning values for each of the additional of data points.

13. The method of claim 11 wherein said upsampling includes matching said increased angular frequency to said second angular frequency such that the target angular frequency of the first set of image data is at least approximately equal to said second angular frequency.

14. The method of claim 13 wherein replacing the at least one of the plurality of first frequency subsets additionally includes blending said portion of the plurality of second frequency subsets with said at least one of the plurality of first frequency subsets such that at least a majority of said fused image data exhibits a resulting angular frequency that is at least approximately equal to said second angular frequency.

15. The method of claim 1 wherein receiving the first set of image data includes providing said first image data as first, second and third data channels, and converting said first set includes converting the first, second and third data channels to a different set of three data channels.

16. The method of claim 15 wherein receiving said first set of image data includes providing said first, second and third channels as R, G and B channels, respectively, and converting the first, second and third data channels includes converting the R, G and B channels to Y, U and V channels.

17. The method of claim 1 wherein receiving the second set of image data includes providing said second set of image data as first, second and third channels, converting said second set of image data includes converting the second set of image data to a different set of three channels, each of which different channels forms one of said plurality of image data subsets.

18. The method of claim 17 wherein receiving said second set of image data includes providing said first, second and third channels as R, G, and B channels, and converting the second set of image data includes converting the R, G and B channels to a different set of three channels including Y, U and V channels.

19. The method of claim 18 including selecting said Y channel as the at least one of the at least one second data channels.

20. The method of claim 1 including modifying said second set of image data by subtracting therefrom said first set of image data.

21. The method of claim 1 wherein producing the fused set of image data includes converting the merged data set from the transform domain by applying thereto at least one of (i) a forward transformation and (ii) an inverse transformation.

22. The method of claim 1 wherein producing the fused set of image data further includes:

identifying at least one spatial feature that is present at a feature position within the first set of image data corresponding to the overlap region, as an identified spatial feature;

identifying a related representation of each such identified spatial feature, in the selected second data channel, such that each related representation at least approximately corresponds to one of the identified spatial features;

registering a selected related representation as corresponding with a feature position of the corresponding identified spatial feature; and

wherein replacing the at least one of the plurality of first frequency subsets includes first modifying at least one identified spatial feature within the first data set based on the corresponding related representation of that feature within the second data set.

23. The method of claim 22 wherein the related representation has a related feature position within the selected second data channel, and identifying the related representation includes identifying a non-zero spatial shift between said related feature position and said feature position.

24. The method of claim 23 wherein identifying said spatial shift includes determining that the spatial shift is caused by parallax between first and second sub-cameras that provide the first and second sets of image data, respectively.

25. The method of claim 22 further including

defining a reference block overlying said feature position and having a shape that overlies a reference portion of said first set of image data, corresponding to the overlap region, such that the reference portion of image data represents said spatial feature, and

defining a search region within the subsets from the selected second data channel,

designating a plurality of candidate blocks within the search region, each of which overlies an associated portion of the selected data channel at a candidate position therein,

wherein said identifying includes

determining a degree of correspondence between (i) the reference portion of image data overlaid by the reference block and (ii) the portion of the selected second data channel associated with each of the plurality of candidate blocks, and

selecting one of the candidate blocks, based on the degree of correspondence, that exhibits the highest degree of correspondence among the candidate blocks.

26. The method of claim 25 wherein designating the plurality of candidate blocks includes defining a first candidate block and a second candidate block that partially overlap one another.

27. The method of claim 25 wherein registering the selected related representation includes associating the candidate position of the selected candidate block with the feature position, and modifying the at least one identified spatial feature includes changing the reference portion of data based on at least some of the portion of data associated with the selected candidate block.

28. The method of claim 27 wherein generating the first set of image data includes

receiving a set of initial image data and

producing said first set of image data from the initial image data by applying a first forward transformation to at least a portion of said initial image data, such that said first set of image data is transformed data in a transform domain, and

wherein said representing the second collection of overlap image data includes applying a second forward transformation to at least some of said second set of image data such that the characteristic of the second collection of overlap image data is represented in said transform domain, and the selected image data subset is a transformed data subset.

29. The method of claim 1, further comprising

configuring a first sub-camera to provide the first set of image data corresponding to a first field of view;

configuring a second sub-camera to provide the second set of image data corresponding to a second field of view; and

arranging an overlap of the first and second fields of view, to generate the overlap region.

30. The method of claim 29, wherein configuring the second sub-camera includes supplying the second sub-camera as a grayscale camera that provides at least a luminance channel.

31. The method of claim 29 wherein

configuring the first sub-camera includes providing a first sensing surface characterized by a first transverse width, and orienting said first sensing surface for receiving the imaged first scene to generate the first set of image data, and

configuring the second sub-camera includes providing a second sensing surface characterized by a second transverse width, and orienting said second sensing surface for receiving the imaged second scene to generate the second set of image data.

32. The method of claim 29 wherein

configuring the first sub-camera includes establishing a first focal length for the first sub-camera, and

configuring the second sub-camera includes establishing a second focal length for the second sub-camera, the first and second focal lengths being different from one another.

33. The method of claim 32 wherein establishing the first focal length causes the first set of image data to exhibit a first level of zoom with respect to said first scene, and establishing the second focal length causes the second set of data to exhibit a second level of zoom with respect to said second scene that is greater than said first level of zoom.

34. The method of claim 32 wherein receiving the first set of image data includes generating said first set of image data with a first angular frequency based at least in part on said first focal length, and said providing the second set of image data includes generating said second set of image data with a second angular frequency based at least in part on said second focal length, and said second angular frequency is higher than said first angular frequency.

35. The method of claim 29 wherein

configuring the first sub-camera includes furnishing said first sub-camera with a first sensor region having a first sensing surface, and aligning the first sensor region such that said imaging of said first scene includes projecting an image of said first scene onto said first sensing surface such that said first sensor region generates said first set of image data, and

configuring the second sub-camera includes furnishing said second sub-camera with a second sensor region having a second sensing surface, and aligning the second sensor region such that said imaging of said second scene includes projecting an image of said second scene onto said second sensing surface such that said second sensor region generates said second set of image data.

36. The method of claim 35 wherein

furnishing said first sub-camera with said first sensor region having said first sensing surface, and furnishing said second sub-camera with said second sensor region having said second sensing surface includes supplying the first and second sensing surfaces with sensing surface shapes and surface areas that substantially correspond with one another.

37. The method of claim 35 wherein furnishing said first sub-camera with said first sensor region having said first sensing surface, and furnishing said second sub-camera with said second sensor region having said second sensing surface, include supplying the first and second sensing surfaces as portions of a single image sensor chip.

38. The method of claim 1 wherein:

the first and second frequency subsets each include at least a low frequency block and a high frequency block, and

replacing the at least one of the plurality of first frequency subsets includes replacing the high frequency block of the first subset with the corresponding high frequency block of the second subset.

39. The method of claim 1 wherein:

the first and second frequency subsets each include at least a low frequency block and a high frequency block, and

replacing the at least one of the plurality of first frequency subsets includes replacing the low frequency block of the first subset with the corresponding low frequency block of the second subset.