Image Data Fusion Systems And Methods

• US 20110064327A1
• Filed: 01/30/2009
• Published: 03/17/2011
• Est. Priority Date: 02/01/2008
• Status: Active Grant

• Current Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Original Assignee: Omnivision Technologies Incorporated (Will Semiconductor Co., Ltd. Shanghai)
• Inventors: Kubala, Kenneth S., Ashok, Amit, Tremblay, David, Dagher, Joseph C.
• Granted Patent: US 8,824,833 B2
• Time in Patent Office: Days
• US Class Current: 382/263
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/75 Unsharp masking

Abstract

Systems and methods for image data fusion include providing first and second sets of image data corresponding to an imaged first and second scene respectively. The scenes at least partially overlap in an overlap region, defining a first collection of overlap image data as part of the first set of image data, and a second collection of overlap image data as part of the second set of image data. The second collection of overlap image data is represented as a plurality of image data subsets such that each of the subsets is based on at least one characteristic of the second collection, and each subset spans the overlap region. A fused set of image data is produced by an image processor, by modifying the first collection of overlap image data based on at least a selected one of, but less than all of, the image data subsets.

Claims

1. A method for image data fusion, comprising:

providing a first set of image data corresponding to an imaged first scene;

providing a second set of image data corresponding to an imaged second scene that at least partially overlaps said first scene in an overlap region, the overlap region defining

a first collection of overlap image data as part of the first set of image data and

a second collection of overlap image data as part of the second set of image data;

representing said second collection of overlap image data as a plurality of image data subsets such that each of the subsets is based on at least one characteristic of the second collection of overlap image data, and each subset spans the overlap region; and

producing a fused set of image data by modifying the first collection of overlap image data based on at least a selected one of, but less than all of, said image data subsets,

wherein the representing and producing steps are performed by an image processor.

2. The method of claim 1 wherein

providing the first set of image data includes providing luminance data as part of the first set of image data, such that said first collection of overlap image data includes first luminance data;

providing the second set of image data includes providing luminance data as part of the second set of image data, such that said selected one of said image data subsets is a luminance channel including second luminance data;

representing the second collection of overlap image data includes utilizing luminance as the characteristic of the second collection of overlap image data, and

modifying the first collection of overlap image data includes combining the first and second luminance data.

3. The method of claim 2 wherein providing the second set of image data includes providing grayscale image data as said second luminance data.

4. The method of claim 1 wherein representing said second collection of overlap image data includes filtering the second collection of overlap image data to form the selected image data subset.

5. The method of claim 4 wherein filtering the second collection of overlap image data includes high-pass filtering the second collection of overlap image data such that high-pass filtered data forms the selected image data subset.

6. The method of claim 4 wherein filtering the second collection of overlap image data includes convolution filtering the second collection of overlap image data.

7. The method of claim 1 wherein representing said second collection of overlap image data includes scaling said second collection of overlap image data.

8. The method of claim 7 wherein

providing the second set of image data includes providing intensity information as part of the second set of image data, such that said second collection of overlap image data includes said intensity information, and

scaling said second collection of overlap image data includes changing at least some of said intensity information.

9. The method of claim 8 wherein scaling said second collection of overlap image data includes applying a gain to said intensity information.

10. The method of claim 1 wherein representing said second collection of overlap image data includes transforming at least some of the second collection of overlap image data such that the characteristic of the second collection of overlap image data is represented in a transform domain and at least the selected image data subset includes transformed data.

11. The method of claim 1 wherein

providing said first set of image data includes establishing a first level of zoom with respect to said first scene, and

providing said second set of image data includes establishing a second level of zoom with respect to said second scene that is greater than said first level of zoom.

12. The method of claim 11 wherein establishing the first and second levels of zoom causes said first set of image data to have first angular frequency information based at least in part on said first level of zoom, and causes said second collection of overlap image data to have second angular frequency information based at least in part on said second level of zoom, and said second angular frequency information represents a higher angular frequency than said first angular frequency information.

13. The method of claim 12 wherein generating the first set of image data includes

producing an initial set of image data and

producing said first set of image data from the initial set of image data by upsampling said initial set of image data to increase the angular frequency represented in said first set of image data to a target angular frequency.

14. The method of claim 13 wherein

producing said initial set of image data generates a group of initial data points, and

upsampling causes the first set of image data to include (i) said group of initial data points and (ii) an additional number of data points produced by interpolating between the initial data points for assigning values for each of the additional of data points.

15. The method of claim 13 wherein said upsampling includes matching said increased angular frequency to said second angular frequency such that the target angular frequency of the first set of image data is at least approximately equal to said second angular frequency.

16. The method of claim 15 wherein modifying the first collection of overlap image data includes blending said second collection of overlap image data with said first set of image data such that at least a majority of said fused image data exhibits a resulting angular frequency that is at least approximately equal to said second angular frequency.

17. The method of claim 1 wherein providing the first set of image data includes providing said first collection of overlap image data as first, second and third data channels, and modifying said first collection of overlap image data includes converting the first, second and third data channels to a different set of three data channels.

18. The method of claim 17 wherein providing said first collection of overlap image data includes providing said first, second and third channels as R, G and B channels, respectively, and converting the first, second and third data channels includes converting the R, G and B channels to Y, U and V channels.

19. The method of claim 1 wherein providing the second set of image data includes providing said second collection of overlap image data as first, second and third channels, and representing said second collection of overlap image data includes converting the second collection of overlap image data to a different set of three channels, each of which different channels forms one of said plurality of image data subsets.

20. The method of claim 19 wherein providing said second collection of overlap image data includes providing said first, second and third channels as R, G, and B channels, and converting the second collection of overlap image data includes converting the R, G and B channels to a different set of three channels including Y, U and V channels.

21. The method of claim 20 including selecting said Y channel as the selected subset of overlap image data.

22. The method of claim 1 including modifying said second collection of overlap image data by subtracting therefrom said first set of overlap image data.

23. The method of claim 1 wherein

providing the first set of image data includes

producing an initial set of initial image data and

producing said first set of image data from the set of initial image data by applying a first forward transformation to at least a portion of said initial image data such that said first set of image data is transformed data in a transform domain and represents, in said transform domain, at least some of said portion of the initial image data;

and

representing the second collection of overlap image data includes applying a second forward transformation to at least some of said second set of image data such that the characteristic of the second collection of overlap image data is represented in said transform domain, and the selected image data subset is a transformed data subset.

24. The method of claim 23 wherein modifying said first collection of overlap image data includes merging the selected one of the image data subsets with the first collection of overlap image data in the transform domain to generate a merged data set in the transform domain.

25. The method of claim 24 wherein producing the fused set of image data includes converting the merged data set from the transform domain by applying thereto at least one of (i) a forward transformation and (ii) an inverse transformation.

26. The method of claim 1 wherein producing the fused set of image data further includes

identifying at least one spatial feature that is present at a feature position within the first collection of overlap image data, as an identified spatial feature;

identifying a related representation of each such identified spatial feature, in the selected image data subset, such that each related representation at least approximately corresponds to one of the identified spatial features;

registering a selected related representation as corresponding with a feature position of the corresponding identified spatial feature; and

wherein changing the first collection of overlap image data includes modifying at least one identified spatial feature based on the corresponding related representation of that feature.

27. The method of claim 26 wherein the related representation has a related feature position within the selected image data subset, and identifying the related representation includes identifying a non-zero spatial shift between said related feature position and said feature position.

28. The method of claim 26 wherein identifying said spatial shift includes determining that the spatial shift is caused by parallax between first and second sub-cameras that provide the first and second sets of image data, respectively.

29. The method of claim 26 further including

defining a reference block overlying said feature position and having a shape that overlies a reference portion of said first collection of overlap image data such that the reference portion of image data represents said spatial feature, and

defining a search region within the selected image data subset,

designating a plurality of candidate blocks within the search region, each of which overlies an associated portion of the selected image data subset at a candidate position therein,

wherein said identifying includes

determining a degree of correspondence between (i) the reference portion of image data overlaid by the reference block and (ii) the portion of the selected image data associated with each of the plurality of candidate blocks, and

selecting one of the candidate blocks, based on the degree of correspondence, that exhibits the highest degree of correspondence among the candidate blocks.

30. The method of claim 29 wherein designating the plurality of candidate blocks includes defining a first candidate block and a second candidate block that partially overlap one another.

31. The method of claim 29 wherein registering the selected related representation includes associating the candidate position of the selected candidate block with the feature position, and modifying the at least one identified spatial feature includes changing the reference portion of data based on at least some of the portion of data associated with the selected candidate block.

32. The method of claim 31 wherein generating the first set of image data includes

producing a set of initial image data and

producing said first set of image data from the initial image data by applying a first forward transformation to at least a portion of said initial image data, such that said first set of image data is transformed data in a transform domain, and

wherein said representing the second collection of overlap image data includes applying a second forward transformation to at least some of said second set of image data such that the characteristic of the second collection of overlap image data is represented in said transform domain, and the selected image data subset is a transformed data subset.

33. The method of claim 32 wherein modifying said first collection of overlap image data includes

merging the selected one of the image data subsets with the first collection of overlap image data in the transform domain, to generate a merged data set in the transform domain, and

converting the merged data set from the transform domain by applying thereto at least one of (i) a forward transformation and (ii) an inverse transformation.

34. The method of claim 1, further comprising

configuring a first sub-camera to provide the first set of image data corresponding to a first field of view;

configuring a second sub-camera to provide the second set of image data corresponding to a second field of view; and

arranging an overlap of the first and second fields of view, to generate the overlap region.

35. The method of claim 34, wherein configuring the second sub-camera includes supplying the second sub-camera as a grayscale camera that provides at least a luminance channel.

36. The method of claim 34 wherein

configuring the first sub-camera includes providing a first sensing surface characterized by a first transverse width, and orienting said first sensing surface for receiving the imaged first scene to generate the first set of image data, and

configuring the second sub-camera includes providing a second sensing surface characterized by a second transverse width, and orienting said second sensing surface for receiving the imaged second scene to generate the second set of image data.

37. The method of claim 34 wherein

configuring the first sub-camera includes establishing a first focal length for the first sub-camera, and

configuring the second sub-camera includes establishing a second focal length for the second sub-camera, the first and second focal lengths being different from one another.

38. The method of claim 37 wherein establishing the first focal length causes the first set of image data to exhibit a first level of zoom with respect to said first scene, and establishing the second focal length causes the second set of data to exhibit a second level of zoom with respect to said second scene that is greater than said first level of zoom.

39. The method of claim 37 wherein providing the first set of image data includes generating said first set of image data with a first angular frequency based at least in part on said first focal length, and said providing the second set of image data includes generating said second set of image data with a second angular frequency based at least in part on said second focal length, and said second angular frequency is higher than said first angular frequency.

40. The method of claim 34 wherein

configuring the first sub-camera includes furnishing said first sub-camera with a first sensor region having a first sensing surface, and aligning the first sensor region such that said imaging of said first scene includes projecting an image of said first scene onto said first sensing surface such that said first sensor region generates said first set of image data, and

configuring the second sub-camera includes furnishing said second sub-camera with a second sensor region having a second sensing surface, and aligning the second sensor region such that said imaging of said second scene includes projecting an image of said second scene onto said second sensing surface such that said second sensor region generates said second set of image data.

41. The method of claim 40 wherein

furnishing said first sub-camera with said first sensor region having said first sensing surface, and furnishing said second sub-camera with said second sensor region having said second sensing surface includes supplying the first and second sensing surfaces with sensing surface shapes and surface areas that substantially correspond with one another.

42. The method of claim 40 wherein furnishing said first sub-camera with said first sensor region having said first sensing surface, and furnishing said second sub-camera with said second sensor region having said second sensing surface, include supplying the first and second sensing surfaces as portions of a single image sensor chip.

43. The method of claim 34 wherein

providing the first set of image data includes

producing an initial set of initial image data and

producing said first set of image data from the set of initial image data by applying a first forward transformation to at least a portion of said initial image data such that said first set of image data is transformed data in a transform domain and represents, in said transform domain, at least some of said portion of the initial image data, and

representing the second collection of overlap image data includes applying a second forward transformation to at least some of said second set of image data such that the characteristic of the second collection of overlap image data is represented in said transform domain, and the selected image data subset is a transformed data subset.

44. The method of claim 43 wherein modifying said first collection of overlap image data includes merging the selected one of the image data subsets with the first collection of overlap image data in the transform domain to generate a merged data set in the transform domain.

45. The method of claim 44 wherein producing the fused set of image data includes converting the merged data set from the transform domain by applying thereto at least one of (i) a forward transformation and (ii) an inverse transformation.