METHOD AND APPARATUS FOR IMAGE PROCESSING

• US 20080260285A1
• Filed: 04/23/2008
• Published: 10/23/2008
• Est. Priority Date: 04/23/2007
• Status: Active Grant

• Current Assignee: Gabor Ligeti
• Original Assignee: Gabor Ligeti
• Inventors: Ligeti, Gabor
• Granted Patent: US 8,422,767 B2
• Time in Patent Office: Days
• US Class Current: 382/276
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard

Abstract

The present invention discloses a system and method of transforming a sample of content data by utilizing known samples in a learning best to best determine coefficients for a linear combination of non-linear filter functions and applying the coefficients to the content data in an operational phase.

Claims

1. An apparatus for content processing filtering in an operational phase that is capable of learning from a training set of known discrete content signals and desired transform values by finding the best coefficients for a transform in a learning phase, the transform comprising linear combinations of pre-selected non-linear functions.

2. The apparatus for content processing filtering according to claim 1, wherein the transform allows fast calculations in the operational phase.

3. The apparatus for content processing filtering according to claim 1, wherein the transform is in the format of:

y=a₀s₀+a₁s₁+ . . . +a_ns_n+ƒ

₁(s)(a_1,0s₀+a_1,1s₁+ . . . +a_1,ns_n)+ƒ

₂(s)(a_2,0s₀+a_2,1s₁+ . . . +a_2,ns_n)+ . . . .

4. The apparatus for content processing filtering according to claim 3, wherein the functions have the property of ƒ

(as+c)=ƒ

(s) for each scalar a and c values where s+c denotes the vector (s₀+c,s₁+c, . . .,s_n+c), and

$\underset{0\le <br><br>i\le <br><br>n}{\sum <br><br>}<br><br>a_i=1$

as well as

$\underset{0\le <br><br>i\le <br><br>n}{\sum <br><br>}<br><br>a_{k,i}=1$

, and the desired filter function is in the format of;

F(as+c)=aF(s)+c, wherein the linear combination is rearranged as follows, biased to s₀;

y−

s₀=a₁(s₁−

s₀)+ . . . +a_n(s_n−

s₀)+ƒ

₁(s)(a_1,1(s₁−

s₀+ . . . +a_1,n(s_n−

s₀))+ . . . , thus optimizing this biased variation.

5. The apparatus for content processing filtering according to claim 3, wherein the functions have the format of:

$\frac{\sum <br><br>c_{\mathrm{ij}}<br><br>{<br><br>s_i-s_j<br><br>}^k}{\sum <br><br>c_i<br><br>{<br><br>s_i-s_0<br><br>}^k}<br><br><br><br>\mathrm{or}<br><br><br><br>\frac{\sum <br><br>c_{\mathrm{ij}}<br><br>{<br><br>s_i-s_j<br><br>}^k}{\max <br><br>\left(c_i<br><br>{<br><br>s_i-s_0<br><br>}^k\right)}<br><br><br><br>\mathrm{for}<br><br><br><br>\mathrm{some}<br><br><br><br>k\in <br><br>N.$

6. The apparatus for content processing filtering according to claim 5, wherein the content comprises a neighborhood of pixels, and wherein the neighborhood of pixels is specifically defined by a set of pixels in the proximity of a base pixel s₀ wherein the functions have nonzero c_ij values only for i,j pairs that line up in a specific direction.

7. The apparatus for content processing filtering according to claim 6, wherein the neighborhood of pixels is selected from among a 3×

3 neighborhood of a two dimensional array of pixels, a 5×

5 neighborhood of a two dimensional array of pixels, a 3×

3×

3 neighborhood of a three dimensional array of pixels and a 5×

5×

5 neighborhood of a three dimensional array of pixels.

8. The apparatus for content processing filtering according to claim 6, wherein the neighborhood of pixels is selected from among a m×

m neighborhood of a two dimensional array of pixels and a m×

m×

m neighborhood of a three dimensional array of pixels, wherein m is an integer.

9. The apparatus for content processing filtering according to claim 6, wherein the discrete content signal is selected from among a sound signal, a image signal and a video signal, and wherein the filter further comprises:

a. a continuous sample value with a point defined by the coordinates x,y[, z . . . ] within a set of pixels; and

b. subpixels within the set of pixels, in the form of n*k[*m . . . ] supersampling;

wherein the training set is created by downscaling the discrete content signal, resulting in downscaled content, and providing the downscaled content and each subpixel or specific values at x,y[,z . . . ] of the original discrete content signals for the learning process, using linear combinations of arbitrarily selected non-linear filters.

10. The apparatus for content processing filtering according to claim 1, wherein the discrete content signal is three dimensional in space, and wherein the content processing filtering comprises adding slices to a non-isotropic medical dataset for optimal three dimensional volume reconstruction.

11. The apparatus for content processing filtering according to claim 10, wherein the medical dataset is selected from among MRI image data and CT image data.

12. The apparatus for content processing filtering according to claim 9, wherein the content is used in consumer electronic devices.

13. The apparatus for content processing filtering according to claim 12, wherein the consumer electronic devices are selected from among digital cameras and scanners.

14. The apparatus for content processing filtering according to claim 1, wherein the content is a video stream, wherein the video stream is three-dimensional in time and space, and wherein all three dimensions are utilized for transforming two dimensions of the content.

15. The apparatus for content processing filtering according to claim 4, wherein the content is obtained from a capture device utilizing a Bayer filter, and wherein the content is transformed with filter values for demosaicing the content.

16. The apparatus for content processing filtering according to claim 15, wherein the content comprises a neighborhood of pixels, and wherein the neighborhood of pixels is specifically defined by a set of pixels in the format of:

g<sub>−

2,−

2</sub> b<sub>−

2,−

1</sub> g<sub>−

2,0</sub> b<sub>−

2,1</sub> g<sub>−

2,2</sub>

r<sub>−

1,−

2</sub> g<sub>−

1,−

1</sub> r<sub>−

1,0</sub> g<sub>−

1,1</sub> r<sub>−

1,2</sub>

g<sub>0,−

2</sub> b<sub>0,−

1</sub> g_0,0 b_0,1 g_0,2

r<sub>1,−

2</sub> g<sub>1,−

1</sub> r_1,0 g_1,1 r_1,2

g<sub>2,−

2</sub> b<sub>2,−

1</sub> g_2,0 b_2,1 g_2,2

wherein

$r_{0,0}^{\mathrm{\prime <br><br>}}=\frac{r_{-1,0}-r_{1,0}}{2}$ $b_{0,0}^{\mathrm{\prime <br><br>}}=\frac{b_{0,-1}-b_{0,1}}{2}$ $r_{0,0}=r_{0,0}^{\mathrm{\prime <br><br>}}+\sum <br><br>a_{i,j}^{\mathrm{\prime <br><br>}}<br><br>\left(g_{i,j}-g_{0,0}\right)+\sum <br><br>a_{i,j}^r<br><br>\left(r_{i,j}-r_{0,0}^{\mathrm{\prime <br><br>}}\right)+\sum <br><br>a_{i,j}^b<br><br>\left(b_{i,j}-b_{0,0}^{\mathrm{\prime <br><br>}}\right)+f<br><br>\left(s\right)<br><br>\left(\sum <br><br>a_{i,j}^g<br><br>\left(g_{i,j}-g_{0,0}\right)+\sum <br><br>a_{i,j}^r<br><br>\left(r_{i,j}-r_{0,0}\right)+\sum <br><br>a_{i,j}^b<br><br>\left(b_{i,j}-b_{0,0}\right)\right)+\mathrm{\dots <br><br>}$

and wherein the transform is in the format of;

$\frac{\begin{array}{c}\mathrm{\alpha <br><br>}<br><br>\left(g_{-1,-1}-{g_{0,0}}^2+{\left(g_{0,0}-g_{1,1}\right)}^2\right)+\\ {\mathrm{\beta <br><br>}<br><br>\left(g_{-1,-1}-g_{1,1}\right)}^2+\mathrm{\gamma <br><br>}({\left(r_{-1,0}-r_{1,-2}\right)}^2+\\ {\left(r_{-1,2}-r_{1,0}\right)}^2+{\left(b_{-2,1}-b_{0,-1}\right)}^2+{\left(b_{0,1}-b_{2,-1}\right)}^2)\end{array}}{\begin{array}{c}\sum <br><br>{a_{i,j}^g<br><br>\left(g_{i,j}-g_{0,0}\right)}^2+\\ \sum <br><br>a_{i,j}^r<br><br>{\left(r_{i,j}-r_{0,0}^{\mathrm{\prime <br><br>}}\right)}^2+\sum <br><br>{a_{i,j}^r<br><br>\left(b_{i,j}-b_{0,0}^{\mathrm{\prime <br><br>}}\right)}^2\end{array}}$

17. The apparatus for content processing filtering according to claim 1, wherein the transform removes artifacts of content signals that turn noisy, such as transmitted over noisy channels or digital compression, wherein the transform is taught using a training set containing corresponding noisy and original images in the learning phase.

18. The apparatus for content processing filtering according to claim 1, wherein the transform comprises image enhancement filters selected from among filters for edge enhancement, sharpening, edge detection and zooming.

19. The apparatus for content processing filtering according to claim 1, further comprising:

a pre-emphasis phase and a de-emphasis phase for both the learning phase and the operational phase, wherein the pre-emphasis phase and the de-emphasis phase utilize an appropriate filter pair.

20. The apparatus for content processing filtering according to claim 19, wherein the filter pair comprises an edge enhancement linear convolution filter and its inverse convolution.

32. The apparatus for content processing filtering according to claim 1, further comprising a system for compressing content data using reslet techniques where a low-pass filter and a high-pass filter of a wavelet method is replaced with a down-sampling and deltas to the reslet method.

33. The apparatus for content processing filtering according to claim 1, further comprising a system for compressing content data using reslet techniques where a low-pass filter and a high-pass filter of a wavelet method is replaced with a down-sampling and deltas to the reslet method.

21. A method of transforming content data using a learning phase and a operational phase comprising:

a training phase comprising;

providing at least one sample of high-quality content and a transform, wherein the transform is defined by a linear combination of pre-selected non-linear functions weighted with coefficients;

the at least one sample of high-quality content to produce a degraded quality content, resulting in a representative target sample;

applying the transform to the representative target sample; and

determining values for the coefficients that will best transform the representative target sample to resemble the at least one sample of high-quality content, resulting in optimized coefficients; and

an operational phase comprising;

providing at least one new sample; and

applying the transform and the optimized coefficients to the at least one new sample, resulting in enhanced content data.

22. The method of transforming content data according to claim 21, wherein the content data is image data captured with a camera using a Bayer filter.

23. The method of transforming content data according to claim 22, wherein the transform at least partially comprises a filter for demosaicing.

24. The method of transforming content data according to claim 21, wherein the content data is image data, and wherein the transform at least partially comprises a filter for performing a function selected from among:

resolution enhancement, filtering, noise reduction, color filter demosaicing, zooming, edge enhancement and sharpening.

25. The method of transforming content data according to claim 21, wherein the content data is medical imaging data.

26. The method of transforming content data according to claim 21, the method further comprising a pre-emphasis phase and a de-emphasis phase.

27. A system for transforming content data comprising:

at least one content data source;

an image processing apparatus comprising;

a transform memory containing a set of at least one pre-selected filter function weighted with at least one coefficient;

a known data memory containing at least one known content sample;

a target sample memory;

a down-sampling processor, wherein the down-sampling processor obtains a known content sample from the known data memory and performs a down-sampling transform, resulting in a target sample;

a target sample memory;

a learning phase processor, wherein the learning phase processor obtains a known content sample, at least one pre-selected filter function, and a target sample, and wherein the learning phase processor determines the best coefficients for transforming the target sample to the known content sample, resulting in at least one optimal coefficient; and

an operational phase processor, wherein the operational phase processor obtains at least one preselected filter function, at least one optimal coefficient and content data from the at least one content data source, and wherein the content data is transformed using the at least one filter function and the at least one optimal coefficient, resulting in transformed content data; and

an output.

28. The system for transforming content data according to claim 27, wherein the at least one pre-selected filter function comprises a linear combination of pre-selected non-linear functions.

29. The system for transforming content data according to claim 27, wherein at least one content data source is selected from among:

image capture devices, scanners, video capture devices, medical imaging devices, streaming video sources, and generated video sources.

30. The system for transforming content data according to claim 27, wherein the image processing apparatus is incorporated into a television cable modem, wherein the at least one content source is a low resolution television signal, wherein the set of at least one pre-selected filter function comprises at least partially a resolution enhancement function, and wherein the output is a high resolution television signal.

31. The system for transforming content data according to claim 30, wherein the low resolution television signal comprises a low resolution interlaced television signal, wherein the set of at least one pre-selected filter function comprises at least partially a deinterlacing function, and wherein the output is a high resolution progressive scan television signal.