Automatically Resizing Demosaicked Full-Color Images Using Edge-Orientation Maps Formed In The Demosaicking Process

• US 20110032269A1
• Filed: 08/05/2009
• Published: 02/10/2011
• Est. Priority Date: 08/05/2009
• Status: Abandoned Application

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Lukac, Rastislav, Shiohara, Ryuichi
• Application Number:
  US12/536,254
• Publication Number:
  US 20110032269A1
• Time in Patent Office: Days
• US Class Current: 345/606
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling

Abstract

Automatically resizing demosaicked full-color images using edge-orientation maps formed in the demosaicking process. In a first example embodiment, a method for automatic upsampling of a demosaicked image includes several acts. First, a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image are received. Next, pixels of the demosaicked image are filled into an upsampled image. Then, edge-orientation values of pixels of the edge-orientation map are filled into an upsampled edge-orientation map. Next, an interpolation direction is determined for each pixel in which upsampling of the demosaicked image should be performed using the upsampled edge-orientation map. Finally, missing pixels in the upsampled image are estimated by performing interpolation along the interpolation direction using available pixels surrounding each missing pixel location.

Claims

1. A method for automatic upsampling of a demosaicked image, the method comprising the following acts:

i) receiving a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) filling pixels of the demosaicked image into an upsampled image;

iii) filling edge-orientation values of pixels of the edge-orientation map into an upsampled edge-orientation map;

iv) determining an interpolation direction for each pixel in which upsampling of the demosaicked image should be performed using the upsampled edge-orientation map; and

v) estimating missing pixels in the upsampled image by performing interpolation along the interpolation direction determined at act iv) using available pixels surrounding each missing pixel location.

2. The method as recited in claim 1, wherein the act ii) comprises filling the pixels of the demosaicked image x into the upsampled image y according to the following equation:

y<sub>(λ

(r−

1)+1,λ

(s−

1)+1)</sub>=x_(r,s)

where;

x is the demosaicked image with pixels x_(r,s)=[x_(r,s)1,x_(r,s)2,x_(r,s)3] with x_(r,s)k indicating an R (k=1), G (k=2), or B (k=3) component,

r=1,2, . . . , K₁ and s=1,2, . . . , K₂ denote, respectively, a row and a column of the demosaicked image x with K₁ rows and K₂ columns,

λ

is an integer upsampling factor, with λ

>

1, and

y is the upsampled image with λ

K₁×

λ

K₂ pixels y_(m,n)=[y_(m,n)1,y_(m,n)2,y_(m,n)3] with y_(m,n)k indicating an R (k=1) , G (k=2), or B (k=3) component, for m=1,2, . . . , λ

K₁ and n=1,2, . . . , λ

K₂.

3. The method as recited in claim 2, wherein the edge-orientation map d includes pixels d_(r,s) and is a two-valued edge-orientation map with d_(r,s)=1 indicating a dominant vertical direction and d_(r,s)=3 indicating a dominant horizontal direction.

4. The method as recited in claim 3, wherein the acts iv) and v) are performed according to the following equations:

5. The method as recited in claim 2, wherein the edge-orientation map b includes pixels b_(r,s) and is a two-valued edge-orientation map with b_(r,s)=1 indicating a dominant 135°

direction and b_(r,s)=3 indicating a dominant 45°

direction.

6. The method as recited in claim 5, wherein the acts iv) and v) are performed according to the following equations:

7. A method for automatic downsampling of a demosaicked image, the method comprising the following acts:

i) receiving a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) selecting a block of demosaicked pixels from the demosaicked image and a corresponding block of edge-orientation values from the edge-orientation map based on the location of each pixel under consideration in a downsampled image and a downsampling factor;

iii) determining the interpolation direction in which downsampling of the demosaicked image should be performed using the value of the selected block of edge-orientation values;

iv) setting weights associated with the demosaicked pixels located inside the block of demosaicked pixels according to the interpolation direction determined in the act iii); and

v) estimating pixels in the downsampled image by performing interpolation along the interpolation direction determined in the act iii) using demosaicked pixels located inside the block of demosaicked pixels and the weights associated with the demosaicked pixels located inside the block of demosaicked pixels.

8. The method as recited in claim 7, wherein the act v) comprises estimating pixels in the downsampled image z by performing interpolation along the interpolation direction determined in the act iii) using demosaicked pixels located inside the θ

×

θ

block of demosaicked pixels x_(i,j) defined by (p−

1)θ

<

i≦

pθ and

(q−

1)θ

<

j≦

qθ and

the weights w_(i,j) associated with the demosaicked pixels located inside the θ

×

θ

block of demosaicked pixels according to the following equation;

9. The method as recited in claim 8, wherein the edge-orientation map d includes pixels d_(r,s) and is a two-valued edge-orientation map with d_(r,s)=1 indicating a dominant vertical direction and d_(r,s)=3 indicating a dominant horizontal direction.

10. The method as recited in claim 9, wherein the downsampling factor θ

=2 and the weights w_(i,j) associated with the demosaicked pixels located inside the 2×

2 block of demosaicked pixels x_(i,j) are determined according to the following equations;

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w<sub>(2p,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2(p−

1)+1,2q)</sub>w<sub>(2p,2(q−

1)+1)</sub> for D>

8

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w<sub>(2p,2(q−

1)+1)</sub>;

w<sub>(2(p−

1)+1,2q)</sub>=w_(2p,2q);

w<sub>(2p,2(q−

1)+1)</sub>>

w<sub>(2(p−

1)+1,2q)</sub> for D≦

8

where;

D=d<sub>(2(p−

1)+1,2(q−

1)+1)</sub>+d<sub>(2(p−

1)+1,2q)+d(2p,2(q−

1)+1)</sub>+d_(2p,2q) is an aggregated edge-orientation value,

D>

8 indicates that interpolation should be performed in the horizontal direction; and

D≦

8 indicates that interpolation should be performed in the vertical direction.

11. The method as recited in claim 8, wherein the edge-orientation map b includes pixels b_(r,s) and is a two-valued edge-orientation map with b_(r,s)=1 indicating a dominant 135°

direction and b_(r,s)=3 indicating a dominant 45°

direction.

12. The method as recited in claim 11, wherein the downsampling factor θ

=2 and the weights w_(i,j) associated with the demosaicked pixels located inside the 2×

2 block of demosaicked pixels x_(i,j) are determined according to the following equations;

w<sub>(2p,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2(p−

1)+1,2q)</sub>>

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub> for B>

8

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2p,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w_(2p,2q)>

w<sub>(2p,2(q−

1)+1)</sub> for B≦

8

where;

B=b<sub>(2(p−

1)+1,2(q−

1)+1)</sub>+b<sub>(2(p−

1)+1,2q)</sub>+b<sub>(2p,2(q−

1)+1)</sub>+b_(2p,2q) is an aggregated edge-orientation value,

B>

8 indicates that interpolation should be performed in a dominant 135°

diagonal direction; and

B≦

8 indicates that interpolation should be performed in a dominant 45°

diagonal direction.

13. One or more computer-readable media having computer-readable instructions thereon which, when executed by a processor, implement a method for automatic upsampling of a demosaicked image, the method comprising the acts of:

i) receiving a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) filling pixels of the demosaicked image into an upsampled image;

iii) filling edge-orientation values of pixels of the edge-orientation map into an upsampled edge-orientation map;

iv) determining an interpolation direction for each pixel in which upsampling of the demosaicked image should be performed using the upsampled edge-orientation map; and

v) estimating missing pixels in the upsampled image by performing interpolation along the interpolation direction determined at act iv) using available pixels surrounding each missing pixel location.

14. The one or more computer-readable media as recited in claim 13, wherein the act ii) comprises filling the pixels of the demosaicked image x into the upsampled image y according to the following equation:

y<sub>(λ

(r−

1)+1,λ

(s−

1)+1)</sub>=x_(r,s)

where;

x is the demosaicked image with pixels x_(r,s)=[x_(r,s)1,x_(r,s)2,x_(r,s)3] with x_(r,s)k indicating an R (k=1), G (k=2), or B (k=3) component,

r=1,2, . . . , K₁ and s=1,2, . . . , K₂ denote, respectively, a row and a column of the demosaicked image x with K₁ rows and K₂ columns,

λ

is an integer upsampling factor, with λ

>

1, and

y is the upsampled image with λ

K₁×

λ

K₂ pixels y_(m,n)=[y_(m,n)1,y_(m,n)2,y_(m,n)3] with y_(m,n)k indicating an R (k=1) , G (k=2), or B (k=3) component, for m=1,2, . . . , λ

K₁ and n=1,2, . . . , λ

K₂.

15. The one or more computer-readable media as recited in claim 14, wherein:

the edge-orientation map d includes pixels d_(r,s) and is a two-valued edge-orientation map with d_(r,s)=1 indicating a dominant vertical direction and

d_(r,s)=3 indicating a dominant horizontal direction; and

the acts iv) and v) are performed according to the following equations;

16. The one or more computer-readable media as recited in claim 14, wherein:

the edge-orientation map b includes pixels b_(r,s) and is a two-valued edge-orientation map with b_(r,s)=1 indicating a dominant 135°

direction and b_(r,s)=3 indicating a dominant 45°

direction; and

the acts iv) and v) are performed according to the following equations;

17. One or more computer-readable media having computer-readable instructions thereon which, when executed by a processor, implement a method for automatic downsampling of a demosaicked image, the method comprising the acts of:

i) receiving a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) selecting a block of demosaicked pixels from the demosaicked image and a corresponding block of edge-orientation values from the edge-orientation map based on the location of each pixel under consideration in a downsampled image and a downsampling factor;

iii) determining the interpolation direction in which downsampling of the demosaicked image should be performed using the value of the selected block of edge-orientation values;

iv) setting weights associated with the demosaicked pixels located inside the block of demosaicked pixels according to the interpolation direction determined in the act iii); and

v) estimating pixels in the downsampled image by performing interpolation along the interpolation direction determined in the act iii) using demosaicked pixels located inside the block of demosaicked pixels and the weights associated with the demosaicked pixels located inside the block of demosaicked pixels.

18. The one or more computer-readable media as recited in claim 17, wherein the act v) comprises estimating pixels in the downsampled image z by performing interpolation along the interpolation direction determined in the act iii) using demosaicked pixels located inside the θ

×

θ

block of demosaicked pixels x_(i,j) defined by (p−

1)θ

<

i≦

pθ and

(q−

1)θ

<

j≦

qθ and

the weights w_(i,j) associated with the demosaicked pixels located inside the θ

×

θ

block of demosaicked pixels according to the following equation;

19. The one or more computer-readable media as recited in claim 18, wherein:

the edge-orientation map d includes pixels d_(r,s) and is a two-valued edge-orientation map with d_(r,s)=1 indicating a dominant vertical direction and d_(r,s)=3 indicating a dominant horizontal direction; and

the downsampling factor θ

=2 and the weights w_(i,j) associated with the demosaicked pixels located inside the 2×

2 block of demosaicked pixels x_(i,j) are determined according to the following equations;

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w<sub>(2p,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2(p−

1)+1,2q)</sub>>

w<sub>(2p,2(q−

1)+1)</sub> for D>

8

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w<sub>(2p,2(q−

1)+1)</sub>;

w<sub>(2(p−

1)+1,2q)</sub>=w_(2p,2q);

w<sub>(2p,2(q−

1)+1)</sub>>

w<sub>(2(p−

1)+1,2q)</sub> for D≦

8

where;

D=d<sub>(2(p−

1)+1,2(q−

1)+1)</sub>+d<sub>(2(p−

1)+1,2q)</sub>+d<sub>(2p,2(q−

1)+1)</sub>+d_(2p,2q) is an aggregated edge-orientation value,

D>

8 indicates that interpolation should be performed in the horizontal direction; and

D≦

8 indicates that interpolation should be performed in the vertical direction.

20. The one or more computer-readable media as recited in claim 18, wherein:

the edge-orientation map b includes pixels b_(r,s) and is a two-valued edge-orientation map with b_(r,s)=1 indicating a dominant 135°

direction and b_(r,s)=3 indicating a dominant 45°

direction; and

the downsampling factor θ

=2 and the weights w_(i,j) associated with the demosaicked pixels located inside the 2×

2 block of demosaicked pixels x_(i,j) are determined according to the following equations;

w<sub>(2p,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2(p−

1)+1,2q)</sub>>

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub> for B>

8

w<sub>(2(p−

1)+1,2(q−

1)+1)</sub>=w_(2p,2q);

w<sub>(2p,2(q−

1)+1)</sub>=w<sub>(2(p−

1)+1,2q)</sub>;

w_(2p,2q)>

w<sub>(2p,2(q−

1)+1)</sub> for B≦

8

where;

B=b<sub>(2(p−

1)+1,2(q−

1)+1)</sub>+b<sub>(2(p−

1)+1,2q)</sub>+b<sub>(2p,2(q−

1)+1)</sub>+b_(2p,2q) is an aggregated edge-orientation value,

B>

8 indicates that interpolation should be performed in a dominant 135°

diagonal direction; and

B≦

8 indicates that interpolation should be performed in a dominant 45°

diagonal direction.

21. An image processing apparatus comprising:

an electronic display; and

a processor in electronic communication with the electronic display; and

one or more computer-readable media in electronic communication with the processor, the one or more computer-readable media having computer-readable instructions thereon which, when executed by the processor, cause the processor to;

i) receive a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) fill pixels of the demosaicked image into an upsampled image;

iii) fill edge-orientation values of pixels of the edge-orientation map into an upsampled edge-orientation map;

iv) determine an interpolation direction for each pixel in which upsampling of the demosaicked image should be performed using the upsampled edge-orientation map;

v) estimate missing pixels in the upsampled image by performing interpolation along the interpolation direction determined at iv) using available pixels surrounding each missing pixel location; and

vi) send the upsampled image to the electronic display for presentation thereon.

22. The image processing apparatus as recited in claim 21, wherein:

the image processing apparatus comprises a photo viewer;

the one or more computer-readable media comprises one or more of a RAM, a ROM, and a flash EEPROM; and

the electronic visual display comprises a liquid crystal display.

23. An image processing apparatus comprising:

an electronic display; and

a processor in electronic communication with the electronic display; and

one or more computer-readable media in electronic communication with the processor, the one or more computer-readable media having computer-readable instructions thereon which, when executed by the processor, cause the processor to;

i) receive a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) select a block of demosaicked pixels from the demosaicked image and a corresponding block of edge-orientation values from the edge-orientation map based on the location of each pixel under consideration in a downsampled image and a downsampling factor;

iii) determine the interpolation direction in which downsampling of the demosaicked image should be performed using the value of the selected block of edge-orientation values;

iv) set weights associated with the demosaicked pixels located inside the block of demosaicked pixels according to the interpolation direction determined at iii);

v) estimate pixels in the downsampled image by performing interpolation along the interpolation direction determined at iii) using demosaicked pixels located inside the block of demosaicked pixels and the weights associated with the demosaicked pixels located inside the block of demosaicked pixels; and

vi) send the downsampled image to the electronic display for presentation thereon.

24. The image processing apparatus as recited in claim 23, wherein:

the image processing apparatus comprises a photo viewer;

the one or more computer-readable media comprises one or more of a RAM, a ROM, and a flash EEPROM; and

the electronic visual display comprises a liquid crystal display.

25. A method for automatically resizing a demosaicked image, the method comprising the following acts:

i) receiving a demosaicked image and an edge-orientation map that was created during the creation of the demosaicked image;

ii) resizing the edge-orientation map to create a resized edge-orientation map;

iii) determine the interpolation direction in which interpolation of the demosaicked image should be performed using the value of the selected block of edge-orientation values; and

iv) estimate pixels in a resized image by performing interpolation along the interpolation direction determined at iii).

26. The method as recited in claim 25, wherein the act iv) comprises the following acts:

iv.a) select a block of demosaicked pixels from the demosaicked image based on the location of each pixel under consideration in a resized image and a resizing factor;

iv.b) set weights associated with the demosaicked pixels located inside the block of demosaicked pixels according to the interpolation direction determined at iii); and

iv.c) estimate pixels in the resized image by performing interpolation along the interpolation direction determined at iii) using demosaicked pixels located inside the block of demosaicked pixels and the weights associated with the demosaicked pixels located inside the block of demosaicked pixels.