Foreground/Background Segmentation in Digital Images

• US 20090273685A1
• Filed: 08/21/2006
• Published: 11/05/2009
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Ciuc, Mihai, Zamfir, Adrian, Capata, Adrian, Corcoran, Peter
• Granted Patent: US 7,868,922 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

An implementation efficient method of distinguishing between foreground and background regions of a digital image of a scene includes capturing two images of nominally the same scene and storing the captured images in DCT-coded format. The first image is taken with the foreground more in focus than the background and the second image is taken with the background more in focus than the foreground. Regions of the first image are assigned as foreground or background according to whether the sum of selected higher order DCT coefficients decreases or increases for the equivalent regions of the second image.

Claims

1. A method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

(a) capturing first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground;

(b) storing the captured images in DCT-coded format;

(c) assigning regions of the first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image; and

(d) rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

2. The method of claim 1, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

3. The method of claim 1, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

4. The method of claim 3, further comprising aligning said first and second images.

5. The method of claim 3, further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

6. The method of claim 3, wherein the first and second images have different pixel resolutions.

7. The method of claim 6, further comprising equating the image resolution of said first and second images.

8. The method of claim 6, wherein the first and second images are captured by a digital camera.

9. The method of claim 8, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

10. The method of claim 8, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

11. The method of claim 8, wherein said calculating and comparing is performed as a post processing stage in said digital camera.

12. The method of claim 3, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

13. The method of claim 3, wherein the selected high order coefficients are the top 25% of coefficients.

14. An image acquisition apparatus including an imaging optical system and digital image detector and storage medium, and having stored therein program instructions for programming one or more processors to perform a method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

(a) capturing first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground;

(b) storing the captured images in DCT-coded format; and

(c) assigning regions of the first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

15. The apparatus of claim 14, the method further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

16. The apparatus of claim 14, the method further comprising equating the image resolution of said first and second images.

17. The apparatus of claim 14, further comprising:

(i) means for capturing first and second images of nominally the same scene and storing the captured images in DCT-coded format, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground, and

(ii) means for assigning regions of the first image as foreground or background according to whether the sum of selected high order DCT coefficients decreases or increases for the equivalent regions of the second image.

18. The apparatus of claim 14, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

19. The apparatus of claim 14, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

20. The apparatus of claim 19, further comprising aligning said first and second images.

21. The apparatus of claim 19, wherein the first and second images have different pixel resolutions.

22. The apparatus of claim 19, wherein the first and second images are captured by a digital camera.

23. The apparatus of claim 22, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

24. The apparatus of claim 22, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

25. The apparatus of claim 22, wherein said calculating and comparing is performed as a post processing stage in said digital camera.

26. The apparatus of claim 14, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

27. The apparatus of claim 14, wherein the selected high order coefficients are the top 25% of coefficients.

28. One or more processor readable storage devices having processor readable code embodied thereon, said processor readable code for programming one or more processors to perform a method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

(a) capturing first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground;

(b) storing the captured images in DCT-coded format; and

(c) assigning regions of the first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

29. The one or more storage devices of claim 28, the method further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

30. The one or more storage devices of claim 28, the method further comprising aligning the first and second images.

31. The one or more storage devices of method of claim 28, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

32. The one or more storage devices of claim 28, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

33. The one or more storage devices of claim 32, wherein the first and second images have different pixel resolutions.

34. The one or more storage devices of claim 33, the method further comprising equating the image resolution of said first and second images.

35. The one or more storage devices of claim 32, wherein the first and second images are captured by a digital camera.

36. The one or more storage devices of claim 35, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

37. The one or more storage devices of claim 35, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

38. The one or more storage devices of claim 35, wherein said calculating and comparing is performed as a post processing stage in said digital camera.

39. The one or more storage devices of claim 32, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

40. The one or more storage devices of claim 32, wherein the selected high order coefficients are the top 25% of coefficients.

41. A method of determining an orientation of an image relative to a digital image acquisition device, comprising:

(a) capturing two images nominally of the same scene with said digital image acquisition device;

(b) comparing at least a portion of said two images adjacent the corresponding edges of said images to determine whether said portion comprises relatively more foreground than background; and

(c) responsive to said portion comprising more than a threshold degree of foreground, determining that said images are oriented with said portion at their bottom.

42. The method of claim 41, wherein said two images comprise a flash image and a non-flash image and in which said comparing comprises comparing luminance levels of pixels of said portion.

43. The method of claim 41, wherein said two images comprise non-flash images and in which said comparing comprises comparing higher order DCT coefficients for at least one block of said portion.

44. The method of claim 41, comprising implying an orientation of said digital image acquisition device in accordance with said image orientation.

45. The method of claim 41, wherein said comparing comprises comparing respective portions adjacent a plurality of edges of said two images, and wherein a portion which is determined to include a greatest degree of foreground relative to other portions is deemed to be located at a bottom of said images.

46. The method of claim 45, wherein the portion is deemed to be located at the bottom of said images based at least in part on its degree of foreground exceeding a degree of foreground for a portion adjacent an opposite edge by a given threshold.

47. The method of claim 46, wherein said threshold is varied according to exposure level of said images or whether said images are classified as being indoor or outdoor, or combinations thereof.

48. The method of claim 45, wherein a portion is deemed to be located at the bottom of said images when its degree of foreground exceeds a degree of foreground for a portion adjacent at least an adjacent edge, and a degree of foreground for a portion adjacent said adjacent edge exceeds a degree of foreground for a portion adjacent an opposite edge.

49. A digital image acquisition system having no photographic film comprising:

(a) means for capturing two images nominally of the same scene;

(b) means for comparing at least a portion of said two images adjacent the corresponding edges of said images to determine whether said portion comprises relatively more foreground than background; and

(b) means, responsive to said portion comprising more than a threshold degree of foreground, for determining that said images are oriented with said portion at their bottom.

50. One or more processor readable storage devices having processor readable code embodied thereon, said processor readable code for programming one or more processors to perform a method of determining an orientation of an image relative to a digital image acquisition device, the method comprising:

(a) capturing two images nominally of the same scene with said digital image acquisition device;

(b) comparing at least a portion of said two images adjacent the corresponding edges of said images to determine whether said portion comprises relatively more foreground than background; and

(c) responsive to said portion comprising more than a threshold degree of foreground, determining that said images are oriented with said portion at their bottom.

51. The one or more storage devices of claim 50, wherein said two images comprise a flash image and a non-flash image and in which said comparing comprises comparing luminance levels of pixels of said portion.

52. The one or more storage devices of claim 50, wherein said two images comprise non-flash images and in which said comparing comprises comparing higher order DCT coefficients for at least one block of said portion.

53. The one or more storage devices of claim 50, comprising implying an orientation of said digital image acquisition device in accordance with said image orientation.

54. The one or more storage devices of claim 50, wherein said comparing comprises comparing respective portions adjacent a plurality of edges of said two images, and wherein a portion which is determined to include a greatest degree of foreground relative to other portions is deemed to be located at a bottom of said images.

55. The one or more storage devices of claim 54, wherein the portion is deemed to be located at the bottom of said images based at least in part on its degree of foreground exceeding a degree of foreground for a portion adjacent an opposite edge by a given threshold.

56. The one or more storage devices of claim 55, wherein said threshold is varied according to exposure level of said images or whether said images are classified as being indoor or outdoor, or combinations thereof.

57. The one or more storage devices of claim 54, wherein a portion is deemed to be located at the bottom of said images when its degree of foreground exceeds a degree of foreground for a portion adjacent at least an adjacent edge, and a degree of foreground for a portion adjacent said adjacent edge exceeds a degree of foreground for a portion adjacent an opposite edge.