FOREGROUND/BACKGROUND SEPARATION USING REFERENCE IMAGES

• US 20080316328A1
• Filed: 06/17/2008
• Published: 12/25/2008
• Est. Priority Date: 12/27/2005
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Ireland Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Ciuc, Mihai, Zamfir, Adrian, Capata, Adrian, Drimbarean, Alexandru, Bigioi, Petronel, Corcoran, Peter
• Granted Patent: US 8,593,542 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: G06V 10/993 Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects

Abstract

A technique involves distinguishing between foreground and background regions of a digital image of a scene. First and second images are captured of nominally a same scene. The first image is a relatively high resolution image taken with the foreground more in focus than the background, while the second image is a relatively low resolution reference image taken with the background more in focus than the foreground. Regions of the captured images are assigned as foreground or background. In accordance with the assigning, one or more processed images are rendered based on the first image or the second image, or both.

Claims

1. A method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

(a) capturing first and second images of nominally the same scene, the first image being a relatively high resolution image taken with the foreground more in focus than the background, and the second image being a relatively low resolution reference image taken with the background more in focus than the foreground;

(b) storing the captured images;

(c) assigning regions of the first image as foreground or background; and

(d) rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

2. The method of claim 1, wherein said storing comprising storing said images in DCT-coded format.

3. The method of claim 2, wherein said assigning comprising assigning regions of said first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

4. The method of claim 3, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

5. The method of claim 3, wherein the selected high order coefficients are the top 25% of coefficients.

6. The method of claim 3, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

7. The method of claim 6, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

8. The method of claim 6, wherein said calculating and comparing is performed as a post processing stage in a digital camera.

9. The method of claim 1, further comprising aligning said first and second images.

10. The method of claim 1, further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

11. The method of claim 1, further comprising equating the image resolution of said first and second images.

12. The method of claim 1, wherein said second image comprises a pre- or post-view version of the first image.

13. A method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

(a) capturing first and second images of nominally the same scene, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution reference image versions of a higher resolution image of said scene also captured by the camera, and the first image being taken with the foreground more in focus than the background, and the second image being taken with the background more in focus than the foreground;

(b) storing the captured images;

(c) assigning regions of the first image as foreground or background; and

(d) rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

14. The method of claim 13, wherein said storing comprising storing said images in DCT-coded format.

15. The method of claim 14, wherein said assigning comprising assigning regions of said first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

16. The method of claim 15, wherein the selected high order coefficients are the top 25% of coefficients.

17. The method of claim 14, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

18. The method of claim 14, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

19. The method of claim 18, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

20. The method of claim 18, wherein said calculating and comparing is performed as a post processing stage in a digital camera.

21. The method of claim 13, further comprising aligning said first and second images.

22. The method of claim 13, further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

23. The method of claim 13, further comprising equating the image resolution of said first and second images.

24. The method of claim 13, wherein said second image comprises a pre- or post-view version of the first image.

25. A method of determining an orientation of an image relative to a digital image acquisition device, comprising:

(a) capturing a main image and one or more relatively low resolution reference images of a same scene with said digital image acquisition device;

(b) comparing at least a portion of said main image and said one or more reference images adjacent the corresponding edges of said images to determine whether said portion comprises relatively more foreground than background; and

(c) responsive to said portion comprising more than a threshold degree of foreground, determining that said images are oriented with said portion at their bottom.

26. The method of claim 25, wherein said two images comprise a flash image and a non-flash image and in which said comparing comprises comparing luminance levels of pixels of said portion.

27. The method of claim 25, wherein said two images comprise non-flash images and in which said comparing comprises comparing higher order DCT coefficients for at least one block of said portion.

28. The method of claim 25, wherein said comparing comprises comparing respective portions adjacent a plurality of edges of said two images, and wherein a portion which is determined to include a greatest degree of foreground relative to other portions is deemed to be located at a bottom of said images.

29. The method of claim 28, wherein the portion is deemed to be located at the bottom of said images based at least in part on its degree of foreground exceeding a degree of foreground for a portion adjacent an opposite edge by a given threshold.

30. The method of claim 29, wherein said threshold is varied according to exposure level of said images or whether said images are classified as being indoor or outdoor, or combinations thereof.

31. The method of claim 28, wherein a portion is deemed to be located at the bottom of said images when its degree of foreground exceeds a degree of foreground for a portion adjacent at least an adjacent edge, and a degree of foreground for a portion adjacent said adjacent edge exceeds a degree of foreground for a portion adjacent an opposite edge.

32. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, an aperture and a photodetector for capturing an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

(b) a processor;

(c) one or more processor-readable media having embedded therein programming code for the processor to perform a digital image processing method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising;

(i) capturing first and second images of nominally the same scene, the first image being a relatively high resolution main image taken with the foreground more in focus than the background, and the second image being a relatively low resolution reference image taken with the background more in focus than the foreground;

(ii) storing the captured images;

(iii) assigning regions of the first image as foreground or background; and

(iv) rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

33. The image capture device of claim 32, wherein said storing comprising storing said images in DCT-coded format.

34. The image capture device of claim 33, wherein said assigning comprising assigning regions of said first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

35. The image capture device of claim 34, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

36. The image capture device of claim 34, wherein the selected high order coefficients are the top 25% of coefficients.

37. The image capture device of claim 34, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

38. The image capture device of claim 37, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

39. The image capture device of claim 38, wherein said calculating and comparing is performed as a post processing stage in a digital camera.

40. The image capture device of claim 32, wherein the method further comprises aligning said first and second images.

41. The image capture device of claim 32, wherein the method further comprises creating a digital map from said assigning said blocks as corresponding to background or foreground.

42. The image capture device of claim 32, wherein the method further comprises equating the image resolution of said first and second images.

43. The image capture device of claim 32, wherein said second image comprises a pre- or post-view version of the first image.

44. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, an aperture and a photodetector for capturing an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

(b) a processor;

(c) one or more processor-readable media having embedded therein programming code for the processor to perform a digital image processing method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising;

(i) capturing first and second images of nominally the same scene, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution reference image versions of a higher resolution image of said scene also captured by the camera, and the first image being taken with the foreground more in focus than the background, and the second image being taken with the background more in focus than the foreground;

(ii) storing the captured images;

(iii) assigning regions of the first image as foreground or background; and

(iv) rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

45. The image capture device of claim 44, wherein said storing comprising storing said images in DCT-coded format.

46. The image capture device of claim 45, wherein said assigning comprising assigning regions of said first image as foreground or background according to whether one or more sums of selected higher order DCT coefficients decreases or increases for equivalent regions of the second image.

47. The image capture device of claim 46, wherein the selected high order coefficients are the top 25% of coefficients.

48. The image capture device of claim 45, wherein regions are assigned as foreground when the one or more sums decreases and as background when the one or more sums increases.

49. The image capture device of claim 45, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image,

(ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image,

(iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image,

(iv) if the sum calculated for a given DCT block of the first image is greater than the sum calculated for the DCT block of the equivalent region of the second image, assigning the given block as corresponding to a foreground region of the image, and

(v) if the sum calculated for a given DCT block of the first image is less than the sum calculated for the DCT block of the equivalent region of the second image, assigning that block as corresponding to a background region of the image.

50. The image capture device of claim 49, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

51. The image capture device of claim 49, wherein said calculating and comparing is performed as a post processing stage in a digital camera.

52. The image capture device of claim 44, wherein the method further comprises aligning said first and second images.

53. The image capture device of claim 44, wherein the method further comprises creating a digital map from said assigning said blocks as corresponding to background or foreground.

54. The image capture device of claim 44, wherein the method further comprises equating the image resolution of said first and second images.

55. The image capture device of claim 44, wherein said second image comprises a pre- or post-view version of the first image.

56. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, an aperture and a photodetector for capturing an original main image and one or more reference images having a temporal or spatial overlap or proximity with the original image, or combinations thereof;

(b) a processor;

(c) one or more processor-readable media having embedded therein programming code for the processor to perform a digital image processing method of determining an orientation of an image relative to a digital image acquisition device, the method comprising;

(i) capturing a main image and one or more relatively low resolution reference images of a same scene with said digital image acquisition device;

(ii) comparing at least a portion of said main image and said one or more reference images adjacent the corresponding edges of said images to determine whether said portion comprises relatively more foreground than background; and

(iii) responsive to said portion comprising more than a threshold degree of foreground, determining that said images are oriented with said portion at their bottom.

57. The image capture device of claim 56, wherein said two images comprise a flash image and a non-flash image and in which said comparing comprises comparing luminance levels of pixels of said portion.

58. The image capture device of claim 56, wherein said two images comprise non-flash images and in which said comparing comprises comparing higher order DCT coefficients for at least one block of said portion.

59. The image capture device of claim 56, wherein said comparing comprises comparing respective portions adjacent a plurality of edges of said two images, and wherein a portion which is determined to include a greatest degree of foreground relative to other portions is deemed to be located at a bottom of said images.

60. The image capture device of claim 59, wherein the portion is deemed to be located at the bottom of said images based at least in part on its degree of foreground exceeding a degree of foreground for a portion adjacent an opposite edge by a given threshold.

61. The image capture device of claim 60, wherein said threshold is varied according to exposure level of said images or whether said images are classified as being indoor or outdoor, or combinations thereof.

62. The image capture device of claim 59, wherein a portion is deemed to be located at the bottom of said images when its degree of foreground exceeds a degree of foreground for a portion adjacent at least an adjacent edge, and a degree of foreground for a portion adjacent said adjacent edge exceeds a degree of foreground for a portion adjacent an opposite edge.