Foreground/background segmentation in digital images

• US 7,868,922 B2
• Filed: 08/21/2006
• Issued: 01/11/2011
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: Tessera Technologies Ireland Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Ciuc, Mihai, Zamfir, Adrian, Capata, Adrian, Corcoran, Peter
• Primary Examiner(s): Tran; Sinh
• Assistant Examiner(s): MONK, MARK T
• Application Number:
  US11/573,713
• Publication Number:
  US 20090273685A1
• Time in Patent Office: 1,604 Days
• Field of Search: 348/222.1
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20052 Discrete cosine transform [DCT]
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation

Abstract

An implementation efficient method of distinguishing between foreground and background regions of a digital image of a scene includes capturing two images of nominally the same scene and storing the captured images in DCT-coded format. The first image is taken with the foreground more in focus than the background and the second image is taken with the background more in focus than the foreground. Regions of the first image are assigned as foreground or background according to whether the sum of selected higher order DCT coefficients decreases or increases for the equivalent regions of the second image.

Claims

1. A method of distinguishing between foreground and background regions of a digital image of a scene, the method comprising:

capturing first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground;

storing the captured images in DCT-coded format;

assigning regions of the first image as foreground or background according to whether a difference between sums of selected higher order DCT coefficients for the first and second images is positive or negative; and

rendering in accordance with the assigning one or more processed images based on the first image or the second image, or both, at a digital rendering device, display or printer, or combinations thereof.

2. The method of claim 1, wherein regions are assigned as foreground when the difference is negative and as background when the difference is positive.

3. The method of claim 1, wherein the assigning further comprises:

(i) calculating the sum of selected high order coefficients for a plurality of DCT blocks of the first image, (ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image, (iii) comparing the sum calculated in step (i) for each DCT block of the first image with the sum calculated in step (ii) for the DCT block of the equivalent region of the second image, (iv) if the difference between the sums calculated for the first and second images is negative, assigning the given block as corresponding to a foreground region of the image, and (v) if the difference between the sums calculated for the first and second images is positive, assigning that block as corresponding to a background region of the image.

4. The method of claim 3, further comprising aligning said first and second images.

5. The method of claim 3, further comprising creating a digital map from said assigning said blocks as corresponding to background or foreground.

6. The method of claim 3, wherein the first and second images have different pixel resolutions.

7. The method of claim 6, further comprising making the image resolution of said first and second images equal.

8. The method of claim 6, wherein the first and second images are captured by a digital camera.

9. The method of claim 8, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

10. The method of claim 8, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

11. The method of claim 8, wherein said calculating and comparing is performed as a post processing stage in said digital camera.

12. The method of claim 3, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

13. The method of claim 3, wherein the selected high order coefficients are the top 25% of coefficients.

14. An image acquisition apparatus, comprising:

including an imaging optical system; and

a digital image detector;

a processor; and

a non-transitory, processor-readable storage medium having stored therein program instructions for programming the processor to perform a method of distinguishing between foreground and background regions of a digital image of a scene, wherein the program instructions stored in the storage medium are configured to program the apparatus to capture first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground; and

to store the captured images in DCT-coded format; and

to assign one or more regions of one or both of the first and second images as foreground or background according to whether a difference between sums of selected higher order DCT coefficients for the first and second images is positive or negative.

15. The apparatus of claim 14, wherein the program instructions stored in the storage medium are configured to program the apparatus to create a digital map from said assigning said blocks as corresponding to background or foreground.

16. The apparatus of claim 14, wherein the program instructions stored in the storage medium are configured to program the apparatus to make the image resolution of said first and second images equal.

17. The apparatus of claim 14, wherein the program instructions stored in the storage medium are configured to program the apparatus to capture first and second images of nominally the same scene and to store the captured images in DCT-coded format, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground, and to assign regions of the first image as foreground or background according to whether a difference between sums of selected higher order DCT coefficients for the first and second images is positive or negative.

18. The apparatus of claim 14, wherein regions are assigned as foreground when the difference is negative and as background when the difference is positive.

19. The apparatus of claim 14, wherein the processor-readable code configured to program the apparatus to assign is configured to program the apparatus to (i) calculate the sum of selected high order coefficients for a plurality of DCT blocks of the first image, and to (ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image, and to (iii) compare the sum calculated in (i) for each DCT block of the first image with the sum calculated in (ii) for the DCT block of the equivalent region of the second image, (iv) if the difference between the sums calculated for the first and second images is negative, then the processor-readable code is configured to program the apparatus to assign the given block as corresponding to a foreground region of the image, and (v) if the difference between the sums calculated for the first and second images is positive, then the processor-readable code is configured to program the apparatus to assign that block as corresponding to a background region of the image.

20. The apparatus of claim 19, wherein the program instructions stored in the storage medium are configured to program the apparatus to align said first and second images.

21. The apparatus of claim 19, wherein the first and second images have different pixel resolutions.

22. The apparatus of claim 19, wherein the first and second images are captured by a digital camera.

23. The apparatus of claim 22, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

24. The apparatus of claim 22, wherein said calculating and comparing is performed as a post processing stage in an external device to said digital camera.

25. The apparatus of claim 22, wherein said calculating and comparing is performed as a post processing stage in said digital camera.

26. The apparatus of claim 14, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

27. The apparatus of claim 14, wherein the selected high order coefficients are the top 25% of coefficients.

28. One or more non-transitory processor readable storage devices having processor readable code embodied thereon, said processor readable code for programming one or more processors to perform a method of distinguishing between foreground and background regions of a digital image of a scene, wherein the processor-readable code is configured to program the one or more processors to capture first and second images of nominally the same scene, the first image being taken with the foreground more in focus than the background and the second image being taken with the background more in focus than the foreground;

and to store the captured images in DCT-coded format; and

to assign regions of one or both of the first and second image as foreground or background according to whether a difference between one or more selected higher order DCT coefficients for equivalent regions of the first and second image is positive or negative.

29. The one or more storage devices of claim 28, wherein the processor-readable code is configured to program the one or more processors to create a digital map from said assigning said blocks as corresponding to background or foreground.

30. The one or more storage devices of claim 28, wherein the processor-readable code is configured to program the one or more processors to align the first and second images.

31. The one or more storage devices of claim 28, wherein regions are assigned as foreground when the difference is negative and as background when the difference is positive.

32. The one or more storage devices of claim 28, wherein the processor-readable code configured to program the one or more processors to assign is configured to program the one or more processors to (i) calculate the sum of selected high order coefficients for a plurality of DCT blocks of the first image, and to (ii) calculating the sum of the same high order coefficients for a plurality of DCT blocks of the second image, and to (iii) compare the sum calculated in (i) for each DCT block of the first image with the sum calculated in (ii) for the DCT block of the equivalent region of the second image, (iv) if the difference between the sums calculated for the first and second images is negative, assigning the given block as corresponding to a foreground region of the image, and (v) if the difference between the sums calculated for the first and second images is positive, assigning that block as corresponding to a background region of the image.

33. The one or more storage devices of claim 32, wherein the first and second images have different pixel resolutions.

34. The one or more storage devices of claim 33, wherein the processor-readable code is configure to program the one or more processors to make the image resolution of said first and second images equal.

35. The one or more storage devices of claim 32, wherein the first and second images are captured by a digital camera.

36. The one or more storage devices of claim 35, wherein the first image is a relatively high resolution image, and wherein the second image is a relatively low resolution pre- or post-view version of the first image.

37. The one or more storage devices of claim 35, wherein the processor-readable code is configured to program the one or more processors to calculate and compare as a post processing stage in an external device to said digital camera.

38. The one or more storage devices of claim 35, wherein the processor-readable code is configured to program the one or more processors to calculate and compare as a post processing stage in said digital camera.

39. The one or more storage devices of claim 32, wherein the first and second images have the same pixel resolution, and wherein the first and second images are captured by a digital camera and are relatively low resolution pre- and/or post-view versions of a higher resolution image of said scene also captured by the camera.

40. The one or more storage devices of claim 32, wherein the selected high order coefficients comprise the top 25% of coefficients.