Image segmentation from focus varied images using graph cuts

• US 9,025,043 B2
• Filed: 09/24/2008
• Issued: 05/05/2015
• Est. Priority Date: 09/24/2008
• Status: Active Grant

• Current Assignee: Nikon Corporation
• Original Assignee: Nikon Corporation
• Inventors: Hong, Li, Takita, Mark, Novak, W. Thomas
• Primary Examiner(s): PRABHAKHER, PRITHAM DAVID
• Application Number:
  US13/119,541
• Publication Number:
  US 20110164152A1
• Time in Patent Office: 2,414 Days
• Field of Search: 348/222.1, 348/239, 348/153, 348/159, 348/211.11, 348/208.1, 382/164, 382/171, 382173-180, 382225-230
• US Class Current: 348/222.1
• CPC Class Codes: G03B 13/18 Focusing aids

Abstract

A system for providing an adjusted image of a scene includes an optical assembly, a capturing system coupled to the optical assembly, and a control system. The optical assembly is adjustable to alternatively be focused on a first focal area and a second focal area that is different than the first focal area. The capturing system captures a first frame of the scene when the optical assembly is focused at the first focal area, and a second frame of the scene when the optical assembly is focused at the second focal area. The first frame includes a plurality of first pixels and the second frame includes a plurality of second pixels. The control system analyzes the first frame and the second frame and utilizes graph cuts techniques to assign a depth label to at least a portion of the first frame.

Claims

1. A system for providing an adjusted image of a scene, the scene including a foreground object and a background object, the system comprising:

an optical assembly that is adjustable to alternatively be focused on a first focal area and a second focal area that is different than the first focal area;

a capturing system coupled to the optical assembly, the capturing system capturing a first frame of the scene when the optical assembly is focused at the first focal area, and subsequently capturing a second frame of the scene when the optical assembly is focused at the second focal area; and

a control system that analyzes at least one of the frames and divides the first frame into a plurality of first image segments, each first image segment being substantially homogeneous in color and including a plurality of pixels, the control system analyzes the first image segments using a segment based energy cost function and the first frame and the second frame, and the control system uses graph cut techniques to solve the energy cost function to determine that one of the first image segments corresponds to one of a portion of the foreground object and a portion of the background object.

2. The system of claim 1 wherein the control system processes the first image segments that are determined to correspond to a portion of the foreground object in a first fashion, and the control system processes the first image segments that are determined to correspond to a portion of the background object in a second fashion.

3. The system of claim 1 wherein the control system utilizes graph cut techniques to assign a depth label of either a background image segment or a foreground image segment to a plurality of the first image segments.

4. The system of claim 3 wherein the control system processes the first image segments that are labeled as foreground image segments in a first fashion, and processes the first image segments that are labeled as background image segments in a second fashion that is different than the first fashion.

5. The system of claim 4 wherein the control system adds artificial blurring to the first image segments labeled as background image segments.

6. The system of claim 1, wherein the energy cost function includes (i) data cost function that assesses a cost to assign a label of a foreground image segment or a background image segment to at least one of the first image segments, and (ii) a smoothness cost function that compares a label of at least one first image segment with a label of its neighbors.

7. The system of claim 6 wherein graph cuts techniques are used by the control system to solve the energy cost function in a way that approximates a minimum global cost for the energy cost function.

8. A method for providing an adjusted image of a scene, the scene including a foreground object and a background object, the method comprising the steps of:

alternatively focusing an optical assembly on a first focal area and a second focal area that is different than the first focal area;

capturing a first frame of the scene when the optical assembly is focused at the first focal area with a capturing system;

capturing a second frame of the scene after the first frame when the optical assembly is focused at the second focal area with the capturing system;

dividing the first frame into a plurality of first image segments with a control system that includes a processor, each first image segment being substantially homogeneous in color and including a plurality of pixels;

using a segment based energy cost function to analyze the first image segments with the control system using the first frame and the second frame; and

solving the segment based energy cost function using graph cut techniques with the control system to determine that one of the first image segments corresponds to one of a portion of the foreground object and a portion of the background object.

9. The method of claim 8 further comprising the step of assigning an initial depth label of either a background image segment or a foreground image segment to a plurality of the first image segments performed prior to the step of using the segment based energy cost function.

10. The method of claim 8 wherein the segment based energy cost function includes (i) a data cost function that assesses a cost to assign a label of a foreground image segment or a background image segment to at least one of the first image segments, and (ii) a smoothness cost function that compares a label of at least one first image segment with a label of its neighbors.

11. A method for identifying a foreground object and a background object in a scene, the method comprising the steps of:

providing a capturing system;

focusing light onto the capturing system with an optical assembly;

alternatively focusing the optical assembly on a first focal area and a second focal area that is different than the first focal area;

capturing a first frame of the scene when the optical assembly is focused at the first focal area with the capturing system;

capturing a second frame of the scene after the first frame when the optical assembly is focused at the second focal area with the capturing system;

dividing the first frame into a plurality of image segments with a control system that includes a processor based on color, each image segment being substantially homogeneous in color and including a plurality of adjacent pixels;

assigning an initial depth label to each of the plurality of image segments with the control system, the initial depth label being either a foreground image segment or a background image segment; and

using a segment based energy cost function on the image segments with the control system using the first frame and the second frame to determine if the initial depth label assigned to each of the plurality of the image segments is correct.

12. The method of claim 11 wherein the step of using the segment based energy cost function includes utilizing graph cut techniques to solve the segment based energy cost function to find a labeling of the image segments that minimizes an energy of the segment based energy cost function and determine if the initial depth label assigned to each of the image segments is correct.

13. The method of claim 11 wherein the step of using the segment based energy cost function includes utilizing graph cut techniques to solve the segment based energy cost function to find a labeling of the image segments in a way that approximates a minimum global cost to the segment based energy cost function.

14. The method of claim 11 wherein the segment based energy cost function includes (i) a data cost function that assesses a cost to assign the initial depth label of a foreground image segment or a background image segment to at least one of the image segments, and (ii) a smoothness cost function that compares an initial depth label of at least one image segment with a depth label of its neighbors.

15. The method of claim 11 further comprising a step of performing image registration to align the first frame with the second frame using the control system.

16. The method of claim 11 wherein the step of assigning the initial depth label is an arbitrary assignment of the initial depth label to each of the image segments.

17. The method of claim 11 wherein the step of assigning the initial depth label includes the step of computing a blur degree around each of the image segments with the control system, comparing the blur degree to a predetermined threshold, and for each image segment assigning the initial depth label of the background image segment if the computed blur degree is higher than the predetermined threshold, and assigning the initial depth label of the foreground image segment if the computed blur degree is lower than the predetermined threshold.

18. The method of claim 11 wherein the step of assigning the initial depth label includes (i) a step of computing a blur degree of each image segment using the control system for the first frame and the second frame, and (ii) for each image segment, comparing the blur degree of the first frame to the blur degree of the second frame to assign the initial depth label.

19. A method for identifying a foreground object and a background object in a scene, the method comprising the steps of:

providing a first frame and a second frame of the scene, the first frame being different from the second frame;

dividing the first frame into a plurality of image segments based on color with a control system that includes a processor so that each image segment is substantially homogeneous in color and includes a plurality of adjacent pixels;

assigning an initial depth label to each of the image segments with the control system, the initial depth label being either a foreground image segment or a background image segment; and

using a segment based energy cost function on the image segments with the control system using the first frame and the second frame to determine if the initial depth label assigned to each of the image segments is correct.

20. The method of claim 19 wherein the step of using the segment based energy cost function includes utilizing graph cut techniques to solve the segment based energy cost function to find a labeling of the image segments that minimizes an energy of the segment based energy cost function and determine if the initial depth label assigned to each of the image segments is correct.

21. The method of claim 19 wherein the step of using the segment based energy cost function includes utilizing graph cut techniques to solve the segment based energy cost function to find a labeling of the image segments in a way that approximates a minimum global cost to the segment based energy cost function.

22. The method of claim 19 wherein the segment based energy cost function includes (i) a data cost function that assesses a cost to assign the initial depth label of a foreground image segment or a background image segment to at least one of the image segments, and (ii) a smoothness cost function that compares an initial depth label of at least one image segment with a depth label of its neighbors.

23. The method of claim 19 further comprising a step of performing image registration to align the first frame with the second frame using the control system.

24. The method of claim 19 wherein the step of assigning the initial depth label is an arbitrary assignment of the initial depth label to each of the image segments.

25. The method of claim 19 wherein the step of assigning the initial depth label includes the step of computing a blur degree around each of the image segments with the control system, comparing the blur degree to a predetermined threshold, and for each image segment assigning the initial depth label of the background image segment if the computed blur degree is higher than the predetermined threshold, and assigning the initial depth label of the foreground image segment if the computed blur degree is lower than the predetermined threshold.

26. The method of claim 19 wherein the step of assigning the initial depth label includes (i) a step of computing a blur degree of each image segment using the control system for the first frame and the second frame, and (ii) for each image segment, comparing the blur degree of the first frame to the blur degree of the second frame to assign the initial depth label.