Depth information for auto focus using two pictures and two-dimensional gaussian scale space theory

• US 20070036427A1
• Filed: 08/15/2005
• Published: 02/15/2007
• Est. Priority Date: 08/15/2005
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Nakamura, Makibi, Wong, Earl
• Granted Patent: US 7,929,801 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 7/571 from focus

Abstract

An imaging acquisition system that generates a depth map from two pictures of a three dimensional spatial scene is described. According to one aspect of the invention, the system generates the depth map based on the relative blur between the two pictures and the absolute blur contributed by the system. According to another aspect of the invention, the system calculates the depth map directly from the relative blur between the two pictures.

Claims

1. A computerized method comprising:

computing a relative blur between two pictures of a three dimensional spatial scene; and

computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map.

2. The computerized method of claim 1, wherein computing the relative blur between the two pictures comprises:

generating a two-dimensional scale space representation from each of the two pictures;

matching each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

determining a relative blur value between closest corresponding block and the block from the first picture.

3. The computerized method of claim 2, wherein the block from the first picture is selected from the group consisting of blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

4. The computerized method of claim 2, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

5. The computerized method of claim 1, further comprising:

applying a clustering algorithm to the depth map.

6. The computerized method of claim 1, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

7. The computerized method of claim 6, wherein computing the picture depth map comprises:

generating a two-dimensional scale space representation from each of the two pictures;

generating the blur map from the two pictures and the set of two-dimensional scale space representation; and

calculating, from a blur map entry, the picture depth map entry using the equation

$d_o=\frac{\mathrm{fD}}{D-f-2<br><br>{\mathrm{rf}}_{\mathrm{number}}},$

where f is the camera lens focal length, D the distance between the image plane inside the camera and the lens, r is the blur radius of the image on the image plane and f_number is the f_number of the camera lens.

8. The computerized method of claim 1, wherein generating the blur map comprises:

computing a blur value from the relative blur value and the absolute blur; and

copying the blur value associated with the closest corresponding block into the corresponding entry of the relative blur difference map.

9. The computerized method of claim 1, wherein the two pictures are captured with an identical blur directionality.

10. The computerized method of claim 1, wherein the computing the picture depth map uses the equation

11. The computerized method of claim 1, wherein the two pictures are captured with the opposite blur directionality.

12. A machine readable medium having executable instructions to cause a processor to perform a method comprising:

computing a relative blur between two pictures of a three dimensional spatial scene; and

computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map.

13. The machine readable medium of claim 12, wherein computing the relative blur between the two pictures comprises:

generating a two-dimensional scale space representation from each of the two pictures;

matching each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

determining a relative blur value between closest corresponding block and the block from the first picture.

14. The machine readable medium of claim 13, wherein the block from the first picture is selected from the group consisting of blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

15. The machine readable medium of claim 13, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

16. The machine readable medium of claim 12, further comprising:

applying a clustering algorithm to the depth map.

17. The machine readable medium of claim 12, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

18. The machine readable medium of claim 17, wherein computing the picture depth map comprises:

generating a two-dimensional scale space representation from each of the two pictures;

generating the blur map from the two pictures and the set of two-dimensional scale space representation; and

calculating, from a blur map entry, the picture depth map entry using the equation

$d_o=\frac{\mathrm{fD}}{D-f-2<br><br>{\mathrm{rf}}_{\mathrm{number}}},$

where f is the camera lens focal length, D the distance between the image plane inside the camera and the lens, r is the blur radius of the image on the image plane and f_number is the f_number of the camera lens.

19. The machine readable medium of claim 18, wherein generating the relative blur difference map comprises:

computing a blur value from the relative blur value and the absolute blur; and

copying the blur value associated with the closest corresponding block into the corresponding entry of the relative blur difference map.

20. The machine readable medium of claim 12, wherein the two pictures are captured with an identical blur directionality.

21. The machine readable medium of claim 12, wherein the computing the picture depth map uses the equation

22. The machine readable medium of claim 12, wherein the two pictures are captured with the opposite blur directionality.

23. An apparatus comprising:

means for computing a relative blur between two pictures of a three dimensional spatial scene; and

means for computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map.

24. The apparatus of claim 23, wherein means for computing the relative blur between the two pictures comprises:

means for generating a two-dimensional scale space representation from each of the two pictures;

means for matching each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

means for determining a relative blur value between closest corresponding block and the block from the first picture.

25. The apparatus of claim 23, further comprising:

means for applying a clustering algorithm to the depth map.

26. The apparatus of claim 23, wherein the means for computing the picture depth map further comprises:

means for computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

27. The apparatus of claim 26, wherein means for computing the picture depth map comprises:

means for generating a two-dimensional scale space representation from each of the two pictures;

means for generating the blur map from the two pictures and the set of two-dimensional scale space representation; and

means for calculating, from a blur map entry, the picture depth map entry using the equation

$d_o=\frac{\mathrm{fD}}{D-f-2<br><br>{\mathrm{rf}}_{\mathrm{number}}},$

where f is the camera lens focal length, D the distance between the image plane inside the camera and the lens, r is the blur radius of the image on the image plane and f_number is the f_number of the camera lens.

28. The apparatus of claim 23, wherein the means for generating the blur map comprises:

means for computing a blur value from the relative blur value and the absolute blur; and

means for copying the blur value associated with the closest corresponding block into the corresponding entry of the relative blur difference map.

29. The apparatus of claim 23, wherein the computing the picture depth map uses the equation

30. A system comprising:

a processor;

a memory coupled to the processor though a bus; and

a process executed from the memory by the processor to cause the processor to compute a relative blur between two pictures of a three dimensional spatial scene, and to compute a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map.

31. The system of claim 30, wherein computing the relative blur between the two pictures comprises:

generating a two-dimensional scale space representation from each of the two pictures;

matching each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

determining a relative blur value between closest corresponding block and the block from the first picture.

32. The system of claim 31, wherein the block from the first picture is selected from the group consisting of blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

33. The system of claim 31, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

34. The system of claim 30, wherein the process further causes the processor to apply a clustering algorithm to the depth map.

35. The system of claim 30, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

36. The system of claim 35, wherein computing the picture depth map comprises:

generating a two-dimensional scale space representation from each of the two pictures;

generating the blur map from the two pictures and the set of two-dimensional scale space representation; and

calculating, from a blur map entry, the picture depth map entry using the equation

$d_o=\frac{\mathrm{fD}}{D-f-2<br><br>{\mathrm{rf}}_{\mathrm{number}}},$

where f is the camera lens focal length, D the distance between the image plane inside the camera and the lens, r is the blur radius of the image on the image plane and f_number is the f_number of the camera lens.

37. The system of claim 35, wherein generating the blur map comprises:

computing a blur value from the relative blur value and the absolute blur; and

copying the blur value associated with the closest corresponding block into the corresponding entry of the relative blur difference map.

38. The system of claim 30, wherein the two pictures are captured with an identical blur directionality.

39. The system of claim 30, wherein the computing the picture depth map uses the equation

40. The system of claim 30, wherein the two pictures are captured with the opposite blur directionality.