Depth information for auto focus using two pictures and two-dimensional Gaussian scale space theory

• US 7,929,801 B2
• Filed: 08/15/2005
• Issued: 04/19/2011
• Est. Priority Date: 08/15/2005
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Nakamura, Makibi, Wong, Earl Quong
• Primary Examiner(s): Le; Vu
• Assistant Examiner(s): Park; Soo Jin
• Application Number:
  US11/204,942
• Publication Number:
  US 20070036427A1
• Time in Patent Office: 2,073 Days
• Field of Search: 382/154, 382/285, 382/246, 382/265
• US Class Current: 382/285
• CPC Class Codes: G02B 27/0075 with means for altering, e.g. increasing, the depth of field or depth of focus
• CPC Class Codes: G06T 7/571 from focus

Abstract

An imaging acquisition system that generates a depth map from two pictures of a three dimensional spatial scene is described. According to one aspect of the invention, the system generates the depth map based on the relative blur between the two pictures and the absolute blur contributed by the system. According to another aspect of the invention, the system calculates the depth map directly from the relative blur between the two pictures.

Claims

1. A computerized method comprising:

generating two two-dimensional scale space representations from two different pictures of a three dimensional spatial scene, wherein the three dimensional spatial scene comprises a plurality of objects at different distances, each of the two-dimensional scale space representations comprises a plurality of pictures, and each of the plurality of pictures included in the respective scale space representation has a resolution that is the same as a resolution of a different picture corresponding to one of the two-dimensional scale space representation;

computing, with an imaging system, a relative blur between the two different pictures using the two two-dimensional scale space representations, wherein the computing the relative blur further includes matching, based on blur comparisons, each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map and the picture depth map is subsequently used to determine values for the different distances of the plurality of objects.

2. The computerized method of claim 1, wherein computing the relative blur between the two pictures comprises:

determining a relative blur value between closest corresponding block and the block from the first picture.

3. The computerized method of claim 2, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

4. The computerized method of claim 1, wherein the each block from the first picture includes a characteristic that is one of being blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

5. The computerized method of claim 1, further comprising:

applying a clustering algorithm to the depth map.

6. The computerized method of claim 1, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

7. The computerized method of claim 6, wherein computing the picture depth map comprises:

generating the two two-dimensional scale space representations from each of the two pictures;

generating the blur map from the two pictures and the two two-dimensional scale space representations; and

calculating, from a blur map entry, the picture depth map entry using the equation

8. The computerized method of claim 1, wherein generating the blur map comprises:

computing a blur value from the relative blur value and an absolute blur; and

copying a blur value associated with a closest corresponding block into a corresponding entry of a relative blur difference map.

9. The computerized method of claim 1, wherein the two pictures are captured with an identical blur directionality.

10. The computerized method of claim 1, wherein the computing the picture depth map uses the equation

11. The computerized method of claim 1, wherein the two pictures are captured with opposite blur directionality.

12. A non-transitory computer readable storage medium having executable instructions to cause a processor to perform a method comprising:

generating two two-dimensional scale space representations from two different pictures of a three dimensional spatial scene, wherein the three dimensional spatial scene comprises a plurality of objects at different distances, each of the two-dimensional scale space representations comprises a plurality of pictures, and each of the plurality of pictures included in the respective scale space representation has a resolution that is the same as a resolution of a different picture corresponding to one of the two-dimensional scale space representation;

computing a relative blur between the two different pictures using the two two-dimensional scale space representations, wherein the computing the relative blur further includes matching, based on blur comparison, each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map and the picture depth map is subsequently used to determine values for the different distances of the plurality of objects.

13. The machine readable storage medium of claim 12, wherein computing the relative blur between the two pictures comprises:

determining a relative blur value between closest corresponding block and the block from the first picture.

14. The machine readable storage medium of claim 13, wherein the block from the first picture includes a characteristic that is one of being blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

15. The machine readable storage medium of claim 13, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

16. The machine readable medium of claim 12, further comprising:

applying a clustering algorithm to the depth map.

17. The machine readable storage medium of claim 12, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

18. The machine readable storage medium of claim 17, wherein computing the picture depth map comprises:

generating the two two-dimensional scale space representations from each of the two pictures;

generating the blur map from the two pictures and the two two-dimensional scale space representations; and

calculating, from a blur map entry, the picture depth map entry using the equation

19. The machine readable storage medium of claim 18, wherein generating the relative blur difference map comprises:

computing a blur value from the relative blur value and an absolute blur; and

copying a blur value associated with a closest corresponding block into a corresponding entry of a relative blur difference map.

20. The machine readable storage medium of claim 12, wherein the two pictures are captured with an identical blur directionality.

21. The machine readable storage medium of claim 12, wherein the computing the picture depth map uses the equation

22. The machine readable storage medium of claim 12, wherein the two pictures are captured with opposite blur directionality.

23. An apparatus comprising:

means for generating two two-dimensional scale space representations from two different pictures of a three dimensional spatial scene, wherein the three dimensional spatial scene comprises a plurality of objects at different distances, each of the two-dimensional scale space representations comprises a plurality of pictures, and each of the plurality of pictures included in the respective scale space representation has a resolution that is the same as a resolution of a different picture corresponding to one of the two-dimensional scale space representation;

means for computing a relative blur between the two different pictures using the two two-dimensional scale space representations, wherein the means for computing the relative blur further includes means for matching, based on blur comparisons, each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures; and

means for computing a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map and the picture depth map is subsequently used to determine values for the different distances of the plurality of objects.

24. The apparatus of claim 23, wherein means for computing the relative blur between the two pictures comprises:

means for determining a relative blur value between closest corresponding block and the block from the first picture.

25. The apparatus of claim 23, further comprising:

means for applying a clustering algorithm to the depth map.

26. The apparatus of claim 23, wherein the means for computing the picture depth map further comprises:

means for computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

27. The apparatus of claim 26, wherein means for computing the picture depth map comprises:

means for generating the two two-dimensional scale space representations from each of the two pictures;

means for generating the blur map from the two pictures and the two-dimensional scale space representations; and

means for calculating, from a blur map entry, the picture depth map entry using the equation

28. The apparatus of claim 23, wherein the means for generating the blur map comprises:

means for computing a blur value from the relative blur value and an absolute blur; and

means for copying a blur value associated with a closest corresponding block into a corresponding entry of a relative blur difference map.

29. The apparatus of claim 23, wherein the computing the picture depth map uses the equation

30. A system comprising:

a processor;

a memory coupled to the processor though a bus; and

a process executed from the memory by the processor to cause the processor to generate two two-dimensional scale space representations from two different pictures of a three dimensional spatial scene, wherein the three dimensional spatial scene comprises a plurality of objects at different distances, each of the two-dimensional scale space representations comprises a plurality of pictures, and each of the plurality of pictures included in the respective scale space representation has a resolution that is the same as a resolution of a different picture corresponding to one of the two-dimensional scale space representation, compute a relative blur between the two different pictures using the two two-dimensional scale space representations, wherein the computing the relative blur further includes matching, based on blur comparisons, each block from a first picture of the two pictures to a closest corresponding block in the two-dimensional scale space representation corresponding to the second picture of the two pictures, and to compute a picture depth map based on the relative blur, wherein an entry in the picture depth map has a corresponding entry in a blur map and the picture depth map is subsequently used to determine values for the different distances of the plurality of objects.

31. The system of claim 30, wherein computing the relative blur between the two pictures comprises:

determining a relative blur value between closest corresponding block and the block from the first picture.

32. The system of claim 31, wherein the each block from the first picture includes a characteristic that is one of being blurrier than a corresponding block from the second picture and has a smaller picture block variance than a corresponding block from the second picture.

33. The system of claim 31, wherein generating a two-dimensional scale space representation comprises applying a family of parametric convolving gaussian kernels to each of the two pictures.

34. The system of claim 30, wherein the process further causes the processor to apply a clustering algorithm to the depth map.

35. The system of claim 30, wherein the computing the picture depth map further comprises:

computing the picture depth map based on the relative blur and an absolute blur contributed by an imaging acquisition unit.

36. The system of claim 35, wherein computing the picture depth map comprises:

generating the two two-dimensional scale space representations from each of the two pictures;

generating the blur map from the two pictures and the two two-dimensional scale space representation; and

calculating, from a blur map entry, the picture depth map entry using the equation

37. The system of claim 35, wherein generating the blur map comprises:

computing a blur value from the relative blur value and an absolute blur; and

copying a blur value associated with a closest corresponding block into a corresponding entry of a relative blur difference map.

38. The system of claim 30, wherein the two pictures are captured with an identical blur directionality.

39. The system of claim 30, wherein the computing the picture depth map uses the equation

40. The system of claim 30, wherein the two pictures are captured with opposite blur directionality.