Real-time capturing and generating stereo images and videos with a monoscopic low power mobile device

• US 20080031327A1
• Filed: 08/01/2006
• Published: 02/07/2008
• Est. Priority Date: 08/01/2006
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Hsiang-Tsun, Wang, Haohong, Manjunath, Sharath
• Granted Patent: US 8,970,680 B2
• Time in Patent Office: Days
• US Class Current: 375/240.12
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 13/236 using varifocal lenses or mirrors
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/673 based on contrast or high frequency components of image signals, e.g. hill climbing method
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10

Abstract

A monoscopic low-power mobile device is capable of creating real-time stereo images and videos from a single captured view. The device uses statistics from an autofocusing process to create a block depth map of a single capture view. Artifacts in the block depth map are reduced and an image depth map is created. Stereo three-dimensional (3D) left and right views are created from the image depth map using a Z-buffer based 3D surface recover process and a disparity map which is a function of the geometry of binocular vision.

Claims

1. A monoscopic low-power mobile device comprising:

a single-sensor camera sensor module operable to capture an image and having an autofocusing sub-module operable to determine a best focus position by moving a lens through an entire focusing range via a focusing process and to select the focus position with a maximum focus value when capturing the image;

a depth map generator assembly operable in a first-stage to develop a block-level depth map automatically using statistics from the autofocusing sub-module and in a second-stage to develop an image depth map; and

an image pair generator module operable to create a missing second view from the captured image to create three dimensional (3D) stereo left and right views.

2. The device of claim 1, wherein the image pair generator module comprises:

a disparity map sub-module which calculates a disparity map based on a distance in pixels between image points in the left and right views of binocular vision geometry for the captured image wherein the captured image represents the left view;

a Z-buffer 3D surface recover sub-module operable to construct a 3D visible surface for the captured image from the right view; and

a stereo view generator sub-module operable to project the 3D surface of the right view onto a projection plane.

3. The device of claim 1, wherein the focusing process of the autofocusing sub-module in a still image mode performs an exhaustive search focusing process to capture a still-image, and in a video mode, to achieve real-time capturing of a video clip, is initiated with the exhaustive search focusing process and follows with a climbing-hill focusing process.

4. The device of claim 3, wherein the depth map generator assembly in the second stage is operable to reduce artifacts with a bilinear filter.

5. The device of claim 4, wherein in the second stage, the depth map generator assembly is operable to:

obtain a depth value of corner points (A, B, C, and D) of each block found during the focusing process, the depth value being an average value of neighboring blocks of a respective block where a depth of a middle point d is defined by

6. The device of claim 5, wherein the bilinear filter is further operable to

calculate the pixel depth value d_P for pixel P (x_P, y_P, d_P) in accordance with the equation

7. The device of claim 3, further comprising a video coding module for coding the video clip captured and providing statistics information for calculating the block-level depth map, the video coding module being operable to determine motion estimation, and the depth map generator assembly being operable in the second stage to detect and estimate depth information for real-time capturing and generation of stereo video using the statistics information from the motion estimation, the focusing process, and history data plus heuristic rules to obtain a final block depth map from which the image depth map is derived.

8. The device of claim 1, further comprising a display and a 3D effects generator module for displaying on the display the 3D stereo left and right views.

9. The device of claim 8, wherein the 3D effects generator module is operable to produce a red-blue anaglyph image of the 3D stereo left and right views on the display.

10. The device of claim 1, wherein the monoscopic low-power mobile device comprises one of a hand-held digital camera, a camcorder, and a single-sensor camera phone.

11. A monoscopic low-power mobile device comprising:

means for capturing an image with a single sensor;

means for autofocusing a lens and determining a best focus position by moving the lens through an entire focusing range and for selecting the focus position with a maximum focus value when capturing the image;

means for generating in a first-stage a block-level depth map automatically using statistics from the autofocusing means and in a second-stage an image depth map; and

means for creating a missing second view from the captured image to create three dimensional (3D) stereo left and right views.

12. The device of claim 11, wherein the creating means comprises:

means for calculating a disparity map based on a distance in pixels between image points in the left and right views of binocular vision geometry for the captured image wherein the captured image represents the left view;

means for 3D surface recovering with Z-buffering for constructing a 3D visible surface for the captured image from a missing right viewpoint; and

means for generating stereo views by projecting the constructed 3D surface onto a projection plane.

13. The device of claim 11, wherein the autofocusing means includes means for performing an exhaustive search focusing process to capture a still-image in a still image mode;

means for initiating the exhaustive search focusing process in a video mode; and

means for climbing-hill focusing in a video mode to capture a real-time video clip.

14. The device of claim 13, wherein the generating means in the second stage includes means for reducing artifacts.

15. The device of claim 14, wherein the means for reducing artifacts includes means for obtaining a depth value of corner points (A, B, C, and D) of each block found by the autofocusing means, the depth value being an average value of neighboring blocks of a respective block where a depth of a middle point d is defined by

16. The device of claim 15, wherein the bilinear filtering means comprises means for calculating the pixel depth value d_P for pixel P (x_P, y_P, d_P) in accordance with equation

17. The device of claim 13, further comprising means for video coding the video clip captured and providing statistics information;

wherein the means for video coding includes means for motion estimating; and

wherein the generating means includes means for detecting and estimating depth information for real-time capturing and generation of stereo video using statistics information from the motion estimating means, the autofocusing means, and history data plus some heuristic rules to obtain a final block depth map from which the image depth map is derived.

18. The device of claim 11, further comprising a display and means for generating 3D effects of the 3D stereo left and right views on the display.

19. The device of claim 18, wherein the 3D effects generating means produces a red-blue anaglyph image of the 3D stereo left and right views on the display.

20. The device of claim 11, wherein the monoscopic low-power mobile device comprises one of a hand-held digital camera, a camcorder, and a single-sensor camera phone.

21. A method for generating real-time stereo images comprising the steps of:

capturing an image with a single sensor;

autofocusing a lens and determining a best focus position by moving the lens through an entire focusing range and selecting the focus position with a maximum focus value when capturing the image;

generating in a first-stage a block-level depth map automatically using statistics from the autofocusing step and in a second-stage generating an image depth map; and

creating a missing second view from the captured image to create three-dimensional (3D) stereo left and right views.

22. The method of claim 21, wherein the creating step comprises the steps of:

calculating a disparity map based on a distance in pixels between image points in the left and right views of binocular vision geometry for the captured image wherein the captured image represents the left view;

3D surface recovering with Z-buffering for constructing a 3D visible surface for the captured image from a missing right viewpoint; and

generating a missing right view by projecting the constructed 3D surface onto a projection plane.

23. The method of claim 21, wherein the autofocusing step includes the steps of performing an exhaustive search focusing process to capture a still-image in a still image mode;

initiating the exhaustive search focusing process in a video mode; and

climbing-hill focusing in a video mode to capture a real-time video clip.

24. The method of claim 23, wherein the generating step in the second stage includes the step of reducing artifacts.

25. The method of claim 24, wherein the reducing artifacts step includes the steps of:

obtaining a depth value of corner points (A, B, C, and D) of each block found by the autofocusing means, the depth value being an average value of neighboring blocks of a respective block where a depth of a middle point d is defined by

26. The method of claim 25, wherein the bilinear filtering step comprises the step of:

calculating the pixel depth value d_P for pixel P (x_P, y_P, d_P) in accordance with equation

27. The method of claim 23, further comprising the step of video coding the video clip and motion estimating, wherein the generating step includes the step of detecting and estimating depth information for real-time capturing and generation of stereo video using statistics from the motion estimating step, the autofocusing step, and history data plus heuristic rules to obtain a final block depth map from which the image depth map is derived.

28. The method of claim 21, further comprising the step of generating 3D effects of the 3D stereo left and right views on a display.

29. The method of claim 28, wherein the step of generating 3D effects includes the step of producing a red-blue anaglyph image of the 3D stereo left and right views on the display.

30. A method for processing still images comprising the steps of:

autofocusing processing a captured still image and estimating depth information of remote objects in the image to detect a block-level depth map; and

approximating an image depth map from the block-level depth map.

31. The method of claim 30, wherein the autofocusing processing step includes the step of processing the image using a coarse-to-fine depth detection process.

32. The method claim 30, wherein the approximating step comprises the step of bilinear filtering the block-level depth map to derive an approximated image depth map.

33. The method of claim 32, wherein the autofocusing processing step comprises the step of determining a depth value of each focus block including corner points (A, B, C and D) when detecting the block-level depth map;

and wherein the bilinear filtering step obtains the depth value of the pixels inside the block.

34. The method of claim 33, wherein position values and the depth values for the corners points (A, B, C, and D) of the block are denoted as (x_A, y_A, d_A), (x_B, y_B, d_B), (x_C, y_C, d_C), (x_D, y_D, d_D) so that the depth value of all the pixels within the block are calculated, wherein for a respective pixel denoted by a point P (x_P, y_P, d_P), the depth value d_P of the respective pixel is defined as

35. A still image capturing device comprising:

an autofocusing module operable to process a captured still image and estimate depth information of remote objects in the image to detect a block-level depth map;

an image depth map module operable to approximate from the block-level depth map an image depth map using bilinear filtering; and

an image pair generator module operable to create a missing second view from the captured image to create three-dimensional (3D) stereo left and right views.

36. The device of claim 35, further comprising a 3D effects generator module operable to display 3D effects of the 3D stereo left and right views.

37. The device of claim 35, wherein a focusing process of the autofocusing module performs an exhaustive search focusing process to capture the still image.

38. The device of claim 35, wherein the image depth map module is operable to reduce artifacts with the bilinear filtering.

39. A video image capturing device comprising:

an autofocusing module operable to process a captured video clip and estimate depth information of remote objects in a scene;

a video coding module operable to code the video clip captured, provide statistics information and determine motion estimation; and

an image depth map module operable to detect and estimate depth information for real-time capturing and generation of stereo video using the statistics information from the motion estimation, the process of the autofocusing module, and history data plus heuristic rules to obtain a final block depth map from which an image depth map is derived.

40. The device of claim 39, wherein a focusing process of the autofocusing module to achieve real-time capturing of a video clip is initiated with the exhaustive search focusing process and follows with a climbing-hill focusing process.

41. The device of claim 39, further comprising an image pair generator module operable to create a missing second view from the captured image to create three-dimensional (3D) stereo left and right views.

42. The device of claim 41, further comprising a 3D effects generator module operable to display 3D effects of the 3D stereo left and right views.

43. The device of claim 39, wherein the depth map module is operable to predict an internal block depth map (P_n(i,j)) and a focus value map (T_n(i,j)) of a current frame n from those of a previous frame by the following equations

44. A program code having program instructions operable upon execution by a processor to:

bilinear filter an image to determine a depth value of each focus block including corner points (A, B, C and D) of a block-level depth map, and determine the depth value (d_P) of all pixels within the block according to the following equation