SYNTHESIZING VIEWS BASED ON IMAGE DOMAIN WARPING

• US 20130057644A1
• Filed: 08/31/2012
• Published: 03/07/2013
• Est. Priority Date: 11/11/2009
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Stefanoski, Nikolce, Lang, Manuel, Hornung, Alexander, Smolic, Aljoscha, Farr, Miquel ., Fricke, Pedro Christian Espinosa, Wang, Oliver
• Granted Patent: US 9,445,072 B2
• Time in Patent Office: Days
• US Class Current: 348/42
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence

Abstract

Techniques are disclosed for generating autostereoscopic video content. A multiscopic video frame is received that includes a first image and a second image. The first and second images are analyzed to determine a set of image characteristics. A mapping function is determined based on the set of image characteristics. At least a third image is generated based on the mapping function and added to the multiscopic video frame.

Claims

1. A computer-implemented method comprising:

receiving a multiscopic video frame comprising at least a first image and a second image;

analyzing the first image and the second image of the multiscopic video frame to determine a set of image characteristics;

determining a mapping function based on the set of image characteristics; and

generating, by operation of one or more computer processors, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

2. The computer-implemented method of claim 1, wherein the set of image characteristics includes a set of sparse disparities between the first image and the second image.

3. The computer-implemented method of claim 2, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

4. The computer-implemented method of claim 3, wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

5. The computer-implemented method of claim 4, wherein the mapping function comprises a non-linear warp function configured to operate in an image domain, based on the set of image characteristics.

6. The computer-implemented method of claim 5, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, and wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

7. The computer-implemented method of claim 6, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

8. A computer-readable storage medium containing a program which, when executed, performs an operation comprising:

receiving a multiscopic video frame comprising at least a first image and a second image;

analyzing the first image and the second image of the multiscopic video frame to determine a set of image characteristics;

determining a mapping function based on the set of image characteristics; and

generating, by operation of one or more computer processors when executing the program, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

9. The computer-readable storage medium of claim 8, wherein the set of image characteristics includes a set of sparse disparities between the first image and the second image.

10. The computer-readable storage medium of claim 9, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

11. The computer-readable storage medium of claim 10, wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

12. The computer-readable storage medium of claim 11, wherein the mapping function comprises a non-linear warp function configured to operate in an image domain, based on the set of image characteristics.

13. The computer-readable storage medium of claim 12, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, and wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

14. The computer-readable storage medium of claim 13, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

15. A system comprising:

one or more computer processors;

a memory containing a program which, when executed by the one or more computer processors, performs an operation comprising;

receiving a multiscopic video frame comprising at least a first image and a second image;

analyzing the first image and the second image of the multiscopic video frame to determine a set of image characteristics;

determining a mapping function based on the set of image characteristics; and

generating at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

16. The system of claim 15, wherein the set of image characteristics includes a set of sparse disparities between the first image and the second image.

17. The system of claim 16, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

18. The system of claim 17, wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

19. The system of claim 18, wherein the mapping function comprises a non-linear warp function configured to operate in an image domain, based on the set of image characteristics.

20. The system of claim 19, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.