Synthesizing views based on image domain warping

• US 9,445,072 B2
• Filed: 08/31/2012
• Issued: 09/13/2016
• Est. Priority Date: 11/11/2009
• Status: Active Grant

• Current Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: ETH Zurich, Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Farr, Miquel , Stefanoski, Nikolce, Smolic, Aljoscha, Lang, Manuel, Hornung, Alexander, Fricke, Pedro Christian Espinosa, Wang, Oliver
• Primary Examiner(s): Wong, Allen C
• Application Number:
  US13/601,363
• Publication Number:
  US 20130057644A1
• Time in Patent Office: 1,474 Days
• Field of Search: 348/43, 382/103, 382/154, 382/162
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence

Abstract

Techniques are disclosed for generating autostereoscopic video content. A multiscopic video frame is received that includes a first image and a second image. The first and second images are analyzed to determine a set of image characteristics. A mapping function is determined based on the set of image characteristics. At least a third image is generated based on the mapping function and added to the multiscopic video frame.

Claims

1. A computer-implemented method of multiscopic video augmentation based on non-linear warp functions, the computer-implemented method comprising:

receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;

analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;

determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and

generating, by operation of one or more computer processors, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

2. The computer-implemented method of claim 1, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

3. The computer-implemented method of claim 2, wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

4. The computer-implemented method of claim 3, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

5. The computer-implemented method of claim 4, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

6. A non-transitory computer-readable medium containing a program which, when executed, performs an operation for multiscopic video augmentation based on non-linear warp functions, the operation comprising:

receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;

analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;

determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and

generating, by operation of one or more computer processors when executing the program, at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

7. The non-transitory computer-readable medium of claim 6, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

8. The non-transitory computer-readable medium of claim 7, wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

9. The non-transitory computer-readable medium of claim 8, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

10. The non-transitory computer-readable medium of claim 9, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

11. A system of multiscopic video augmentation based on non-linear warp functions, the system comprising:

one or more computer processors;

a memory containing a program which, when executed by the one or more computer processors, performs an operation comprising;

receiving a multiscopic video frame comprising at least a first image and a second image, at least one image of which has a first set of pixels and a second set of pixels;

analyzing the first image and the second image of the multiscopic video frame in order to determine a set of image characteristics including a set of sparse disparities between the first and second images;

determining, based on the set of image characteristics, a mapping function comprising a non-linear warp function configured to, in an image domain, warp the first set of pixels to a greater extent than the second set of pixels; and

generating at least a third image based on the determined mapping function and not based on any dense source stereoscopic information pertaining to the multiscopic video frame, wherein the multiscopic video frame is augmented to include the generated third image.

12. The system of claim 11, wherein the set of image characteristics further includes a saliency map of at least one of the first image and the second image.

13. The system of claim 12, wherein the saliency map is determined by analyzing a predefined set of image attributes based on a quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation.

14. The system of claim 13, wherein the set of image characteristics further includes an edge map of at least one of the first image and the second image, wherein the non-linear warp function is determined by optimizing an energy equation based on the set of image characteristics and a set of associated weights.

15. The system of claim 14, wherein the multiscopic video frame comprises a stereoscopic video frame, wherein the third image is generated further based on at least one of interpolation and extrapolation;

wherein the set of sparse disparities between the first image and second image are determined based on feature matching and optical flow analysis;

wherein the saliency map is determined by analyzing a predefined set of image attributes based on quaternion transform, wherein the predefined set of image attributes include contrast, orientation, color, and intensity, wherein the quaternion transform comprises a two-dimensional Fourier transformation;

wherein the edge map is determined based on edge detection and vertical line detection, wherein the edge detection is based on a Canny algorithm, wherein the vertical line detection is based on a Hough transform;

wherein the determined mapping function is configured to, when generating the third image, preserve vertical edges and salient regions to a greater extent than other image aspects, wherein each of the first image, the second image, and third image is multiscopically distinct.

16. The system of claim 15, wherein the multiscopic video frame is augmented by an image processing application, wherein the image processing application is configured to, in respective instances, determine each individual image characteristic selected from:

(i) image-based saliency of the at least one image; and

(ii) disparity-based saliency of the at least one image.

17. The system of claim 16, wherein the multiscopic video frame is generated from at least one monoscopic video frame, wherein the third image is generated based on at least one predefined aspect of disparity, wherein the image processing application is configured to, in respective instances, generate the third image based on each individual aspect of disparity selected from:

(i) a disparity range of the at least one image;

(ii) a disparity sensitivity of the at least one image;

(iii) a disparity gradient of the at least one image; and

(iv) a disparity velocity of the at least one image.

18. The system of claim 17, wherein the disparity range defines a stereoscopic comfort zone for viewing the multiscopic video frame;

wherein the disparity sensitivity specifies a level of ease with which different depths are distinguished in the multiscopic video frame;

wherein the disparity gradient is determined from a disparity of a feature depicted in the multiscopic video frame and an angular separation of the feature;

wherein the disparity velocity specifies an extent to which a disparity in the multiscopic video frame changes over time in one or more subsequent multiscopic video frames.

19. The system of claim 18, wherein at least one predefined aspect of disparity is associated with a corresponding predefined disparity mapping operator that is implemented using the mapping function, wherein the third image is generated using at least one predefined disparity mapping operator, wherein the image processing application is configured to, in respective instances, generate the third image using each individual disparity mapping operator selected from:

(i) a linear operator configured to perform global linear adaptation of a disparity in the multiscopic video frame to a target range;

(ii) a non-linear operator configured to perform non-linear disparity compression in the multiscopic video frame;

(iii) a gradient domain operator configured to remap disparity gradients of the multiscopic video frame; and

(iv) a temporal operator configured to perform temporal smoothing at a scene transition of the multiscopic video frame.

20. The system of claim 19, wherein the non-linear disparity compression is performed via a non-linear function on a sum of:

(i) a product of;

(A) a disparity value of the multiscopic video frame and (B) a predefined scaling factor; and

(ii) a predefined constant;

wherein the non-linear function comprises a logarithm;

wherein a plurality of non-linear functions, including the non-linear function, is used to perform the non-linear disparity compression, each of the plurality of non-linear functions configured to map a respective set of disparity values of the multiscopic video frame and falling within a respective source disparity range, to a respective target disparity range;

wherein each of the plurality of non-linear functions is distinct, wherein each set of disparity values of the multiscopic video frame is distinct, wherein each corresponding target disparity range is distinct;

wherein the non-linear operator comprises a predefined function including an integral, over a specified interval, over a set of saliency values corresponding to first derivates of the plurality of non-linear functions, wherein the image processing application is configured to, in respective instances, perform each individual operation selected from;

(i) generating the set of saliency values based on user input; and

(ii) generating the set of saliency values based on the multiscopic video frame and without requiring any user input.