Generation of a stereo video from a mono video

• US 9,137,519 B1
• Filed: 01/04/2012
• Issued: 09/15/2015
• Est. Priority Date: 01/04/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mukherjee, Debargha, Wu, Chen
• Primary Examiner(s): HALLENBECK-HUBER, JEREMIAH CHARLES
• Application Number:
  US13/343,245
• Time in Patent Office: 1,350 Days
• Field of Search: 348/44
• US Class Current: 1/1
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion

Abstract

A system and methodology provide for generation of a stereo video from a mono video. A mono video is partitioned into shots, where each shot including one or more frames of the mono video. The mono video frames are used as the left frames in the stereo video. Depth parameters are determined for each shot. A pixel depth map is created for each frame of each shot. The right frames for the stereo video are created by shifting pixels from the left frames laterally to occupy new locations in the right frames. A pixel'"'"'s shift is based on the depth parameters and the pixel depth. In aggregate, pixel shifts will cause objects to appear in different locations in the right frame relative to where they appeared in the left frame. The effect upon the viewer is that the stereo video will provide an enhanced illusion of depth for the viewer.

Claims

1. A method for generating stereo video from mono video comprising:

identifying a shot in the mono video, the shot comprising a plurality of mono frames;

identifying a scene type for the shot, the scene type identified from one of a plurality of different scene types, each scene type associated with a different set of depth parameters, wherein each set of depth parameters comprises K_near, a numerical value representing depth scaling of a closest object in one of the mono frames to a viewer as a function of a horizontal resolution of the mono video, and K_far, a numerical value representing depth scaling of a farthest object in one of the mono frames to the viewer as a function of the horizontal resolution of the mono video;

determining depth parameters for the shot based on the identified scene type;

selecting a first frame from the shot to use as either a left frame or a right frame in a stereo frame pair;

determining a pixel depth map for the first frame based on the scene type, the pixel depth map comprising a pixel depth for each pixel in the first frame; and

generating a second frame based on the first frame, the second frame to be used as either the right frame or the left frame so as to be another frame in the stereo frame pair, comprising;

determining a pixel shift amount for each pixel in the first frame, the pixel shift amount based on its pixel depth and the depth parameters for the shot; and

generating a pixel in the second frame by shifting a corresponding pixel in the first frame by its pixel shift amount.

2. The method of claim 1 wherein the pixels of the mono video frame comprise YCbCr color components.

3. The method of claim 1 wherein the pixel depth is based on a red difference component and a blue difference component of the pixel.

4. The method of claim 3 wherein the pixel depth is further based eigenvalues multiplied by the red and blue difference components, wherein the eigenvalues are based on the red and blue difference components of all of the pixels of the first frame.

5. The method of claim 1 wherein the pixel depth is based on a direction of greatest spread between red difference components and blue difference components of the pixels of the first frame.

6. The method of claim 1 comprising performing saturation correction on the pixel depth map.

7. The method of claim 5 wherein the saturation correction comprises adjusting the pixel depth of a pixel based upon a saturation value of the pixel, the saturation value based on the larger of a red difference component and a blue difference component for the pixel.

8. The method of claim 5 wherein the saturation correction comprises a nonlinear function.

9. The method of claim 1 comprising performing a gamma correction on the pixel depths.

10. The method of claim 9 wherein the gamma correction comprises a nonlinear function based on a luma component of each pixel.

11. The method of claim 1 wherein responsive to a first and second pixel in the first frame being pixel shifted to a same pixel in the second frame, determining the color of the pixel in the second frame from the pixel with the smallest or most negative pixel shift amount.

12. The method of claim 1 wherein responsive to a first and second pixel in the first frame being pixel shifted to a same pixel in the second frame by a same pixel shift amount, determining the color of the pixel in the second frame from the last of the two pixels to be shifted.

13. The method of claim 1 comprising interpolating a color of an empty pixel in the second frame formed by pixel shifting based on the colors of pixels of nearby pixels.

14. The method of claim 1 comprising:

upsampling the first frame to create an upsampled first frame prior to the determining the depth parameters for the shot and the pixel depth map for the first frame; and

downsampling the second frame.

15. A system for generating stereo video from mono video comprising:

a shot detection module configured to determine a first plurality of frames from a shot of a mono video, each frame of the first plurality of frames to be used as either a left frame or a right frame in a stereo frame pair;

a scene classification module configured to;

determine depth parameters for the shot, where the depth parameters determined for the shot are one of a plurality of different sets of depth parameters, wherein each set of depth parameters comprises K_near, a numerical value representing depth scaling of a closest object in one of the first plurality of frames to a viewer as a function of a horizontal resolution of the mono video, and K_far, a numerical value representing depth scaling of a farthest object in one of the first plurality of frames to the viewer as a function of the horizontal resolution of the mono video, and

determine a pixel depth map for each frame of the first plurality of frames of the shot, the pixel depth maps comprising a pixel depth for each pixel in each frame of the first plurality of frames; and

a depth image based rendering (DIBR) module configured to

determine a pixel shift amount for each pixel in each frame of the first plurality of frames based on the depth parameters and the pixel depth maps; and

generate a second plurality of frames based on the first plurality of frames laterally shifting the pixels of the first plurality of frames by the pixel shift amounts, each frame of the second plurality of frames to be used as either a right frame or a left frame so as to be another frame in a stereo frame pair.

16. The system of claim 15 wherein the depth parameters are the same for each frame of the first plurality of frames of the shot.

17. The system of claim 15 wherein the pixel depth is based on a red difference component and a blue difference component of the pixel.

18. The system of claim 15 wherein the pixel depth is further based eigenvalues multiplied by the red and blue difference components, wherein the eigenvalues are based on the red and blue difference components of all of the pixels of each frame of the first plurality of frames.

19. The system of claim 15 wherein the pixel depth is based on a direction of greatest spread between red difference components and blue difference components of the pixels of each frame of the first plurality of frames.

20. The system of claim 15 wherein responsive to a first and second pixel in frame of the first plurality of frames being pixel shifted to a same pixel in a frame of the second plurality of frames, determining the color of the pixel in the frame of the second plurality of frames from the pixel with the smallest or most negative pixel shift amount.

21. A method for generating stereo video from mono video comprising:

identifying a shot in the mono video, the shot comprising a plurality of mono frames;

identifying a scene type for the shot, the scene type is identified from one of a plurality of different scene types including at least an indoor scene type, an outdoor scene type, and a generic scene type, each of these scene types associated with a different set of depth parameters, wherein each set of depth parameters comprises K_near, a numerical value representing depth scaling of a closest object in one of the mono frames to a viewer as a function of a horizontal resolution of a mono video, and K_far, a numerical value representing depth scaling of a farthest object in one of the mono frames to the viewer as a function of the horizontal resolution of the mono video;

determining depth parameters for the shot based on the identified scene type;

selecting a first frame from the shot, to use as either a left frame or a right frame in a stereo frame pair;

determining a pixel depth map for the first frame based on the scene type, the pixel depth map comprising a pixel depth for each pixel in the first frame; and

generating a second frame based on the first frame, the second frame to be used as either the right frame or the left frame so as to be another frame in the stereo frame pair comprising;

determining a pixel shift amount for each pixel in the first frame, the pixel shift amount based on its pixel depth and the depth parameters for the shot; and

generating a pixel in the second frame by shifting a corresponding pixel in the first frame by its pixel shift amount.