Multi-View Synthesis in Real-Time With Fallback to 2D from 3D to Reduce Flicker in Low or Unstable Stereo-Matching Image Regions

• US 20150269737A1
• Filed: 03/24/2014
• Published: 09/24/2015
• Est. Priority Date: 03/24/2014
• Status: Active Grant

• Current Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Original Assignee: Hong Kong Applied Science And Technology Research Institute Co., Ltd.
• Inventors: LAM, Che Yuen Brian, CHEUNG, Wei Lun Alan
• Granted Patent: US 9,407,896 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/351 for displaying simultaneously
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

Multi view images are generated with reduced flickering. A first depth map is generated from stereo images by stereo-matching. When stereo-matching is poor or varies too much from frame to frame, disparity fallback selects a second depth map that is generated from a single view without stereo-matching, preventing stereo-matching errors from producing visible artifacts or flickering. Flat or textureless regions can use the second depth map, while regions with good stereo-matching use the first depth map. Depth maps are generated with a one-frame delay and buffered. Low-cost temporal coherence reduces costs used for stereo-matching when the pixel location selected as the lowest-cost disparity is within a distance threshold of the same pixel in a last frame. Hybrid view synthesis uses forward mapping for smaller numbers of views, and backward mapping from the forward-mapping results for larger numbers of views. Rotated masks are generated on-the-fly for backward mapping.

Claims

1. A multi-view synthesizer comprising:

a first depth generator that generates a first depth map by stereo-matching a first stereo image to a second stereo image;

a consistency checker that generates a stereo-matching rate indicating a quality of the first depth map, the consistency checker comparing matched pixel locations when starting with the first stereo image to pixel locations when starting with the second stereo image to generate the stereo-matching rate;

a second depth generator that generates a second depth map from no more than one single view image;

a disparity fallback selector that selects the second depth map as a selected depth map instead of the first depth map when the stereo-matching rate is below a matching threshold;

a forward mapper that shifts sub-pixels from a source image by an offset to generate a destination image, the offset being read from or generated from the selected depth map, wherein the offset is scaled to a different value for each view image in a plurality of multi view images output as the destination image; and

a backward mapper that reads a plurality of view masks to locate a selected view mask that generates a sub-pixel location in the destination image, and locates a source sub-pixel in the source image using a view offset for the selected view mask, the backward mapper generating the destination image by copying the source sub-pixel to the sub-pixel location in the destination image.

2. The multi-view synthesizer of claim 1 further comprising:

a hybrid mapper that activates the forward mapper to generate destination images as intermediate images, and activates the backward mapper to use the intermediate images as the source image to generate the destination image as an output image.

3. The multi-view synthesizer of claim 1 further comprising:

a mask rotator that rotates a view mask by an angle to generate a rotated view mask for use by the backward mapper;

wherein view masks are stored in a memory while the rotated view mask is generated and not stored in the memory.

4. The multi-view synthesizer of claim 3 wherein the mask rotator further comprises:

a sub-pixel calculator that calculates locations of Red, Blue, and Green sub-pixels;

a reverse rotator that rotates locations of the Red, Blue, and Green sub-pixels to generate fractional sub-pixel locations;

an address generator that generates base sub-pixel addresses in a base mask for the fractional sub-pixel locations; and

an adder that adds fractions of sub-pixels in the base mask read at the base sub-pixel addresses to generate the rotated view mask.

5. The multi-view synthesizer of claim 1 further comprising:

a frame delay buffer that stores the selected depth map from the disparity fallback selector that was generated in a prior frame;

whereby depth maps are generated for a prior frame and delayed by the frame delay buffer for use by the forward mapper and the backward mapper in a current frame.

6. The multi-view synthesizer of claim 1 further comprising:

a low-cost temporal coherence adjustor that reduces costs used to generate the first depth map when a matched pixel location in a current frame is less than a distance threshold from the matched pixel location in a prior frame.

7. The multi-view synthesizer of claim 1 further comprising:

a frame-to-frame disparity fallback selector that selects the second depth map as the selected depth map when a difference between the stereo-matching rate in a current frame and the stereo-matching rate in a prior frame is more than a variation threshold.

8. The multi-view synthesizer of claim 1 further comprising:

a downsampler that down-samples a high-resolution input image to generate the first stereo image and the second stereo image having a lower resolution than the high-resolution input image;

wherein the one single view image input to the second depth generator is the first stereo image.

9. The multi-view synthesizer of claim 1 further comprising:

a cost generator that compares a reference pixel from the first stereo image to a range of pixels surrounding a reference location that corresponds to the reference pixel in the second stereo image to generate a plurality of costs;

an optimizer that selects as a matching pixel location in the second stereo image a pixel location having a lowest cost in the plurality of costs.

10. The multi-view synthesizer of claim 1 further wherein the selected depth map comprises depth values, disparity values, or offsets for pixels in an image.

11. A method for generating an output depth map from a first view image and a second view image comprising:

generating a first depth map from the first and second view images, wherein depth values in the first depth map are generated based on disparities generated by matching pixels in the first view image with pixels in the second view image, a disparity indicating a shift in position of a matching pixel in the first view image and in the second view image;

determining a stereo-matching rate when generating the first depth map, the stereo-matching rate indicating a consistency of results when the first view image and the second view image are reversed as inputs during generation of the first depth map;

generating a second depth map from the first view image, wherein depth values in the second depth map are generated based on visual cues in the first view image without any input from the second view image;

selecting the first depth map as the output depth map when the stereo-matching rate is above a match threshold; and

selecting the second depth map as the output depth map when the stereo-matching rate is not above the match threshold.

12. The method of claim 11 further comprising:

wherein the first view image and the second view image are for a current frame in a video sequence of frames, and wherein the stereo-matching rate is for the current frame;

storing the stereo-matching rate for a prior frame in the video sequence;

comparing the stereo-matching rate for the prior frame to the stereo-matching rate for the current frame to generate a rate variation between the current frame and the prior frame;

when the rate variation exceeds a variation threshold, selecting the second depth map as the output depth map,

whereby larger frame-to-frame variations in stereo-matching rates cause fallback to the second depth map generated without stereo-matching.

13. The method of claim 11 further comprising:

dividing a current frame into a plurality of regions;

selecting the second depth map for pixels in a target region in the plurality of regions for inclusion in the output depth map when the stereo-matching rate for the target region is not above the match threshold,

wherein the output depth map contains multiple regions, some regions having portions of the first depth map and other regions having portions of the second depth map.

14. The method of claim 13 further comprising:

adjusting the match threshold to have different threshold values for different regions in the plurality of regions,

whereby different match thresholds are used for different regions.

15. The method of claim 13 further comprising:

determining when a region in the plurality of regions is a textureless region having a low variation of pixels within the region;

selecting the second depth map for pixels in the textureless region in the plurality of regions for inclusion in the output depth map.

16. The method of claim 11 further comprising forward mapping by:

generating offsets from values stored in the output depth map, wherein an offset for a sub-pixel location is scaled by a plurality of scale factors to generate a plurality of view offsets for each sub-pixel location;

shifting sub-pixels in a source image by the view offsets to generate a plurality of view images;

filling holes having no sub-pixel in a sub-pixel location in the plurality of view images;

masking out colors that are not present in a view image to generate a plurality of masked view images from the plurality of view images; and

summing the plurality of view images to generate a multi-view image for output to a three-dimensional display;

and further comprising backward mapping by;

generating offsets from values stored in the output depth map, wherein an offset for a sub-pixel location is scaled by a plurality of scale factors to generate a plurality of view offsets for each sub-pixel location;

for each output sub-pixel location in an output image;

determining a generating view mask in the plurality of view masks that generates an output sub-pixel in the output sub-pixel location;

obtaining a generating scale factor for the generating view mask;

reading the output depth map to obtain a target offset for the output sub-pixel location;

multiplying the target offset by the generating scale factor to generate a view-adjusted offset;

adding the view-adjusted offset to the output sub-pixel location to generate a source sub-pixel address;

reading a target source sub-pixel from the source image and copying the target source sub-pixel to the output sub-pixel location in the output image; and

repeating for other sub-pixel locations in the output image until the output image is built.

17. The method of claim 16 further comprising:

using forward mapping to generate a first plurality of views;

using backward mapping to generate a second plurality of views when a total number of views exceeds a view threshold, wherein the view threshold is at least 2 views.

18. A multi-view system comprising:

a first map generator that receives a first view image and a second view image, the first view image and the second view image being different view perspectives of a same captured image, the first map generator generating a first map having disparity or depth values for pixels, the disparity values indicating a shift in pixel position between the first view image and the second view image, wherein the disparity values are inversely proportional to the depth values;

a second map generator that receives the first view image and not the second view image, the second map generator using a non-stereo technique to generate depth values or disparity values for inclusion in a second map;

wherein the first map is generated using a stereo image technique while the second map is generated using a non-stereo technique;

a stereo-matching rate generator that generates a stereo-matching rate when the first map generator generates the disparity or depth values;

a disparity fallback selector that selects values from the second map for inclusion in an output map when the stereo-matching rate is below a rate threshold, and selects values from the first map for inclusion in the output map when the stereo-matching rate is above the rate threshold;

a cost generator that generates matching costs for pixels in the first view image being matched to pixels in the second view image;

an optimizer that selects a winning pixel location having a lowest matching cost; and

a low-cost temporal coherence adjustor that adjusts the matching costs when the winning pixel location for a current frame is within a distance threshold of the winning pixel location for a prior frame.

19. The multi-view system of claim 18 wherein the disparity fallback selector further selects values from the second map for inclusion in the output map when the stereo-matching rate for the current frame is more than a variation threshold from the stereo-matching rate of the prior frame,

whereby larger frame-to-frame stereo-matching rate variation falls back to the second map.

20. The multi-view system of claim 18 further comprising:

a forward mapper that shifts a sub-pixel in the first view image by a scaled view offset that is generated from a value in the output map to generate an output image;

a backward mapper that identifies an identified view mask that generates a sub-pixel location in the output image, and generates a view adjusted offset that is based on the identified view mask and values in the output map, and uses the view adjusted offset to find a source sub-pixel in the first view image to copy to the output image; and

a hybrid mapper that activates the forward mapper to generate a first plurality of views, and activates the backward mapper to generate a second plurality of views, when a total number of views exceeds a view threshold, wherein the view threshold is at least 2 views.