Method for generating spatial-temporally consistent depth map sequences based on convolution neural networks

• US 10,540,590 B2
• Filed: 12/29/2016
• Issued: 01/21/2020
• Est. Priority Date: 12/29/2016
• Status: Active Grant

• Current Assignee: Zhejiang Gongshang University
• Original Assignee: Zhejiang Gongshang University
• Inventors: Wang, Xun, Zhao, Xuran
• Primary Examiner(s): Schnee, Hal
• Application Number:
  US16/067,819
• Publication Number:
  US 20190332942A1
• Time in Patent Office: 1,118 Days
• CPC Class Codes: G06F 17/13 Differential equations using digital differential analysers G06F7/64
• CPC Class Codes: G06F 17/16 Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof

Abstract

A method for generating spatial-temporal consistency depth map sequences based on convolutional neural networks for 2D-3D conversion of television works includes steps of: 1) collecting a training set, wherein each training sample thereof includes a sequence of continuous RGB images, and a corresponding depth map sequence; 2) processing each image sequence in the training set with spatial-temporal consistency superpixel segmentation, and establishing a spatial similarity matrix and a temporal similarity matrix; 3) establishing the convolution neural network including a single superpixel depth regression network and a spatial-temporal consistency condition random field loss layer; 4) training the convolution neural network; and 5) recovering a depth maps of a RGB image sequence of unknown depth through forward propagation with the trained convolution neural network; which avoids that clue-based depth recovery method is greatly depended on scenario assumptions, and inter-frame discontinuity between depth maps generated by conventional neural networks.

Claims

1. A method for generating spatial-temporally consistent depth map sequences based on convolution neural networks, comprising steps of:

1) collecting a training set, wherein each training sample of the training set comprises a continuous RGB (red, green, blue) image sequence of m frames, and a corresponding depth map sequence;

2) processing each image sequence in the training set with spatial-temporal consistency superpixel segmentation, and establishing a spatial similarity matrix S^(s) and a temporal similarity matrix S^(t);

3) building a convolution neural network structure, wherein the convolution neural network comprises a single superpixel depth regression network with a parameter W, and a spatial-temporal consistency condition random field loss layer with a parameter α

;

4) training the convolution neural network established in the step

     3) with the continuous RGB image sequence and the corresponding depth map sequence in the training set, so as to obtain the parameter W and the parameter α

; and

5) recovering a depth map sequence of a depth-unknown RGB image sequence through forward propagation with the convolution neural network trained;

wherein the step

     2) specifically comprises steps of;

(2.1) processing the continuous RGB image sequence in the training set with the spatial-temporal consistency superpixel segmentation, wherein an input sequence is marked as I=[I₁, . . . , I_m], where I_t is a t-th frame of the m frames in total;

the m frames are respectively divided into n₁, . . . , n_m superpixels by the spatial-temporal consistency superpixel segmentation while a corresponding relation between all superpixels in a later frame and superpixels corresponding to a same object in a former frame is generated;

the whole image sequence comprises

2. The method, as recited in claim 1, wherein in the step 3), the convolution neural network comprises the single superpixel depth regression network and a spatial-temporal consistency condition random field loss layer;

wherein the step

     3) specifically comprises steps of;

(3.1) for the single superpixel depth regression network comprising first 31 layers of a VGG16 network, a superpixel pooling layer and three fully connected layers, which performs average pooling in each superpixel space of the superpixel pooling layer;

wherein an input of the network is continuous RGB images of m frames and an output of the network is an n-dimensional vector z=[z₁, . . . z_n], in which a p-th element z_p is an estimated depth value, without considering any constraint, of the superpixel p of the continuous RGB image sequence after the spatial-temporal consistency superpixel segmentation;

a convolution neural network parameter need to be learned is W; and

(3.2) using the output z=[z₁, . . . z_n] of the single superpixel depth regression network obtained in the step (3.1), the real depth vector d=[d₁;

. . . ;

d_n] of the superpixels obtained in the step (2.1), and the spatial similarity matrix S_pq^(s) obtained in the step (2.2) as well as the temporal similarity matrix S_pq^(t) obtained in the step (2.3) as an input of the spatial-temporal consistency condition random field loss layer;

wherein a conditional probability function of a spatial-temporal consistency condition random field is;

3. The method, as recited in claim 2, wherein in the step 4), training the convolution neural network specifically comprises steps of:

(4.1) optimizing the parameters W, α

^(s) and α

^(t) using stochastic gradient descent, wherein for each iteration, the parameters are updated as;

4. The method, as recited in claim 3, wherein in the step 5), recovering the depth-unknown RGB image sequence specifically comprises steps of:

(5.1) processing the RGB image sequence with the spatial-temporal consistency superpixel segmentation, and calculating the spatial similarity matrix S^(s) and the temporal similarity matrix S^(t);

(5.2) applying the forward propagation to the RGB image sequence with the convolution neural network trained, so as to obtain a single superpixel network output z;

(5.3) calculating a depth output {circumflex over (d)}=[{circumflex over (d)}₁;

. . . ;

{circumflex over (d)}_n] with spatial-temporal consistency constraint by;

{circumflex over (d)}=L<sup>−

1</sup>z

wherein the matrix L is calculated in the step (3.2);

{circumflex over (d)}_p represents an estimated depth value of a superpixel p in the RGB image sequence; and

(5.4) applying {circumflex over (d)}_p to a corresponding position of a corresponding frame of the superpixel p for obtaining a depth map of the m frames.