METHOD AND DEVICE FOR QUASI-GIBBS STRUCTURE SAMPLING BY DEEP PERMUTATION FOR PERSON IDENTITY INFERENCE

• US 20180181842A1
• Filed: 12/22/2016
• Published: 06/28/2018
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Original Assignee: TCL Research America, Inc. (TCL Technology Group Corp.)
• Inventors: LIAO, Xinpeng, SUN, Xinyao, REN, Xiaobo, WANG, Haohong
• Granted Patent: US 10,339,408 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24143 Distances to neighbourhood prototypes, e.g. restricted Coulomb energy networks [RCEN]
• CPC Class Codes: G06F 18/24147 Distances to closest patterns, e.g. nearest neighbour classification
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

The present disclosure provides a method and device for visual appearance based person identity inference. The method may include obtaining a plurality of input images. The input images include a gallery set of images containing, persons-of-interest and a probe set of images containing person detections, and one input image corresponds to one person. The method may further include extracting N feature maps from the input images using a Deep Neural Network, N being a natural number; constructing N structure samples of the N feature maps using conditional random field (CRF) graphical models; learning the N structure samples from an implicit common latent feature space embedded in the N structure samples; and according to the learned structures, identifying one or more images from the probe set containing a same person-of-interest as an image in the gallery set.

Claims

1. A method for visual appearance based person identity inference, comprising:

obtaining a plurality of input images, wherein the input images include a gallery set of images containing persons-of-interest and a probe set of images containing person detections;

extracting N feature maps from the input images using a Deep Neural Network, N being a natural number;

constructing N structure samples of the N feature maps using conditional random field (CRF) graphical models;

learning the N structure samples from an implicit common latent feature space embedded in the N structure samples; and

according to the learned structures, identifying one or more images from the probe set containing a same person-of-interest as an image in the gallery set.

2. The method according to claim 1, wherein constructing N structure samples of the N feature maps further comprises:

for a feature map, constructing an initial graph structure by K Nearest Neighbor (KNN) based on feature similarity in a feature space corresponding to the feature map, the graph model including nodes and edges, a node representing one person;

performing structure permutations by a plurality of iterations of KNN computation in N feature spaces with a Quasi-Gibbs Structure Sampling (QGSS) process;

assigning labels to the nodes that minimize a conditional random field (CRF) energy function over all possible labels, wherein the all possible labels represent all different persons-of-interest in the gallery set; and

deriving the N structure samples from the plurality of iterations and the assigned labels.

3. The method according to claim 2, wherein:

the plurality of iterations include first a iterations and later b iterations, wherein a and b are natural numbers;

results of the first a iterations are discarded, and

the N structure samples are derived from the later b iterations.

4. The method according to claim 2, wherein:

a node in the graph model has m possible states, m representing a quantity of different persons-of-interest in the gallery set.

5. The method according to claim 2, wherein:

the labels are assigned to the nodes according to the graph structure after the plurality of iterations are finished.

6. The method according to claim 1, wherein:

a graph of a CRF model representing a re-identification structure is learned through the N structure samples; and

an energy minimization with sparse approach is performed to cut the graph into a plurality of clusters, each cluster containing images corresponding to one of the persons-of-interest.

7. The method according to claim 1, wherein

N different kernels are used in the DNN for convolutions with the images in the gallery set and the probe set;

and

the N feature maps are produced by a last couple of convolution layers in the DNN.

8. The method according to claim 2, wherein the QGSS sampling process for a posterior conditional P in a deep feature space in iteration i is:

(Q_j, Y_j)⁽ⁱ⁾˜

P(QY_j=(Q_j, Y_j)|X;

, . . . , QY<sub>j−

1</sub>=(Q<sub>j−

1</sub>, Y<sub>j−

1</sub>)⁽ⁱ⁾, QY_j+1=(Q_j+1, Y_j+1)<sup>(i−

1)</sup> . . . ;

θ

)

wherein;

j is the index for N structure samples, Q_j=G (V, E) denotes the graph structure constructed by KNN, and θ

denotes parameters in the CRF models.

9. The method according to claim 6, wherein the CRF model with pairwise potentials is:

10. A device for visual appearance based person identity inference, comprising one or more processors configured to:

obtain a plurality of input images, wherein the input images include a gallery set of images containing persons-of-interest and a probe set of images containing person detections, and one input image corresponds to one person;

extract N feature maps from the input images using a Deep Neural Network, N being a natural number;

construct N structure samples of the N feature naps using conditional random field (CRF) graphical models;

learn the N structure samples from an implicit common latent feature space embedded in the N structure samples; and

according to the learned structures, identify one or more images from the probe set containing a same person-of-interest as an image in the gallery set.

11. The device according to claim 10, wherein the one or more processors constructing N structure samples of the N feature maps further comprises:

for a feature map, constructing an initial graph structure by K Nearest Neighbor (KNN) based on feature similarity in a feature space corresponding to the feature map, the graph model including nodes and edges, a node representing one person;

performing structure permutations by a plurality of iterations of KNN computation in N feature spaces with a Quasi-Gibbs Structure Sampling (QGSS) process;

assigning labels to the nodes that minimize a conditional random field (CRF) energy function over all possible labels, wherein the all possible labels represent all different persons-of-interest in the gallery set; and

deriving the N structure samples from the plurality of iterations and the assigned labels.

12. The device according to claim 11, wherein:

the plurality of iterations include first a iterations and later b iterations, wherein a and b are natural numbers;

results of the first a iterations are discarded, and

the N structure samples are derived from the later b iterations.

13. The device according to claim 11, wherein:

a node in the graph model has m possible states, m representing a quantity of different persons-of-interest in the gallery set.

14. The device according to claim 11, wherein:

the labels are assigned to the nodes according to the graph structure after the plurality of iterations are finished.

15. The device according to claim 10, wherein:

a graph of a CRF model representing a re-identification structure is learned through the N structure samples; and

an energy minimization with sparse approach is performed to cut the graph into a plurality of clusters, each cluster containing images corresponding to one of the persons-of-interest.

16. The device according to claim 10, wherein

N different kernels are used in the DNN for convolutions with the images in the gallery set and the probe set;

and

the N feature maps are produced by a last couple of convolution layers in the DNN.

17. The device according to claim 11, wherein the QGSS sampling process for a posterior conditional P in a deep feature space in iteration i is:

(Q_j, Y_j)⁽ⁱ⁾˜

P(QY_j=(Q_j, Y_j)|X;

, . . . , QY<sub>j−

1</sub>=(Q<sub>j−

1</sub>, Y<sub>j−

1</sub>)⁽ⁱ⁾, QY_j+1=(Q_j+1, Y_j+1)<sup>(i−

1)</sup> . . . ;

θ

)

wherein;

j is the index for N structure samples, Q_j=G (V, E) denotes the graph structure constructed by KNN, and θ

denotes parameters in the CRF models.

18. The device according to claim 15, wherein the CRF model with pairwise potentials is: