Finding missing persons by learning features for person attribute classification based on deep learning

• US 10,163,042 B2
• Filed: 08/02/2016
• Issued: 12/25/2018
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cheng, Yu, Feris, Rogerio S., Pickover, Clifford A., Vukovic, Maja, Wang, Jing
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/226,339
• Publication Number:
  US 20180039867A1
• Time in Patent Office: 875 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

An embodiment of the invention provides a method for finding missing persons by learning features for person attribute classification based on deep learning. A first component of a neural network identifies geographic locations of training images; and, a second component of the neural network identifies weather information for each of the identified geographic locations. A third component of the neural network generates image pairs from the training images. For each image pair of the image pairs, the third component of the neural network determines whether images of the image pair include the same person. The neural network generates neural network parameters with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person.

Claims

1. A method, comprising:

identifying geographic locations of training images;

identifying weather information for each of the identified geographic locations;

generating image pairs from the training images;

for each image pair of the image pairs, determining whether images of the image pair include the same person; and

generating network parameters for a neural network with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person,

wherein the identified geographic locations, identified weather information, and determination of whether the images of the image pair include the same person are obtained without human annotation.

2. The method according to claim 1, wherein said identifying of the geographic locations of training images is performed by a first component of the neural network,

wherein said identifying of the weather information for each of the identified geographic locations is performed by a second component of the neural network,

wherein said generating of the image pairs from the training images is performed by a third component of the neural network, and

wherein said determining of whether the images of the image pairs include the same person is performed by the third component of the neural network.

3. The method according to claim 1, further comprising:

applying the neural network with the network parameters to an input image to extract a feature vector from the input image;

identifying at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

identifying at least one facial attribute in the input image with the classifier; and

comparing the identified clothing attribute and the identified facial attribute to a description of a missing person by a processor to determine a probability that the missing person is shown in the input image.

4. The method according to claim 3, wherein said identifying of the at least one facial attribute in the input image includes generating a probability score for each facial attribute, the probability score indicating a probability that the input image includes the facial attribute.

5. The method according to claim 3, wherein said identifying of the at least one clothing attribute in the input image includes generating a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

6. The method according to claim 3, wherein said applying of the neural network to the input image includes:

performing a sequence of hierarchical computations to produce a set of scalar values in each branch of the network; and

forming the feature vector by concatenating the sets of scalar values.

7. A method for finding missing persons by learning features for person attribute classification based on deep learning, said method comprising:

identifying geographic locations of training images by a first component of a neural network;

identifying weather information for each of the identified geographic locations by a second component of the neural network;

generating image pairs from the training images by a third component of the neural network, the image pairs being generated automatically by a video processing module, the video processing module including an object detector and tracker to detect and track people over a video sequence, and an image pair generator;

for each image pair of the image pairs, determining whether images of the image pair include the same person by the third component of the neural network; and

generating neural network parameters by the neural network, the neural network parameters being generated with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person,

wherein the identified geographic locations, identified weather information, and determination of whether the images of the image pair include the same person are obtained without human annotation.

8. The method according to claim 7, wherein said image pair generator:

labels a first person and a second person as different when the first person occurs in the same video frame as the second person;

labels a third person and a fourth person as different when the third person and the fourth person occur in video frames acquired at different locations, and

labels a fifth person and a sixth person as the same when the fifth person and the sixth person occur in the same region in consecutive video frames.

9. The method according to claim 7, wherein said identifying of the geographic locations of the training images is performed by a first neural network branch of the neural network,

wherein said identifying of the weather information for each of the identified geographic locations is performed by a second neural network branch of the neural network, and

wherein said determining of whether images of the image pair include the same person is performed by a Siamese neural network of the neural network.

10. The method according to claim 7, wherein the neural network is trained to predict the geographic locations and weather.

11. The method according to claim 7, further comprising:

Applying the neural network with the neural network parameters to an input image to extract a feature vector from the input image;

identifying at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

identifying at least one facial attribute in the input image with the classifier; and

comparing the identified clothing attribute and the identified facial attribute to a description of a missing person by a processor to determine a probability that the missing person is shown in the input image.

12. The method according to claim 11, wherein said identifying of the at least one clothing attribute in the input image includes generating a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

13. The method according to claim 11, wherein said identifying of the at least one facial attribute in the input image includes generating a probability score for each facial attribute, the probability score indicating that the input image includes the facial attribute.

14. The method according to claim 11, herein said applying of the neural network to the input image include:

performing a sequence of hierarchical computations to produce a set of scalar values in each layer of the neural network; and

forming the feature vector by concatenating the sets of scalar values.

15. A non-transitory computer-readable medium having computer-readable instructions stored thereon which when executed by a computer cause the computer to perform a method comprising:

identifying geographic locations of training images;

identifying weather information for each of the identified geographic locations;

generating image pairs from the training images;

for each image pair of the image pairs, determining whether images of the image pair include the same person; and

generating network parameters for a neural network with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person,

wherein the identified geographic locations, identified weather information, and determination of whether the images of the image pair include the same person are obtained without human annotation.

16. The non-transitory computer-readable medium according to claim 15, further comprising:

applying the network with the network parameters to an input image to extract a feature vector from the input image;

identifying at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

identifying at least one facial attribute in the input image; and

comparing the identified clothing attribute and the identified facial attribute to a description of a missing person to determine a probability that the missing person is shown in the input image.

17. The non-transitory computer-readable medium according to claim 16, wherein the identifying of at least one facial attribute in the input image includes generating a probability score for each facial attribute, the probability score indicating a probability that the input image includes the facial attribute.

18. The non-transitory computer-readable medium according to claim 16, wherein the identifying of at least one clothing attribute includes generating a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

19. The non-transitory computer-readable medium according to claim 16, wherein the applying of the network with the network parameters includes

performing a sequence of hierarchical computations to produce a set of scalar values in each layer of the network;

and

forming the feature vector by concatenating the sets of scalar values.