Finding Missing Persons by Learning Features for Person Attribute Classification Based on Deep Learning

• US 20180039867A1
• Filed: 08/02/2016
• Published: 02/08/2018
• Est. Priority Date: 08/02/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cheng, Yu, Feris, Rogerio S., Pickover, Clifford A., Vukovic, Maja, Wang, Jing
• Granted Patent: US 10,163,042 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

An embodiment of the invention provides a method for finding missing persons by learning features for person attribute classification based on deep learning. A first component of a neural network identifies geographic locations of training images; and, a second component of the neural network identifies weather information for each of the identified geographic locations. A third component of the neural network generates image pairs from the training images. For each image pair of the image pairs, the third component of the neural network determines whether images of the image pair include the same person. The neural network generates neural network parameters with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person.

Claims

1. A method comprising:

identifying geographic locations of training images;

identifying weather information for each of the identified geographic locations;

generating image pairs from the training images;

for each image pair of the image pairs, determining whether images of the image pair include the same person; and

generating network parameters by the network, the network parameters being generated with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person.

2. The method according to claim 1, wherein the network includes:

a first branch trained to predict location;

a second branch trained to predict weather; and

a third branch trained to predict whether two images belong to the same person.

3. The method according to claim 1, wherein said identifying of the geographic locations of training images is performed by a first component of the network,

wherein said identifying of the weather information for each of the identified geographic locations is performed by a second component of the network,

wherein said generating of the image pairs from the training images is performed by a third component of the network, and

wherein said determining of whether the images of the image pairs include the same person is performed by the third component of the network.

4. The method according to claim 1, further comprising:

applying the network with the network parameters to an input image to extract a feature vector from the input image;

identifying at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

identifying at least one facial attribute in the input image with the classifier; and

comparing the identified clothing attribute and the identified facial attribute to a description of a missing person by a processor to determine a probability that the missing person is shown in the input image.

5. The method according to claim 4, wherein said identifying of the at least one facial attribute in the input image includes generating a probability score for each facial attribute, the probability score indicating a probability that the input image includes the facial attribute.

6. The method according to claim 4, wherein said identifying of the at least one clothing attribute in the input image includes generating a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

7. The method according to claim 4, wherein said applying of the network to the input image includes:

performing a sequence of hierarchical computations to produce a set of scalar values in each branch of the network; and

forming the feature vector by concatenating the sets of scalar values.

8. A method for finding missing persons by learning features for person attribute classification based on deep learning, said method comprising:

identifying geographic locations of training images by a first component of a neural network;

identifying weather information for each of the identified geographic locations by a second component of the neural network;

generating image pairs from the training images by a third component of the neural network, the image pairs being generated automatically by a video processing module, the video processing module including an object detector and tracker to detect and track people over a video sequence, and an image pair generator;

for each image pair of the image pairs, determining whether images of the image pair include the same person by the third component of the neural network; and

generating neural network parameters by the neural network, the neural network parameters being generated with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person.

9. The method according to claim 8, wherein said image pair generator:

labels a first person and a second person as different when the first person occurs in the same video frame as the second person,

labels a third person and a fourth person as different when the third person occurs in a video frame at a location and the fourth person occurs in a video frame at a different location, and

labels a fifth person and a sixth person as the same when the fifth person and the sixth person occur in the same region in consecutive video frames.

10. The method according to claim 8, wherein said identifying of the geographic locations of the training images is performed by a first neural network branch of the neural network,

wherein said identifying of the weather information for each of the identified geographic locations is performed by a second neural network branch of the neural network, and

wherein said determining of whether images of the image pair include the same person is performed by a Siamese neural network of the neural network.

11. The method according to claim 8, wherein the neural network is trained to predict the geographic locations and weather.

12. The method according to claim 8, further comprising:

applying the neural network with the neural network parameters to an input image to extract a feature vector from the input image;

identifying at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

identifying at least one facial attribute in the input image with the classifier; and

comparing the identified clothing attribute and the identified facial attribute to a description of a missing person by a processor to determine a probability that the missing person is shown in the input image.

13. The method according to claim 12, wherein said identifying of the at least one clothing attribute in the input image includes generating a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

14. The method according to claim 12, wherein said identifying of the at least one facial attribute in the input image includes generating a probability score for each facial attribute, the probability score indicating a probability that the input image includes the facial attribute.

15. The method according to claim 12, wherein said applying of the neural network to the input image includes:

performing a sequence of hierarchical computations to produce a set of scalar values in each layer of the neural network; and

forming the feature vector by concatenating the sets of scalar values.

16. A computer program product comprising:

a computer readable storage medium having stored thereon;

first program instructions executable by a device to cause the device to identify geographic locations of training images;

second program instructions executable by the device to cause the device to identify weather information for each of the identified geographic locations;

third program instructions executable by the device to cause the device to generate image pairs from the training images;

fourth program instructions executable by the device to cause the device to, for each image pair of the image pairs, determine whether images of the image pair include the same person; and

fifth program instructions executable by the device to cause the device to generate network parameters, the network parameters being generated with the identified geographic locations, the weather information for each of the identified geographic locations, and the determination of whether the images of the image pairs include the same person.

17. The computer program product according to claim 16, further comprising:

sixth program instructions executable by the device to cause the device to apply the network with the network parameters to an input image to extract a feature vector from the input image;

seventh program instructions executable by the device to cause the device to identify at least one clothing attribute in the input image with a classifier that receives as input the feature vector;

eighth program instructions executable by the device to cause the device to identify at least one facial attribute in the input image; and

ninth program instructions executable by the device to cause the device to compare the identified clothing attribute and the identified facial attribute to a description of a missing person to determine a probability that the missing person is shown in the input image.

18. The computer program product according to claim 16, wherein said eighth program instructions generates a probability score for each facial attribute, the probability score indicating a probability that the input image includes the facial attribute.

19. The computer program product according to claim 16, wherein said seventh program instructions generates a probability score for each clothing attribute, the probability score indicating a probability that the input image includes the clothing attribute.

20. The computer program product according to claim 16, wherein said sixth program instructions:

perform a sequence of hierarchical computations to produce a set of scalar values in each layer of the network; and

form the feature vector by concatenating the sets of scalar values.