Cross-modiality image matching method

• US 10,346,464 B2
• Filed: 09/27/2016
• Issued: 07/09/2019
• Est. Priority Date: 09/27/2016
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Ye, Getian
• Primary Examiner(s): Ky, Kevin
• Application Number:
  US15/277,211
• Publication Number:
  US 20180089534A1
• Time in Patent Office: 1,015 Days
• Field of Search: None
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

A computer-implementable method for determining a score for matching images includes extracting and determining. Features are extracted from each of a query image acquired in a first modality and a database image acquired in a second modality. A vector distance distribution is determined based on a codebook for the first modality for each of the query image and the database image represented by the extracted features. The method determines a set of distances between the vector distance distribution determined for the query image and the vector distance distribution determined for the database image. The score for matching the query image and the database image is determined based on the determined set of distances.

Claims

1. A method for an electronic device, the method comprising:

extracting features from each of a query image acquired in a first modality and a database image acquired in a second modality, wherein the first modality is different from the second modality, the query image is a face image and the database image is a face image having a head pose that is different from a head pose of the query image, and the query image and the database image are not aligned when the features are extracted;

determining, based on a codebook for the first modality, a vector distance distribution for the query image represented by the extracted features and a vector distance distribution for the database image represented by the extracted features;

generating a set of distances between the vector distance distribution determined for the query image and the vector distance distribution determined for the database image;

determining a score for matching the query image and the database image based on the determined set of distances; and

transmitting information related to the determined score to a server to cause the server to match the query image to the database image using the transmitted information and to display matched images to a user of the server.

2. The method according to claim 1, further comprising creating the codebook from a learned codebook for the first modality and a learned codebook for the second modality.

3. The method according to claim 1, further comprising building the codebook for the first modality using a range of training images.

4. The method according to claim 1,

wherein the first modality relates to one of the following:

image capture using a visible light sensor, an infrared sensor, and a sensor for a time-of-flight camera, and

wherein the second modality relates to one of the following;

image capture using a visible light sensor, an infrared sensor, and a sensor for a time-of-flight camera.

5. The method according to claim 1, wherein determining the vector distance distribution includes determining a vector representation for the features of the query image and determining a vector representation for the features of the database image.

6. The method according to claim 1, wherein determining the vector distance distribution for the query image includes determining distance between a vector representation of the features of the query image and each codeword of the codebook.

7. The method according to claim 1, wherein determining the vector distance distribution for the database image includes determining distance between a vector representation of the features of the database image and each codeword of the codebook.

8. The method according to claim 1, wherein the set of distances is determined using a function learned using a set of training images.

9. The method according to claim 1, further comprising ranking the score of the query image and the database image in comparison with a score of the query image and another database image to match the query image.

10. The method according to claim 1, further comprising matching the query image and the database image based on the determined score.

11. The method according to claim 1, wherein the vector distance distributions include first vector distance distributions, and the set of distances includes a first set of distances, the method further comprising:

selecting a second codebook associated with the second modality;

determining a second vector distance distribution based on a codebook for the second modality for each of the query image and the database image;

determining a second set of distances between the determined second vector distance distribution for the query image and the determined second vector distance distribution for the database image; and

determining the score based on the first set of distances and the second set of distances.

12. The method according to claim 1, wherein the query image is a face image captured at a first time instant and the database image is a face image captured at a second time instant that is different from the first time instant, the method further comprising matching the database face image to the query face image based on the determined score.

13. The method according to claim 12,

wherein the first modality relates to one of:

image capture using a visible light sensor, an infrared sensor, and a sensor for a time-of-flight camera,

wherein the second modality relates to one of;

image capture using a visible light sensor, an infrared sensor, and a sensor for a time-of-flight camera, wherein the image capture related to the first modality is different from the image capture related to the second modality, and

wherein matching the database face image to a query face image includes using other than a facial landmark or a face alignment.

14. The method according to claim 1, wherein a first camera associated with the first modality has a field of view and a second camera associated with the second modality has a field of view that does not overlap with the field of view of the first camera.

15. The method according to claim 14,

wherein a viewpoint of the query image is different to a viewpoint the database image, or

wherein a pose in the query image is different to a pose in the database image.

16. A method for an electronic device, the method comprising:

extracting features from each of a query image acquired in a first modality and a database image acquired in a second modality, wherein the first modality is different from the second modality, the query image is a face image and the database image is a face image having a head pose that is different from a head pose of the query image, and the query image and the database image are not aligned when the features are extracted;

determining a first vector distance distribution based on a codebook for the first modality and a second vector distance distribution based on a codebook for the second modality, wherein the first and second vector distance distributions is for each of the query image and the database image represented by the features respectively;

determining a first set of distances between the determined first vector distance distributions for the query image and the database image;

determining a second set of distances between the determined second vector distance distributions for the query image and the database image;

determining a score based on the first set of distances and the second set of distances;

matching the query image to the database image using the determined score; and

causing a user interface to display matched images to a user.

17. The method according to claim 16, wherein determining the score includes normalizing the first set of distances and the second set of distances.

18. The method according to claim 16, wherein determining the score includes determining a mean of the first set of distances and the second set of distances.

19. A camera comprising:

a sensor configured to capture a query image using a first modality;

a processor; and

a memory, the memory having instructions thereon executable by the processor to perform operations including;

extracting features from each of the query image acquired in a first modality and a database image acquired in a second modality, wherein the first modality is different from the second modality, the query image is a face image and the database image is a face image having a head pose that is different from a head pose of the query image, and the query image and the database image are not aligned when the features are extracted,

determining, based on a codebook for the first modality, a vector distance distribution for the query image represented by the extracted features and a vector distance distribution for the database image represented by the extracted features,

generating a set of distances between the vector distance distribution determined for the query image and the vector distance distribution determined for the database image,

determining a score for matching the query image and the database image based on the determined set of distances, and

transmitting information related to the determined score via a network.

20. A non-transitory computer readable storage medium having a computer program stored thereon to cause a computer to perform a method for an electronic device, the method comprising:

extracting features from each of a query image acquired in a first modality and a database image acquired in a second modality, wherein the first modality is different from the second modality, the query image is a face image and the database image is a face image having a head pose that is different from a head pose of the query image, and the query image and the database image are not aligned when the features are extracted;

determining, based on a codebook for the first modality, a vector distance distribution for the query image represented by the extracted features and a vector distance distribution for the database image represented by the extracted features;

generating a set of distances between the vector distance distribution determined for the query image and the vector distance distribution determined for the database image;

determining a score for matching the query image and the database image based on the determined set of distances; and

transmitting information related to the determined score to a server to cause the server to match the query image to the database image using the transmitted information and to display matched images to a user of the server.

21. A system comprising:

an image capture device configured to capture a query image using a first modality;

a database having database images captured using a second modality, wherein the first modality is different from the second modality;

a processor; and

a memory, the memory having instructions thereon executable by the processor to perform operations including;

extracting features from each of the query image and one of the database images, wherein the query image is a face image and the database image is a face image having a head pose that is different from a head pose of the query image, and the query image and the database image are not aligned when the features are extracted,

determining, based on a codebook for the first modality, a vector distance distribution for the query image represented by the extracted features and a vector distance distribution for the database image represented by the extracted features,

generating a set of distances between the vector distance distribution determined for the query image and the vector distance distribution determined for the database image,

determining a score for matching the query image and the database image based on the determined set of distances, and

transmitting information related to the determined score to a server to cause the server to match the query image to the database image using the transmitted information and to display matched images to a user of the server.