Image classification and information retrieval over wireless digital networks and the internet

• US 9,224,035 B2
• Filed: 12/02/2013
• Issued: 12/29/2015
• Est. Priority Date: 09/28/2005
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: 9051147 Canada Incorporated (Motorola Solutions, Inc.)
• Inventors: Myers, Charles A., Shah, Alex
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US14/094,739
• Publication Number:
  US 20140105467A1
• Time in Patent Office: 757 Days
• Field of Search: 382/118, 382/190
• US Class Current: 1/1
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/169 Holistic features and representations, i.e. based on the facial image taken as a whole
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

A method and system for matching an unknown facial image of an individual with an image of a celebrity using facial recognition techniques and human perception is disclosed herein. The invention provides a internet hosted system to find, compare, contrast and identify similar characteristics among two or more individuals using a digital camera, cellular telephone camera, wireless device for the purpose of returning information regarding similar faces to the user The system features classification of unknown facial images from a variety of internet accessible sources, including mobile phones, wireless camera-enabled devices, images obtained from digital cameras or scanners that are uploaded from PCs, third-party applications and databases. Once classified, the matching person'"'"'s name, image and associated meta-data is sent back to the user. The method and system uses human perception techniques to weight the feature vectors.

Claims

1. A method, for matching an unknown facial image of an individual with a known facial image of an individual, comprising:

storing on a server supporting a service usable by multiple users through respective remote user computing devices accessing the service over a distributed network a plurality of reference photos comprising identified facial images of individual users of the service and, for each of a plurality of identified facial images, an identity of the individual and at least one reference feature vector generated from the facial images to provide a plurality of stored reference feature vectors, the server comprising at least one processor that accesses at least one storage media and being programmed with executable instructions;

processing, by the server, an unknown facial image of an individual in a subject photo to generate a subject feature vector for the individual in the subject photo;

matching, by the server, the individual in the subject photo to an identified individual in the plurality of reference photos by executing a matching algorithm for comparing the subject feature vector to each of a plurality of stored reference feature vectors and selecting a reference feature vector that is most similar to the subject feature vector;

based on the selection, determining a predicted identify for the unknown facial image;

transmitting the predicted identify to a at least one computing device accessing the service for display with the unknown facial image to collect human perception feedback;

receiving, by the server, human perception feedback regarding the identity of the unknown facial image from at least one user of the service; and

updating, by the server, the matching algorithm based on the human perception feedback.

2. The method of claim 1, wherein the service is a social networking service.

3. The method of claim 1, wherein each feature vector comprises at least one variable that represents a feature of a corresponding facial image.

4. The method of claim 1, wherein the matching algorithm applies a statistical model for comparison of feature vectors.

5. The method of claim 4, wherein the statistical model is updated such that human perception feedback of what is a good match is a factor in the statistical model.

6. The method of claim 4 or 5, wherein the statistical model is updated such that human perception feedback of what is not a good match is a factor in the statistical model.

7. The method of claim 4, wherein the human perception feedback is used to weight one or more of the subject feature vector and at least one reference feature vector.

8. The method of claim 7, wherein one or more of the subject feature vector and at least one reference feature vector is weighted so that characteristics of the image that are most closely related to human perception are weighted more heavily.

9. The method of claim 5, 6, 7 or 8, wherein the statistical model is represented by a neural network, support vector machine or Bayesian network.

10. The method of claim 1, further comprising, upon determining that the predicted identity matches the identity based on human perception, storing the subject feature vector in association with an identifier for the identified individual for subsequent use by the matching algorithm in determining a predicted identity of a facial image of the individual in one or more other photos.

11. The method of claim 10, further comprising combining the subject feature vector and at least one reference feature vector of the individual for subsequent use by the matching algorithm in determining a predicted identity of a facial image of the individual in one or more other photos.

12. The method of claim 1, wherein, the subject photo is received by the server from a computer over the internet, or from a mobile telephone over a wireless network.

13. The method of claim 1, wherein the plurality of reference photos comprises multiple photos of at least one individual in different poses and facial expressions.

14. The method of claim 1, the plurality of reference feature vectors including at least one feature vector based on a plurality of reference photos.

15. The method of claim 1, further comprising analyzing, by the server, the unknown facial image of the subject photo to determine if the unknown facial image is acceptable.

16. The method of claim 1, further comprising processing the unknown facial image, by the server, to normalize the image to an appropriate scale and rotation.

17. A system for matching an unknown facial image of an individual with a known facial image of an individual, comprising:

storage supporting a service usable by multiple users through respective remote user computing devices accessing the service over a distributed network, the storage storing a plurality of reference photos comprising identified facial images of individual users of the service and, for each of a plurality of identified facial images, an identity of the individual and at least one reference feature vector generated from the facial images to provide a plurality of stored reference feature vectors;

at least one networked server supporting the service, wherein the at least one networked server has access to the storage and is configured to implement computer programming to perform the following;

processing an unknown facial Image of an individual in a subject photo to generate a subject feature vector for the individual in the subject photo;

matching the individual in the subject photo to an identified individual in the plurality of reference photos by executing a matching algorithm for comparing the subject feature vector to each of a plurality of stored reference feature vectors and selecting a reference feature vector that is most similar to the subject feature vector;

based on the selection, determining a predicted identify for the unknown facial image;

transmitting the predicted identify to a at least one computing device accessing the service for display with the unknown facial image to collect human perception feedback;

receiving human perception feedback regarding the identity of the unknown facial image from at least one user of the service; and

updating the matching algorithm based on the human perception feedback.

18. The system of claim 17, wherein the service is a social networking service.

19. The system of claim 17, wherein each feature vector comprises at least one variable that represents a feature of a corresponding facial image.

20. The system of claim 17, wherein the matching algorithm applies a statistical model for comparison of feature vectors.

21. The system of claim 20, wherein the server is configured to implement computer programming to update the statistical model such that human perception feedback of what is a good match is a factor in the statistical model.

22. The system of claim 20 or 21, wherein the server is configured to implement computer programming to update the statistical model such that human perception feedback of what is not a good match is a factor in the statistical model.

23. The system of claim 20, wherein the server is configured to implement computer programming to use the human perception feedback to weight one or more of the subject feature vector and at least one reference feature vector.

24. The system of claim 23, wherein one or more of the subject feature vector and at least one reference feature vector is weighted so that characteristics of the image that are most closely related to human perception are weighted more heavily.

25. The system of claim 21, 22, 23 or 24, wherein the statistical model is represented by a neural network, support vector machine or Bayesian network.

26. The system of claim 17, wherein the server is configured to implement computer programming to, upon determining that the predicted identity matches the identity based on human perception, store the subject feature vector in association with an identifier for the identified individual for subsequent use by the matching algorithm in determining a predicted identity of a facial image of the individual in one or more other photos.

27. The system of claim 26, wherein the server is configured to implement computer programming to combine the subject feature vector and at least one reference feature vector of the individual for subsequent use by the matching algorithm in determining a predicted identity of a facial image of the individual in one or more other photos.

28. The system of claim 17, wherein the server is configured to receive the subject photo from a computer over the Internet, or from a mobile telephone over a wireless network.

29. The system of claim 17, wherein the plurality of reference photos comprises multiple photos of at least one individual in different poses and facial expressions.

30. The system of claim 17, the plurality of reference feature vectors including at least one feature vector based on a plurality of reference photos.

31. The system of claim 17, wherein the server is configured to implement computer programming to analyze the unknown facial image of the subject photo to determine if the unknown facial image is acceptable.

32. The system of claim 17, wherein the server is configured to implement computer programming to process the unknown facial image to normalize the image to an appropriate scale and rotation.