System and method for the identification of personal presence and for enrichment of metadata in image media

• US 9,798,742 B2
• Filed: 12/21/2015
• Issued: 10/24/2017
• Est. Priority Date: 12/21/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cardonha, Carlos H., Sultanum, Nicole B.
• Primary Examiner(s): Hutton, Nan
• Application Number:
  US14/977,189
• Publication Number:
  US 20170177621A1
• Time in Patent Office: 673 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/9038 Presentation of query results
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 5/048 Fuzzy inferencing

Abstract

A method and system for the identification of personal presence and enrichment of metadata in image media is disclosed. The method includes obtaining user presence information for user images. Feature extraction is performed on the images and media databases are searched for images based on the presence information, which includes filtering based on known metadata and filtering based on the feature extraction. The user confirms their presence in the filtered images and the user provides new metadata known to the user for the images. The system then infers metadata for the filtered images based on the new metadata and presents the inferred metadata to the user. The user validates the inferred metadata and the inferred metadata confirmed to be valid is stored.

Claims

1. A method for the identification of personal presence and enrichment of metadata in image media, comprising:

obtaining from a user time and location information of the presence of the user at a specific location;

storing one or more images of the specific location obtained from the user in an image database;

performing feature extraction on the one or more user obtained images, including generating a classifier for each extracted feature using an unsupervised learning algorithm;

searching one or more media databases for candidate image media related to the specific location based on the location and time information, the candidate image media not necessarily being uploaded to the databases by the user;

filtering the candidate image media based on known metadata of the candidate image media that matches the location and time information and

filtering the candidate image media based on the feature extraction on the one or more user obtained images to provide filtered candidate image media, including applying the classifier to the candidate image media and retrieving a set of neighbor images from the image database based on metadata of the candidate image media;

storing the filtered candidate image media in the image database;

presenting to the user the filtered candidate image media;

receiving an input from the user indicating whether or not the user can confirm the presence of the user in the filtered candidate image media;

receiving from the user new metadata for the filtered candidate image media, the new metadata providing information not present in metadata of the metadata related image media;

storing the new metadata in the image database;

inferring metadata for the filtered candidate image media based on the new metadata, including inferring metadata from metadata of the set of neighbor images;

presenting the inferred metadata for the filtered candidate image media to the user;

receiving an input from the user indicating whether or not the user can confirm the validity of the inferred metadata; and

storing in the image database the inferred metadata for the filtered candidate image media confirmed by the user to be valid.

2. The method of claim 1, wherein obtaining from a user time and location information of the presence of the user at a location includes obtaining tracking information from a mobile device in the possession of the user.

3. The method of claim 1, further including applying the classifier to a newly stored image obtained from the user, the newly stored image having at least location metadata, retrieving a set of neighbor images from the image database based on the location metadata of the newly stored image and inferring metadata for the newly stored image from metadata of the set of neighbor images.

4. The method of claim 3, further including presenting the inferred metadata for the newly stored image to the user and receiving an input from the user indicating whether or not the user can confirm the validity of the new metadata for the newly stored image and storing in the image database the inferred metadata for the newly stored image confirmed by the user to be valid.

5. The method of claim 1, wherein filtering the candidate image media based on the feature extraction includes performing image recognition.

6. The method of claim 1, wherein searching one or more media databases includes searching databases identified by the user.

7. The method of claim 5, further including presenting a message to the user indicating the presence of the user in the filtered candidate image media as a result of the image recognition.

8. A non-transitory article of manufacture tangibly embodying computer readable instructions, which when implemented, cause a computer to perform the steps of a method for the identification of personal presence and enrichment of metadata in image media, the method comprising:

obtaining from a user time and location information of the presence of the user at a specific location;

storing one or more images of the specific location obtained from the user in a image database;

performing feature extraction on the one or more user obtained images, including generating a classifier for each extracted feature using an unsupervised learning algorithm;

searching one or more media databases for candidate image media related to the specific location based on the location and time information, the candidate image media not necessarily being uploaded to the databases by the user;

filtering the candidate image media based on known metadata of the candidate image media that matches the location and time information and filtering the candidate image media based on the feature extraction on the one or more user obtained images to provide filtered candidate image media, including applying the classifier to the candidate image media and retrieving a set of neighbor images from the image database based on metadata of the candidate image media;

storing the filtered candidate image media in the image database;

presenting to the user the filtered candidate image media;

receiving an input from the user indicating whether or not the user can confirm the presence of the user in the filtered candidate image media;

receiving from the user new metadata for the filtered candidate image media, the new metadata providing information not present in metadata of the metadata related image media;

storing the new metadata in the image database;

inferring metadata for the filtered candidate image media based on the new metadata, including inferring metadata from metadata of the set of neighbor images;

presenting the inferred metadata for the filtered candidate image media to the user;

receiving an input from the user indicating whether or not the user can confirm the validity of the inferred metadata; and

storing in the image database the inferred metadata for the filtered candidate image media confirmed by the user to be valid.

9. The non-transitory article of manufacture of claim 8, wherein obtaining from a user time and location information of the presence of the user at a location includes obtaining tracking information from a mobile device in the possession of the user.

10. The non-transitory article of manufacture of claim 8, further including applying the classifier to a newly stored image obtained from the user, the newly stored image having at least location metadata, retrieving a set of neighbor images from the image database based on the location metadata of the newly stored image and inferring metadata for the newly stored image from metadata of the set of neighbor images.

11. The non-transitory article of manufacture of claim 10, further including presenting the inferred metadata for the newly stored image to the user and receiving an input from the user indicating whether or not the user can confirm the validity of the new metadata for the newly stored image and storing in the image database the inferred metadata for the newly stored image confirmed by the user to be valid.

12. The non-transitory article of manufacture of claim 8, wherein filtering the candidate image media based on the feature extraction includes performing image recognition and the method further includes presenting a message to the user indicating the presence of the user in the filtered candidate image media as a result of the image recognition.

13. The non transitory article of manufacture of claim 8, wherein searching one or more media databases includes searching databases identified by the user.

14. A computer system for the identification of personal presence and enrichment of metadata in image media, the computer system comprising:

one or more computer processors;

one or more non-transitory computer-readable storage media;

program instructions, stored on the one or more non-transitory computer-readable storage media, which when implemented by the one or more processors, cause the computer system to perform the steps of;

obtaining from a user time and location information of the presence of the user at a specific location;

storing one or more images of the specific location obtained from the user in a image database;

performing feature extraction on the one or more user obtained images, including generating a classifier for each extracted feature using an unsupervised learning algorithm;

searching one or more media databases for candidate image media related to the specific location based on the location and time information, the candidate image media not necessarily being uploaded to the databases by the user;

filtering the candidate image media based on known metadata of the candidate image media that matches the location and time information and filtering the candidate image media based on the feature extraction on the one or more user obtained images to provide filtered candidate image media, including applying the classifier to the candidate image media and retrieving a set of neighbor images from the image database based on metadata of the candidate image media;

storing the filtered candidate image media in the image database;

presenting to the user the filtered candidate image media;

receiving an input from the user indicating whether or not the user can confirm the presence of the user in the filtered candidate image media;

receiving from the user new metadata for the filtered candidate image media, the new metadata providing information not present in metadata of the metadata related image media;

storing the new metadata in the image database;

inferring metadata for the filtered candidate image media based on the new metadata, including inferring metadata from metadata of the set of neighbor images;

presenting the inferred metadata for the filtered candidate image media to the user;

receiving an input from the user indicating whether or not the user can confirm the validity of the inferred metadata; and

storing in the image database the inferred metadata for the filtered candidate image media confirmed by the user to be valid.

15. The computer system of claim 14, wherein obtaining from a user time and location information of the presence of the user at a location includes obtaining tracking information from a mobile device in the possession of the user.

16. The computer system of claim 14, further including applying the classifier to a newly stored image obtained from the user, the newly stored image having at least location metadata, retrieving a set of neighbor images from the image database based on the location metadata of the newly stored image and inferring metadata for the newly stored image from metadata of the set of neighbor images.

17. The computer system of claim 16, further including presenting the inferred metadata for the newly stored image to the user and receiving an input from the user indicating whether or not the user can confirm the validity of the new metadata for the newly stored image and storing in the image database the inferred metadata for the newly stored image confirmed by the user to be valid.

18. The computer system of claim 14, wherein filtering the candidate image media based on the feature extraction includes performing image recognition and the method further includes presenting a message to the user indicating the presence of the user in the filtered candidate image media as a result of the image recognition.

19. The computer system of claim 14, wherein searching one or more media databases includes searching databases identified by the user.