Methods and systems of personalized photo albums based on social media data

• US 10,346,466 B2
• Filed: 04/18/2016
• Issued: 07/09/2019
• Est. Priority Date: 04/18/2016
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Cai, Ke Ke, Duan, Dong Xu, Sun, Changhua, Zhang, Li, Zhao, Shi Wan
• Primary Examiner(s): Motsinger, Sean T
• Application Number:
  US15/131,324
• Publication Number:
  US 20170300782A1
• Time in Patent Office: 1,177 Days
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

Embodiments include methods, and mobile computing devices, and computer program products for creating personalized photo albums on mobile computing devices based on social media data. Aspects include: accessing, via mobile computing device, various photos taken by a user of mobile computing device, retrieving, from one or more social media streams, various media photos posted by user of the mobile computing device, extracting image features from the photos and the media photos, and text features from the media photos, generating photo clusters based on the image features of the photos, and media photo clusters based on the image features of the media photos, respectively, matching the photo clusters and the media photo clusters, tagging the photo clusters and the media photo clusters matched based on the text features extracted from the media photos, and generating a personalized photo album based on the photo clusters tagged and the media photo clusters tagged.

Claims

1. A method of creating a personalized photo album on a mobile computing device based on social media data, comprising:

accessing, by a processor, a plurality of photos stored on the mobile computing device, wherein the plurality of photos were taken by a user of the mobile computing device;

retrieving, by the processor, a plurality of media photos from one or more social media streams for the user, wherein the plurality of media photos were posted by the user of the mobile computing device;

extracting, by the processor, a plurality of image features from the plurality of photos and the plurality of media photos, and a plurality of text features from the plurality of media photos, wherein the image features of the plurality of photos and image features of the plurality of media photos each include a time when each photo was taken and a location where each photo was taken, and wherein the text features comprise a name of each tagged person in the plurality of photos and a name of each tagged object in the plurality of photos;

generating, by the processor, a plurality of photo clusters based on the plurality of image features of the plurality of photos, and a plurality of media photo clusters based on the plurality of image features of the plurality of media photos, respectively;

matching, by the processor, the plurality of photo clusters and the plurality of media photo clusters;

tagging, by the processor, the plurality of photo clusters with the plurality of text features extracted from the plurality of media photos;

generating, by the processor, a personalized photo album on the mobile computing device based on the plurality of photo clusters tagged and the plurality of media photo clusters tagged; and

displaying the personalized photo album on the mobile computing device, wherein the personalized photo album is browseable and navigable based on a tagged structure associated with the personalized photo album.

2. The method of claim 1, wherein the mobile computing device comprises at least one of:

a cellular phone;

a smartphone;

a personal digital assistant (PDA);

a tablet;

a laptop computer;

a portable gaming device;

a portable media player;

a google glass; and

a wearable electronic device,

wherein the mobile computing device comprises;

access to the Internet, one or more cameras, and a storage device configured to store a plurality of photos taken by the one or more cameras.

3. The method of claim 1, wherein the plurality of image features of a photo further comprises at least one of:

a size of the photo;

one or more tagged objects in the photo;

one or more tagged people in the photo; and

a plurality of weights of one or more layers of a convolution neural networks,

wherein the image features comprise a multi-dimensional vector space, and a similarity of two photos is measured in Euclidean distance of corresponding vector spaces of the two photos.

4. The method of claim 1, wherein the generating photo clusters comprises:

grouping one or more photos from the plurality of photos according to the date, time and location of the one or more photos; and

grouping one or more media photos from the plurality of media photos according to the date, time and location of the one or more media photos.

5. The method of claim 4, wherein the matching comprises:

calculating an Euclidean distance of a photo cluster and a media photo cluster; and

determining a match between the photo cluster and the media photo cluster when the Euclidean distance of the photo cluster and the media photo cluster is less than a predetermined threshold.

6. The method of claim 1 further comprising generating photo caption, wherein a photo caption of a media photo comprises:

one or more lines of text associated with the media photo; and

one or more lines of text manually entered by the user.

7. The method of claim 1, wherein the personalized photo album comprises:

a hierarchical structure in photo clusters; and

a user interface having text and keywords search capability.

8. A mobile computing device comprising a memory storing computer executable instructions for the mobile computing device, and a processor for executing the computer executable instructions, the computer executable instructions including:

accessing a plurality of photos stored on the mobile computing device, wherein the plurality of photos were taken by a user of the mobile computing device;

retrieving a plurality of media photos from one or more social media streams for the user, wherein the plurality of media photos were posted by the user of the mobile computing device;

extracting a plurality of image features from the plurality of photos and the plurality of media photos, and a plurality of text features from the plurality of media photos, wherein the image features of the plurality of photos and image features of the plurality of media photos each include a time when each photo was taken and a location where each photo was taken, and wherein the text features comprise a name of each tagged person in the plurality of photos and a name of each tagged object in the plurality of photos;

generating a plurality of photo clusters based on the plurality of image features of the plurality of photos, and a plurality of media photo clusters based on the plurality of image features of the plurality of media photos, respectively;

matching the plurality of photo clusters and the plurality of media photo clusters;

tagging the plurality of photo clusters with the plurality of text features extracted from the plurality of media photos;

generating a personalized photo album on the mobile computing device based on the plurality of photo clusters tagged and the plurality of media photo clusters tagged; and

displaying the personalized photo album on the mobile computing device, wherein the personalized photo album is browseable and navigable based on a tagged structure associated with the personalized photo album.

9. The mobile computing device of claim 8, wherein the mobile computing device comprises at least one of:

a cellular phone;

a smartphone;

a personal digital assistant (PDA);

a tablet;

a laptop computer;

a portable gaming device;

a portable media player;

a google glass; and

a wearable electronic device,

wherein the mobile computing device comprises;

access to the Internet, one or more cameras, and a storage device configured to store a plurality of photos taken by the one or more cameras.

10. The mobile computing device of claim 8, wherein the plurality of image features of a photo further comprises at least one of:

a size of the photo;

one or more tagged objects in the photo;

one or more tagged people in the photo; and

a plurality of weights of one or more layers of a convolution neural networks,

wherein the image features comprise a multi-dimensional vector space, and a similarity of two photos is measured in Euclidean distance of corresponding vector spaces of the two photos.

11. The mobile computing device of claim 8, wherein the generating photo clusters comprises:

grouping one or more photos from the plurality of photos according to the date, time and location of the one or more photos; and

grouping one or more media photos from the plurality of media photos according to the date, time and location of the one or more media photos.

12. The mobile computing device of claim 11, wherein the matching comprises:

calculating an Euclidean distance of a photo cluster and a media photo cluster; and

determining a match between the photo cluster and the media photo cluster when the Euclidean distance of the photo cluster and the media photo cluster is less than a predetermined threshold.

13. A non-transitory computer program product operable on a mobile computing device for creating a personalized photo album on the mobile computing device based on social media data comprising a computer readable storage medium having computer executable instructions embodied therewith, the computer executable instructions executable by a processor located on the mobile computing device to cause the processor to perform:

accessing a plurality of photos stored on the mobile computing device, wherein the plurality of photos were taken by a user of the mobile computing device;

retrieving a plurality of media photos from one or more social media streams for the user, wherein the plurality of media photos were posted by the user of the mobile computing device;

extracting a plurality of image features from the plurality of photos and the plurality of media photos, and a plurality of text features from the plurality of media photos, wherein the image features of the plurality of photos and image features of the plurality of media photos each include a time when each photo was taken and a location where each photo was taken, and wherein the text features comprise a name of each tagged person in the plurality of photos and a name of each tagged object in the plurality of photos;

generating a plurality of photo clusters based on the plurality of image features of the plurality of photos, and a plurality of media photo clusters based on the plurality of image features of the plurality of media photos, respectively;

matching the plurality of photo clusters and the plurality of media photo clusters;

tagging the plurality of photo clusters with the plurality of text features extracted from the plurality of media photos; and

generating a personalized photo album on the mobile computing device based on the plurality of photo clusters tagged and the plurality of media photo clusters tagged; and

displaying the personalized photo album on the mobile computing device, wherein the personalized photo album is browseable and navigable based on a tagged structure associated with the personalized photo album.

14. The non-transitory computer program product of claim 13, wherein the mobile computing device comprises at least one of:

a cellular phone;

a smartphone;

a personal digital assistant (PDA);

a tablet;

a laptop computer;

a portable gaming device;

a portable media player;

a google glass; and

a wearable electronic device,

wherein the mobile computing device comprises;

access to the Internet, one or more cameras, and a storage device configured to store a plurality of photos taken by the one or more cameras.

15. The non-transitory computer program product of claim 13, wherein the plurality of image features further comprises at least one of:

a size of the photo;

one or more tagged objects in the photo;

one or more tagged people in the photo; and

a plurality of weights of one or more layers of a convolution neural networks,

wherein the image features comprise a multi-dimensional vector space, and a similarity of two photos is measured in Euclidean distance of corresponding vector spaces of the two photos.

16. The non-transitory computer program product of claim 13, wherein the generating photo clusters comprises:

grouping one or more photos from the plurality of photos according to the date, time and location of the one or more photos; and

grouping one or more media photos from the plurality of media photos according to the date, time and location of the one or more media photos.

17. The non-transitory computer program product of claim 16, wherein the matching comprises:

calculating an Euclidean distance of a photo cluster and a media photo cluster; and

determining a match between the photo cluster and the media photo cluster when the Euclidean distance of the photo cluster and the media photo cluster is less than a predetermined threshold.