Techniques for user customization in a photo management system

• US 9,626,384 B2
• Filed: 11/18/2014
• Issued: 04/18/2017
• Est. Priority Date: 09/27/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Carceroni, Rodrigo, Sanketi, Pannag Raghunath, Subramanya, Amarnag
• Primary Examiner(s): AGGARWAL, YOGESH K
• Application Number:
  US14/547,078
• Publication Number:
  US 20150074574A1
• Time in Patent Office: 882 Days
• Field of Search: 34820812, 34820814, 705 1468, 705 1469
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet

Abstract

A computer-implemented technique can receive a plurality of photos and automatically select a subset of the plurality of photos having a high degree of representativeness by jointly maximizing both photo quality and photo diversity to obtain a photo album. The technique can determine one or more clusters for the photo album using a hierarchical clustering algorithm, and store the photo album according to the one or more clusters. The technique can control the manner in which the photo album is displayed using the one or more clusters. The technique can adjust at least one of the one or more clusters and the automatic photo album generation based on user input. The user input can include at least one of adding, deleting, and moving a photo with respect to the one or more clusters. The technique can then re-cluster, automatically generate a new photo album, and/or adjust the presentation.

Claims

1. A computer-implemented method comprising:

receiving, by a computing device including one or more processors, a plurality of photos;

selecting, by the computing device, a first subset of the plurality of photos based on maximizing photo quality and photo diversity using a quality metric for each of the plurality of photos and a similarity matrix for the plurality of photos, the quality metric for a photo being indicative of a quality of such photo, the similarity matrix being indicative of similarity between each pair of photos of the plurality of photos, wherein the first subset of photos is selected using joint global maximization

determining, by the computing device, a first set of one or more clusters for the first subset of the plurality of photos using hierarchical clustering, the first set of clusters including less than or equal to all of the first subset of the plurality of photos, each of the one or more clusters in the first set of one or more clusters including one or more distinct photos from the first subset of the plurality of photos;

storing, by the computing device, the photos arranged by the first set of one or more clusters;

receiving, by the computing device, user input with respect to the first subset of the plurality of photos, the user input corresponding to at least one of adding a first specific photo from a remainder of the plurality of photos to one of the one or more clusters in the first set of one or more clusters, and deleting a second specific photo from one of the one or more clusters in the first set of one or more clusters;

adjusting, at the computing device, at least one of the quality metric and the similarity matrix based on the user input;

selecting, by the computing device, a second subset of the plurality of photos based on maximizing photo quality and photo diversity using the adjusted at least one of the quality metric and the similarity matrix;

determining, by the computing device, a second set of one or more clusters for the second subset of the plurality of photos using hierarchical clustering; and

storing, at the computing device, second subset of the plurality of photos arranged in the second set of one or more clusters.

2. A computer-implemented method comprising:

receiving, by a computing device including one or more processors, a plurality of photos;

selecting, by the computing device, a first subset of the plurality of photos based on maximizing photo quality and photo diversity using a quality metric for each of the plurality of photos and a similarity matrix for the plurality of photos, the quality metric for a photo being indicative of a quality of such photo, the similarity matrix being indicative of similarity between each pair of photos of the plurality of photos, wherein the first subset of photos is selected using joint global maximization

determining, by the computing device, a first set of one or more clusters for the first subset of the plurality of photos using hierarchical clustering, the first set of clusters including less than or equal to all of the first subset of the plurality of photos, each of the one or more clusters in the first set of one or more clusters including one or more distinct photos from the first subset of the plurality of photos; and

storing, by the computing device, the photos arranged by the first set of one or more clusters.

3. The computer-implemented method of claim 2, further comprising:

receiving, by the computing device, user input with respect to the first subset of the plurality of photos, the user input corresponding to at least one of adding a first specific photo from a remainder of the plurality of photos to one of the one or more clusters in the first set of one or more clusters, and deleting a second specific photo from one of the one or more clusters in the first set of one or more clusters.

4. The computer-implemented method of claim 3, further comprising:

adjusting, at the computing device, at least one of the quality metric and the similarity matrix based on the user input.

5. The computer-implemented method of claim 4, further comprising:

selecting, by the computing device, a second subset of the plurality of photos based on maximizing photo quality and photo diversity using the adjusted at least one of the quality metric and the similarity matrix.

6. The computer-implemented method of claim 5, further comprising:

determining, by the computing device, a second set of one or more clusters for the second subset of the plurality of photos using hierarchical clustering.

7. The computer-implemented method of claim 6, further comprising:

storing, at the computing device, second subset of the plurality of photos arranged in the second set of one or more clusters.

8. The computer-implemented method of claim 2, wherein the quality metric and the similarity matrix are each determined based an analysis of a reference photo collection that includes a plurality of reference photos, a quality weight for each reference photo, and a similarity weight for each unique pair of reference photos.

9. The computer-implemented method of claim 2, wherein the user input further corresponds to moving the second specific photo from the specific cluster to another one of the plurality of clusters.

10. The computer-implemented method of claim 2, wherein selecting the first subset of photos is selected by performing the joint global maximization of photo quality and photo diversity using the quality metrics and the similarity matrix including using a determinantal point process (DPP).

11. The computer-implemented method of claim 2, wherein selecting the subset of the plurality of photos further includes:

extracting, at the computing device, a set of quality features for each of the plurality of photos, wherein the set of quality features for a specific photo includes at least one of photometric features, saliency-based features, and content-based features for the specific photo.

12. A computing device comprising:

an input/output device configured to receive a plurality of photos;

one or more processors configured to;

receive a plurality of photos;

select a first subset of the plurality of photos based on maximizing photo quality and photo diversity using a quality metric for each of the plurality of photos and a similarity matrix for the plurality of photos, the quality metric for a photo being indicative of a quality of such photo, the similarity matrix being indicative of similarity between each pair of photos of the plurality of to photos, wherein the first subset of photos is selected using joint global maximization

determine a first set of one or more clusters for the first subset of the plurality of photos using hierarchical clustering, the first set of clusters including less than or equal to all of the first subset of the plurality of photos, each of the one or more clusters in the first set of one or more clusters including one or more distinct photos from the first subset of the plurality of photos; and

store the photos arranged by the first set of one or more clusters.

13. The computing device of claim 12, further configured to:

receive user input with respect to the first subset of the plurality of photos, the user input corresponding to at least one of adding a first specific photo from a remainder of the plurality of photos to one of the one or more clusters in the first set of one or more clusters, and deleting a second specific photo from one of the one or more clusters in the first set of one or more clusters.

14. The computing device of claim 13, further configured to:

adjust at least one of the quality metric and the similarity matrix based on the user input; and

select, by the computing device, a second subset of the plurality of photos based on maximizing photo quality and photo diversity using the adjusted at least one of the quality metric and the similarity matrix.

15. The computing device of claim 14, further configured to:

determine a second set of one or more clusters for the second subset of the plurality of photos using hierarchical clustering; and

store the second subset of the plurality of photos arranged in the second set of one or more clusters.

16. The computing device of claim 12, wherein the quality metric and the similarity matrix are each determined based an analysis of a reference photo collection that includes a plurality of reference photos, a quality weight for each reference photo, and a similarity weight for each unique pair of reference photos.

17. The computing device of claim 13, wherein the user input further corresponds to moving the second specific photo from the specific cluster to another one of the plurality of clusters.

18. The computing device of claim 12, wherein the first subset of photos is selected by performing joint global maximization of photo quality and photo diversity using the quality metrics and the similarity matrix including using a determinantal point process (DPP).

19. The computing device of claim 12, wherein selecting the subset of the plurality of photos further includes extracting, at the computing device, a set of quality features for each of the plurality of photos, wherein the set of quality features for a specific photo includes at least one of photometric features, saliency-based features, and content-based features for the specific photo.