Automatic event recognition and cross-user photo clustering

• US 10,140,552 B2
• Filed: 01/23/2014
• Issued: 11/27/2018
• Est. Priority Date: 02/18/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Steiner, Matthew S.
• Primary Examiner(s): Yen, Syling
• Assistant Examiner(s): Tran, Bao
• Application Number:
  US14/162,708
• Publication Number:
  US 20140153837A1
• Time in Patent Office: 1,769 Days
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for automatic event recognition and photo clustering. In one aspect, methods include receiving, from a first user, first image data corresponding to a first image, receiving, from a second user, second image data corresponding to a second image, comparing the first image data and the second image data, and determining that the first image and the second image correspond to a coincident event based on the comparing.

Claims

1. A system comprising:

a data processing apparatus including at least one programmable processor; and

a non-transitory computer storage device encoded with a computer program, the computer program comprising instructions that when executed by the data processing apparatus cause the data processing apparatus to perform operations comprising;

receiving, from a first user device of a first user, first check-in data of the first user device checking in to a first location, wherein a first image is captured by the first user device while the first user device is near the first location;

receiving, from a second user device of a second user, second check-in data of the second user device checking in to a second location, wherein a second image is captured by the second user device while the second user device is near the second location;

determining the first location specified by the first check-in data;

determining the second location specified by the second check-in data;

comparing the first location and the second location to determine a geographical distance between the first location and the second location;

generating a distance-related confidence score based, at least in part, on determining the geographical distance meets a threshold distance; and

determining that the first image and the second image each correspond to a same event, based on a total confidence score meeting a threshold total confidence score, wherein the total confidence score is based on individual confidence scores of each of two or more different types of elements including the distance-related confidence score of the geographical distance.

2. The system of claim 1, wherein the operations further comprise adding the first image and the second image to an event web page.

3. The system of claim 1, wherein the-second user associated with the second user device is a member of a social network of the first user associated with the first user device, wherein the first user is different from the second user.

4. The system of claim 1, wherein the first check-in data comprises a first timestamp and the second check-in data comprises a second timestamp.

5. The system of claim 4, wherein the operations further comprise generating a time-related confidence score based, at least in part, on determining that a difference between the first timestamp and the second timestamp is less than a threshold difference, and wherein the total confidence score is further based on the time-related confidence score.

6. The system of claim 1, wherein the two or more different types of elements include at least one of a same or similar time, a same landmark or a same person, and wherein the operations further comprise generating the respective individual confidence score for each of the two or more different types of elements.

7. A method for automatic event recognition and cross-user photo clustering, comprising the steps of:

receiving, from a first user device of a first user, first check-in data of the first user device checking in to a first location, wherein a first image is captured by the first user device while the first user device is near the first location;

receiving, from a second user device of a second user, second check-in data of the second user device checking in to a second location, wherein a second image is captured by the second user device while the second user device is near the second location;

determining a first location specified by the first check-in data;

determining a second location specified by the second check-in data;

comparing the first location and the second location to determine a geographical distance between the first location and the second location;

generating a distance-related confidence score based, at least in part, on determining the geographical distance meets a threshold distance; and

determining that the first image and the second image each correspond to a same event based on a total confidence score meeting a threshold total confidence score, wherein the total confidence score is based, at least in part, on how many of two or more different types of elements each have a respective individual confidence score, wherein the two or more different types of elements include the distance-related confidence score of the geographical distance.

8. The method of claim 7, further comprising adding the first image and the second image to an event web page.

9. The method of claim 7, wherein the second user associated with the second user device is a member of a social network of the first user associated with the first user device, wherein the first user is different from the second user.

10. The method of claim 7, wherein the first check-in data comprises a first timestamp and the second check-in data comprises a second timestamp.

11. The method of claim 10, further including generating a time-related confidence score based, at least in part, on determining a difference between the first timestamp and the second timestamp is less than a threshold difference, and wherein the total confidence score is further based on the time-related confidence score.

12. The method of claim 7, wherein the two or more different types of elements include at least one of a same or similar time, a same landmark or a same person, and the method further comprises generating the respective individual confidence score for each of the two or more different types of elements.

13. The method of claim 7, wherein the total confidence score is higher where more of the two or more different types of elements have respective individual confidence scores that meet a respective threshold.

14. A non-transitory computer readable storage medium having computer program logic recorded thereon for automatic event recognition and cross-user photo clustering, comprising:

a receiving module configured to receive from a first user device of a first user, first check-in data of the first user device checking in to a first location, wherein a first image is captured by the first user device while the first user device is near the first location, and for receiving, from a second user device of a second user, second check-in data of the second user device checking in to a second location, wherein a second image is captured by the second user device while the second user device is near the second location;

a first determining module configured to determine a first location specified by the first check-in data, a second location specified by the second check-in data, and a geographical distance between the first location and the second location;

a first scoring module configured to generate a distance-related confidence score, at least in part, on determining the geographical distance meets a threshold distance; and

a second determining module configured to determine that the first image and the second image each correspond to a same event based on a total confidence score meeting a threshold total confidence score, wherein the total confidence score is based, at least in part, on how many of two or more different types of elements each have a respective individual confidence score, wherein the two or more different types of elements include the distance-related confidence score of the geographical distance.

15. The computer readable storage medium of claim 14, further comprising a module for adding the first image and the second image to an event web page.

16. The computer readable storage medium of claim 14, wherein the first check-in data comprises a first timestamp and the second check-in data comprises a second timestamp.

17. The computer readable storage medium of claim 16, further including a second scoring module configured to generate a time-related confidence score based, at least in part, on determining a difference between the first timestamp and the second timestamp is less than a threshold difference, and wherein the total confidence score is further based on the time-related confidence score.

18. The computer readable storage medium of claim 14, wherein the two or more different types of elements include at least one of a similar or same time, a same landmark or a same person, and

wherein the first scoring module is further configured to generate the respective individual confidence score for each of the two or more different types of elements.

19. The computer readable storage medium of claim 14, wherein the total confidence score is higher where more of the two or more different types of elements have respective individual confidence scores that meet a respective threshold.