Computer-vision-assisted location check-in

• US 8,891,832 B2
• Filed: 06/03/2011
• Issued: 11/18/2014
• Est. Priority Date: 06/03/2011
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Tseng, Erick
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US13/152,576
• Publication Number:
  US 20120308077A1
• Time in Patent Office: 1,264 Days
• Field of Search: 382/118, 382/305, 382/321
• US Class Current: 382/118
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

In one embodiment, an uploaded multimedia object comprising a photo image or video is subjected to computer vision algorithms to detect and isolate objects within the multimedia object, and the isolated object is searched against a photographic location database containing images of a plurality of locations. Upon detecting a matching object, the location information associated with the photograph in the database containing the matching object may be leveraged to automatically check the user in to the associated location.

Claims

1. A method comprising, by one or more computer systems:

receiving, from a user, a multimedia object, the multimedia object comprising one or more images;

applying one or more object image recognition algorithms to the multimedia object;

comparing at least a portion of the multimedia object to a plurality of stored images in a database, each of the plurality of stored images associated with a physical location;

determining a location of the user based on the comparison; and

automatically associating the user with a concept node of a social graph representing a social network, the concept node being associated with the determined location.

2. The method of claim 1, further comprising automatically checking the user in to the determined location.

3. The method of claim 1, wherein the multimedia object is received in conjunction with a search request, and further comprising:

ranking, based at least in part on the determined location, a list of places generated in response to the search request; and

transmitting a response to the user including a ranked list of places.

4. The method of claim 3, wherein the search request comprises a user position, and the ranking based at least in part on the received user position.

5. The method of claim 1, wherein the multimedia object comprises an image.

6. The method of claim 1, wherein the multimedia object comprises a video, and wherein comparing at least a portion of the multimedia object comprises:

extracting one or more frames from the video; and

comparing the extracted one or more frames to the plurality of stored images in the database.

7. The method of claim 1, wherein the determined location comprises a longitude latitude coordinate pair.

8. The method of claim 1, wherein applying one or more object image recognition algorithms comprises:

applying optical character recognition to extract text from the multimedia object;

comparing the extracted text to a database of text entries, each of the plurality of entries associated with a location, and based on the comparison, narrowing the set of the plurality of stored images.

9. The method of claim 1, wherein the multimedia object is received from a mobile computing device with a camera.

10. The method of claim 9, wherein the mobile computing device lacks a global positioning system receiver.

11. The method of claim 1, further comprising automatically tagging an object associated with the determined location in the received multimedia object.

12. The method of claim 1, further comprising:

applying a facial-recognition algorithm to the received image, and, for each of one or more detected faces;

comparing, for each of the user'"'"'s first-degree connections, the detected face to one or more photographs containing the face of the user'"'"'s first degree connection.

13. The method of claim 12, further comprising:

for each of the user'"'"'s first-degree connections having one or more photographs matching one of the detected faces, automatically checking-in the detected first-degree connection to the determined location.

14. The method of claim 1, further comprising:

receiving a character string;

generating a list of the user'"'"'s first-degree connections with user names matching the character string;

applying a facial-recognition algorithm to the received image, and, for each of one or more detected faces;

comparing, for each of the user'"'"'s first-degree connections, the detected face to one or more photographs containing the face of the user'"'"'s first degree connection;

upon finding a match, increasing the rank of the detected first-degree connection; and

transmitting the ranked list to the user.

15. A non-transitory, computer-readable media containing instructions that, when executed, cause a processor to:

receive, from a user, a multimedia object, the multimedia object comprising one or more images;

apply one or more object image recognition algorithms to the multimedia object;

compare at least a portion of the multimedia object to a plurality of stored images in a database, each of the plurality of stored images associated with a physical location;

determine a location of the user based on the comparison; and

automatically associate the user with a concept node of a social graph representing a social network, the concept node being associated with the determined location.

16. The media of claim 15, further comprising instructions that, when executed, cause the processor to automatically check the user in to the determined location.

17. The media of claim 15, wherein the multimedia object comprises an image.

18. The media of claim 15, wherein applying one or more object image recognition algorithms comprises:

applying optical character recognition to extract text from the multimedia object;

comparing the extracted text to a database of text entries, each of the plurality of entries associated with a location, and

based on the comparison, narrowing the plurality of stored images.

19. The media of claim 18, further comprising instructions that, when executed, cause the processor to:

for each of the user'"'"'s first-degree connections having one or more photographs matching one of the detected faces, automatically check-in the detected first-degree connection to the determined location.

20. The media of claim 15, further comprising instructions that, when executed, cause the processor to:

receiving a character string;

generating a list of the user'"'"'s first-degree connections with user names matching the character string;

applying a facial-recognition algorithm to the received image, and, for each of one or more detected faces;

comparing, for each of the user'"'"'s first-degree connections, the detected face to one or more photographs containing the face of the user'"'"'s first degree connection;

upon finding a match, increasing the rank of the detected first-degree connection; and

transmitting the ranked list to the user.