Computer-vision content detection for connecting objects in media to users

• US 10,346,929 B2
• Filed: 04/14/2017
• Issued: 07/09/2019
• Est. Priority Date: 08/18/2011
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Mitchell, Justin, Odio, Samuel, Garcia, David Harry
• Primary Examiner(s): Nguyen, Quang N
• Application Number:
  US15/488,316
• Publication Number:
  US 20170221158A1
• Time in Patent Office: 816 Days
• Field of Search: 709204, 709206, 709219, 709203, 709217, 709223, 709224, 709226, 709228, 709230, 709232, 709238, 707709, 715206, 725 32, 382218
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

Methods, apparatuses and systems directed to detecting objects in user-uploaded multimedia such as photos and videos, determining the location at which the media was captured, inferring a set of users of a social network who were physically present at the time and place of capture, and pushing remarketing content to the set of inferred users for the detected objects, or alternatively, the competitors of the detected concepts.

Claims

1. A method comprising, by one or more computer systems of an online social network:

accessing, by one or more computer systems, a multimedia object received from a first user of the online social network;

detecting, by one or more computer systems, one or more concepts associated with the multimedia object, wherein each detected concept corresponds to a particular entity of the online social network;

identifying, by one or more computer systems, one or more second users who have been exposed to the detected one or more concepts based on whether the one or more second users have interacted with the multimedia object on the online social network, wherein the first user is within a threshold degree of separation from each of the one or more second users of the online social network;

recording, by one or more computer systems, an inferred exposure to the detected one or more concepts by the identified second users based on their interactions with the multimedia object; and

pushing, by one or more computing systems, content associated with one or more of the detected concepts to one or more client systems of the one or more second users;

wherein the content pushed comprises advertisements associated with the particular entity that corresponds to a detected concept of the detected one or more concepts.

2. The method of claim 1, wherein identifying the second users comprises:

determining, by one or more computer systems, a physical location that is associated with the multimedia object;

determining, by one or more computer systems, a time period during which the multimedia object was captured;

identifying the interactions with the multimedia object, wherein the interactions with the multimedia object are identified based at least in part on a calculated probability that the interactions with the multimedia object occurred both at the physical location and within the time period; and

identifying, by one or more computer systems, the second users associated with the identified interactions.

3. The method of claim 2, wherein:

the first user is associated with an online social network; and

determining the physical location comprises querying the social network for one or more locations associated with the multimedia object.

4. The method of claim 2, wherein determining a physical location comprises:

applying, by one or more computer systems, one or more object image recognition algorithms to the multimedia object;

comparing, by one or more computer systems, at least a portion of the multimedia object to a plurality of stored images in a database, each of the plurality of stored images associated with one or more physical locations; and

based on the comparison, by one or more computer systems, determining the physical location.

5. The method of claim 1, wherein:

at least one of the concepts is associated with a concept node of a social network;

the method further comprises determining, for the at least one concept node, an entity associated with that concept node; and

the content pushed to the second users is associated with a competitor of the entity.

6. The method of claim 1, wherein the interactions with the multimedia object comprise one or more of:

a check-in to a location associated with the multimedia object;

an RSVP for an event associated with the multimedia object;

or

a tag in one or more other multimedia objects associated with the event.

7. The method of claim 1, wherein detecting one or more concepts associated with the multimedia object comprises:

applying, by one or more computer systems, one or more image object recognition algorithms to the multimedia object;

comparing, by one or more computer systems, at least a portion of the multimedia object to a plurality of stored objects in a database, each of the plurality of stored objects associated with one or more concepts;

based on the comparison, identifying, by one or more computer systems, one or more concepts associated with the multimedia object; and

attaching, by one or more computer systems, meta-data identifying one or more of the identified concepts to the multimedia object.

8. The method of claim 7, wherein:

the multimedia object comprises an image; and

the plurality of stored objects comprises a plurality of stored images.

9. The method of claim 7, wherein the multimedia object comprises a video and comparing at least a portion of the multimedia object comprises:

extracting, by one or more computer systems, one or more frames from the video; and

comparing, by one or more computer systems, one or more of the extracted frames to the plurality of stored objects, wherein the plurality of stored objects comprises a plurality of stored images.

10. The method of claim 1, wherein the multimedia object comprises a video and detecting one or more concepts comprises:

extracting, by one or more computer systems, one or more audio waveforms from the video; and

comparing, by one or more computer systems, one or more of the extracted waveforms to a plurality of stored objects in a database, each of the plurality of stored objects associated with one or more concepts, wherein the plurality of stored objects comprises a plurality of stored audio waveforms;

based on the comparison, by one or more computer systems, identifying one or more concepts associated with the multimedia object; and

attaching, by one or more computer systems, meta-data identifying one or more of the identified concepts to the multimedia object.

11. The method of claim 1, wherein:

at least one of the concepts is associated with a concept node of a social network;

the concept node corresponds to a brand, product, or company.

12. The method of claim 1, further comprising:

accessing a social graph associated with a social network, wherein the social graph comprises a plurality of nodes and a plurality of edges connecting the nodes, each connection between two nodes establishing a single degree of separation between the two nodes, wherein the plurality of nodes comprises;

a first node corresponding to the first user;

one or more second nodes corresponding to the one or more second users, respectively, each second node being within one degree of separation from the first node; and

one or more concept nodes corresponding to the one or more concepts, respectively.

13. The method of claim 12, wherein the specified degree of separation is one, two, three, or all.

14. One or more non-transitory computer-readable storage media embodying software that is operable when executed to:

access a multimedia object received from a first user of an online social network;

detect one or more concepts associated with the multimedia object, wherein each detected concept corresponds to a particular entity of the online social network;

identify one or more second users who have been exposed to the detected one or more concepts based on whether the one or more second users have interacted with the multimedia object on the online social network, wherein the first user is within a threshold degree of separation from each of the one or more second users of the online social network;

record an inferred exposure to the detected one or more concepts for the identified second users based on their interactions with the multimedia object; and

push content associated with one or more of the detected concepts to one or more client systems of the one or more second users;

wherein the content pushed comprises advertisements associated with the particular entity that corresponds to a detected concept of the detected one or more concepts.

15. The media of claim 14, wherein the software is further operable when identifying the second users, to:

determine a physical location that is associated with the multimedia object;

determine a time period during which the multimedia object was captured;

identify the interactions with the multimedia object, wherein the interactions with the multimedia object are identified based at least in part on a calculated probability that the interactions with the multimedia object occurred both at the physical location and within the time period; and

identify the second users associated with the identified interactions.

16. The media of claim 14, wherein:

at least one of the concepts is associated with a concept node of a social network;

the instructions are further operable when executed to determine, for the at least one concept node, an entity associated with that concept node; and

the content pushed to the second users is associated with a competitor of the entity.

17. The media of claim 14, wherein the interactions with the multimedia object comprise one or more of:

a check-in to a location associated with the multimedia object;

an RSVP for an event associated with the multimedia object;

or

a tag in one or more other multimedia objects associated with the event.

18. A system comprising:

one or more processors; and

a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to;

access a multimedia object received from a first user of an online social network;

detect one or more concepts associated with the multimedia object, wherein each detected concept corresponds to a particular entity of the online social network;

identify one or more second users who have been exposed to the detected one or more concepts based on whether the one or more second users have interacted with the multimedia object on the online social network, wherein the first user is within a threshold degree of separation from each of the one or more second users of the online social network;

record an inferred exposure to the detected one or more concepts for the identified second users based on their interactions with the multimedia object; and

push content associated with one or more of the detected concepts to one or more client systems of the one or more second users;

wherein the content pushed comprises advertisements associated with the particular entity that corresponds to a detected concept of the detected one or more concepts.