Computer-vision content detection for connecting objects in media to users

• US 9,672,496 B2
• Filed: 08/18/2011
• Issued: 06/06/2017
• Est. Priority Date: 08/18/2011
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Odio, Samuel, Mitchell, Justin, Garcia, David Harry
• Primary Examiner(s): Nguyen, Quang N
• Application Number:
  US13/212,344
• Publication Number:
  US 20130046823A1
• Time in Patent Office: 2,119 Days
• Field of Search: None
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 50/01 Social networking

Abstract

Methods, apparatuses and systems directed to detecting objects in user-uploaded multimedia such as photos and videos, determining the location at which the media was captured, inferring a set of users of a social network who were physically present at the time and place of capture, and pushing remarketing content to the set of inferred users for the detected objects, or alternatively, the competitors of the detected concepts.

Claims

1. A method comprising:

receiving, by one or more computer systems, a multimedia object from a first user of an online social network, wherein the first user is represented by a first user node of a social graph associated with the online social network;

detecting, by one or more computer systems, one or more concepts associated with the multimedia object, the one or more concepts being represented by one or more concept nodes of the social graph, respectively; and

identifying, by one or more computer systems, one or more second users of the online social network as being exposed to the one or more concepts associated with the multimedia object, each second user being identified based at least in part on one or more social actions associated with the second user with respect to the multimedia object, wherein the one or more second users are represented by one or more second user nodes of the social graph, respectively, each second user node being within a specified degree of separation of the first user node.

2. The method of claim 1, wherein identifying the second users who were exposed to the concept, comprises:

determining, by one or more computer systems, a physical location that is associated with the multimedia object;

determining, by one or more computer systems, a time period during which the multimedia object was captured;

identifying the social actions, wherein the social actions are identified based at least in part on a calculated probability that the social actions occurred both at the physical location and within the time period; and

identifying, by one or more computer systems, the second users whose respective nodes are associated with the identified social actions.

3. The method of claim 2, wherein determining the physical location comprises querying the social network for one or more locations explicitly associated with the multimedia object.

4. The method of claim 2, wherein determining a physical location comprises:

applying, by one or more computer systems, one or more object image recognition algorithms to the multimedia object;

comparing, by one or more computer systems, at least a portion of the multimedia object to a plurality of stored images in a database, each of the plurality of stored images associated with one or more physical locations; and

based on the comparison, by one or more computer systems, determining the physical location.

5. The method of claim 1, further comprising pushing content associated with one or more of the detected concepts to one or more of the second users.

6. The method claim 5, wherein:

each of the concepts are associated one or more concept nodes on the social network; and

the content pushed to the second users is associated with a competitor of an owner or administrator of one or more of the concept nodes.

7. The method of claim 1, wherein the social actions comprise one or more of:

a check-in to a location;

an RSVP for an event;

or

a tag in one or more other multimedia objects associated with an event.

8. The method of claim 1, wherein detecting one or more concepts associated with the multimedia object comprises:

applying, by one or more computer systems, one or more image object recognition algorithms to the multimedia object;

comparing, by one or more computer systems, at least a portion of the multimedia object to a plurality of stored objects in a database, each of the plurality of stored objects associated with one or more concepts;

based on the comparison, identifying, by one or more computer systems, one or more concepts associated with the multimedia object; and

attaching, by one or more computer systems, meta-data identifying one or more of the concepts to the multimedia object.

9. The method of claim 8, wherein:

the multimedia object comprises an image; and

the plurality of stored objects comprises a plurality of stored images.

10. The method of claim 8, wherein the multimedia object comprises a video and comparing at least a portion of the multimedia object comprises:

extracting, by one or more computer systems, one or more frames from the video; and

comparing, by one or more computer systems, one or more of the extracted frames to the plurality of stored objects, wherein the plurality of stored objects comprises a plurality of stored images.

11. The method of claim 1, wherein the multimedia object comprises a video and detecting one or more concepts comprises:

extracting, by one or more computer systems, one or more audio waveforms from the video; and

comparing, by one or more computer systems, one or more of the extracted waveforms to a plurality of stored objects in a database, each of the plurality of stored objects associated with one or more concepts, wherein the plurality of stored objects comprises a plurality of stored audio waveforms;

based on the comparison, by one or more computer systems, identifying one or more concepts associated with the multimedia object; and

attaching, by one or more computer systems, meta-data identifying one or more of the concepts to the multimedia object.

12. The method of claim 1, wherein each concept node corresponds to a brand, product, or company.

13. The method of claim 1, further comprising:

accessing the social graph, wherein the social graph comprises a plurality of nodes and a plurality of edges connecting the nodes, each connection between two nodes establishing a single degree of separation between the two nodes, wherein the plurality of nodes comprises;

the first user node;

a plurality of second user nodes representing a plurality of second users, respectively; and

a plurality of concept nodes representing a plurality of concept nodes, respectively.

14. The method of claim 1, wherein the specified degree of separation is one, two, three, or all.

15. One or more non-transitory computer-readable storage media embodying software that is operable when executed to:

receive a multimedia object from a first user of an online social network, wherein the first user is represented by a first user node of a social graph associated with the online social network;

detect one or more concepts associated with the multimedia object, the one or more concepts being represented by one or more concept nodes of the social graph, respectively; and

identify one or more second users of the online social network as being exposed to the one or more concepts associated with the multimedia object, each second user being identified based at least in part on one or more social actions associated with the second user with respect to the multimedia object, wherein the one or more second users are represented by one or more second user nodes of the social graph, respectively, each second user node being within a specified degree of separation of the first user node.

16. The media of claim 15, wherein the software is further operable when identifying the second users who were exposed to the concept, to:

determine a physical location that is associated with the multimedia object;

determine a time period during which the multimedia object was captured;

identify the social actions, wherein the social actions are identified based at least in part on a calculated probability that the social actions occurred both at the physical location and within the time period; and

identify the second users whose respective nodes are associated with the identified social actions.

17. The media of claim 15, wherein the instructions are further operable when executed to push content associated with one or more of the detected concepts to one or more of the second users.

18. The media of claim 17, wherein:

each of the concepts are associated with one or more concept nodes on the social network; and

the content pushed to the second users is associated with a competitor of an owner or administrator of one or more of the concept nodes.

19. The media of claim 15, wherein the social actions comprise one or more of:

a check-in to a location;

an RSVP for an event;

or

a tag in one or more other multimedia objects associated with an event.

20. A system comprising:

one or more processors; and

a memory coupled to the processors comprising instructions executable by the processors, the processors being operable when executing the instructions to;

receive a multimedia object from a first user of an online social network, wherein the first user is represented by a first user node of a social graph associated with the online social network;

detect one or more concepts associated with the multimedia object, the one or more concepts being represented by one or more concept nodes of the social graph, respectively; and

identify one or more second users of the online social network as being exposed to the one or more concepts associated with the multimedia object, each second user being identified based at least in part on one or more social actions associated with the second user with respect to the multimedia object, wherein the one or more second users are represented by one or more second user nodes of the social graph, respectively, each second user node being within a specified degree of separation of the first user node.