Method and system for providing context relevant media augmentation

• US 10,354,425 B2
• Filed: 12/18/2015
• Issued: 07/16/2019
• Est. Priority Date: 12/18/2015
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Yan, Rong, Sellis, Peter, Cooper, Andrew G, Zheng, Yantao
• Primary Examiner(s): Debrow, James J
• Application Number:
  US14/974,321
• Publication Number:
  US 20170263029A1
• Time in Patent Office: 1,306 Days
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859

Abstract

Systems and methods are presented that provide for receiving, at a server computer, a content item from a client device, determining, by the server computer, that a characteristic of the content item matches a trigger for a media overlay, generating, by the server computer, a media overlay associated with the characteristic of the content item, and providing, by the server computer, the media overlay to the client device wherein the media overlay is displayed on a user interface of the client device.

Claims

1. A server comprising:

one or more hardware processors; and

a non-transitory computer-readable storage medium storing a set of instructions that, when executed by the one or more hardware processors, cause the server to perform operations comprising;

receiving, from a client device, a content item comprising an image or video captured by the client device;

detecting audio content playing in the background of the video or during capture of the image or video;

matching a fingerprint of the audio content to a known audio fingerprint of an audio track;

determining that the audio track matches a trigger for a media overlay;

generating the media overlay comprising information associated with the audio track based on the detected audio content playing in the background of the video or during capture of the image or video;

providing the media overlay comprising the information associated with the audio track to the client device wherein the media overlay is selected by the client device to augment the content item; and

receiving, from the client device, a message comprising the content item augmented with the media overlay comprising information associated with the audio track; and

sending the message to at least a second client device.

2. The server of claim 1, wherein the audio content is one of a plurality of characteristics of the content item, and wherein the plurality of characteristics of the content item further include at least one of a group comprising:

geolocation, audio, speed, timestamp, data associated with a machine-readable code, a particular object in the content item, and a recipient for the content item.

3. The server of claim 1, wherein the media overlay is a first media overlay, and wherein before sending the message to the second client device, the operations comprise:

generating a second media overlay;

updating the message to augment the content item with the second media overlay; and

sending the content item augmented with the first media overlay and the second media overlay to the second client device.

4. The server of claim 1, wherein information associated with the audio track comprises at least one of a title associated with the audio track, an artist name associated with the audio track, an album name associated with the audio track, and album artwork associated with the audio track.

5. The server of claim 1, wherein the operations further comprise:

receiving a selected geolocation

generating a user-based media overlay based on the content item and the selected geolocation,

supplying the client device with the user-based media overlay in response to a geolocation of the client device being within the selected geolocation.

6. The server of claim 5, wherein the operations further comprise:

receiving an identification of a period of time associated with the content item and the selected geolocation; and

supplying the client device with the user-based media overlay within the selected geolocation during the period of time.

7. The server of claim 1, wherein the operations further comprise:

receiving a first content item from a first merchant and a second content item from a second merchant;

receiving a first geolocation information from the first merchant, and a second geolocation information from the second merchant, and identifying a common geolocation based on the first geolocation information and the second geolocation information;

receiving a first bid amount from the first merchant and a second bid amount from the second merchant, and identifying a highest bid amount;

generating a merchant-based media overlay based on the content item of the merchant with the highest bid amount and the common geolocation; and

supplying the merchant-based media overlay to the client device in response to a geolocation of the client device being within the common geolocation.

8. The server of claim 7, wherein the operations further comprise:

disabling the merchant-based media overlay after a predetermined duration has elapsed.

9. The server of claim 7, wherein the common geolocation includes a common region formed between a first geolocation from the first merchant and a second geolocation from the second merchant.

10. The server of claim 1, wherein the operations further comprise:

identifying a live event associated with a geolocation of the client device;

accessing live event data related to the live event; and

generating a live event media overlay based on the live event data and the geolocation of the client device.

11. The server of claim 1, wherein the operations further comprise:

accessing social network data based on social network information from the client device; and

generating a social network media overlay based on the social network data and the social network data from the client device.

12. The server of claim 1, wherein the operations further comprise:

generating a set of media overlays including the media overlay for a merchant for a predefined geolocation of the merchant;

randomly selecting one media overlay from the set of media overlays; and

providing the randomly selected media overlay to the client device in response to a geolocation of the client device corresponding to the predefined geolocation of the merchant.

13. The server of claim 1, wherein the operations further comprise:

generating a progressive use media overlay for a predefined geolocation; and

adjusting a content item of the progressive use media overlay in response to a number of prior uses of the progressive use media overlay.

14. The server of claim 13, wherein the operations further comprise:

disabling the progressive use media overlay after the number of prior uses of the progressive use media overlay reaches a predefined progressive use limit.

15. The server of claim 1, wherein the operations further comprise:

generating a viral use media overlay for a predefined geolocation;

providing the viral use media overlay to a first client device located at the predefined geolocation;

receiving a request from the first client device located at the predefined geolocation to provide the viral use media overlay to a second client device located outside the predefined geolocation; and

providing the viral use media overlay to the second client device located outside the predefined geolocation.

16. The server of claim 1, wherein the audio track is associated with a live event and where in the media overlay further comprises information about the live event.

17. The server of claim 1, wherein the audio content is ambient noise or environmental acoustic sound.

18. The server of claim 1, the operations further comprising:

receiving, from the client device, a portion of audio related to an image or video being captured by the client device;

determining whether the portion of audio matches a trigger for a third media overlay;

generating the third media overlay based on the portion of the audio matching the trigger for the third media overlay; and

providing the third media overlay to the client device.

19. A method comprising:

receiving, at a server computer from a client device, a content item comprising an image or video captured by the client device;

detecting, by the server computer, audio content playing in the background of the video or during capture of the image or video;

matching, by the server computer, a fingerprint of the audio content to a known audio fingerprint of an audio track;

determining, by the server computer, that the audio track matches a trigger for a media overlay;

generating, by the server computer, the media overlay comprising information associated with the audio track based on the detected audio content playing in the background of the video or during capture of the image or video;

providing, by the server computer, the media overlay comprising the information associated with the audio track to the client device, wherein the media overlay is selected by the client device to augment the content item; and

receiving, from the client device, a message comprising the content item augmented with the media overlay comprising information associated with the audio track; and

sending the message to at least a second client device.

20. A non-transitory computer-readable storage medium storing a set of instructions that, when executed by a processor of a machine, cause the machine to perform operations comprising:

receiving, from a client device, a content item comprising an image or video captured by the client device;

detecting audio content playing in the background of the video or during capture of the image or video;

matching a fingerprint of the audio content to a known audio fingerprint of an audio track;

determining that the audio track matches a trigger for a media overlay;

generating the media overlay comprising information associated with the audio track based on the detected audio content playing in the background of the video or during capture of the image or video;

providing the media overlay comprising the information associated with the audio track to the client device wherein the media overlay is selected by the client device to augment the content item; and

receiving, from the client device, a message comprising the content item augmented with the media overlay comprising information associated with the audio track; and

sending the message to at least a second client device.