Generating a stitched data stream

• US 10,581,782 B2
• Filed: 03/27/2017
• Issued: 03/03/2020
• Est. Priority Date: 03/27/2017
• Status: Active Grant

• Current Assignee: Snap, Inc.
• Original Assignee: Snap, Inc.
• Inventors: Tang, Kevin Dechau
• Primary Examiner(s): Blair, Douglas B
• Application Number:
  US15/470,004
• Publication Number:
  US 20180278562A1
• Time in Patent Office: 1,072 Days
• CPC Class Codes: H04L 51/10 Multimedia information
• CPC Class Codes: H04L 51/216 Handling conversation history, e.g. grouping of messages in sessions or threads
• CPC Class Codes: H04L 51/222 using geographical location information, e.g. messages transmitted or received in proximity of a certain spot or area
• CPC Class Codes: H04L 51/52 for supporting social networking services
• CPC Class Codes: H04L 63/107 wherein the security policies are location-dependent, e.g. entities privileges depend on current location or allowing specific operations only from locally connected terminals
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 21/2743 Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4524 involving the geographical location of the client retrieval from the Internet by querying based on geographical locations G06F16/9537; systems specially adapted for using geographical information in broadcast systems H04H60/70; protocols in which the network application is adapted for the location of the user terminal in communication control or processing H04L67/52; services making use of the location of users or terminals in wireless networks H04W4/02; locating users or terminals in wireless networks H04W64/00
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104

Abstract

Systems and methods provide for a server computer to receive a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream, determine a subset of messages of the plurality of messages associated with a similar geolocation and time period, determine a set of messages of the subset of messages based on a match score for each pair of messages, and stitch together the set of messages to generate a stitched data stream from the data streams for each message of the set of messages based on a time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there may be more than one data stream for a given time period.

Claims

1. A method comprising:

receiving, at a server computer, a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining, by the server computer, a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting, by the server computer, an audio fingerprint for each message in the subset of messages;

grouping, by the server computer, the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing, by the server computer, audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining, by the server computer, a set of messages of the subset of messages associated with a common audio timeline based on the match score for each pair of messages;

generating, by the server computer, a stitched data stream from the data streams of each message of the set of messages by performing operations comprising;

determining, for each message, a time period corresponding to the common audio timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together, by the server computer, the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing, by the server computer, the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

2. The method of claim 1, wherein determining a subset of messages of the plurality of messages associated with a similar geolocation comprises determining that a geolocation for each of the subset of messages is associated with the same predetermined region of global positioning system (GPS) coordinates.

3. The method of claim 1, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

4. The method of claim 1, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.

5. The method of claim 1, wherein the stitched data stream comprises messages that are prioritized based on a quality score for an order of display in the timeline.

6. The method of claim 1, wherein the stitched data stream comprises messages that are prioritized based on a random selection for an order of display in the timeline.

7. The method of claim 1, wherein audio associated with the stitched data stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

8. The method of claim 1, wherein stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, further comprises:

determining a first data stream from a message, of a set of messages that each have a time period that intersects with the time period of the start message;

selecting the first data stream as the second data stream for the stitched data stream; and

continuing to determine a data stream that has the longest duration of a set of messages that each have a time period that intersects with a previously selected data stream to select a next data stream for the stitched data stream.

9. The method of claim 1, wherein extracting the audio fingerprint for each message in the subset of messages comprises:

analyzing the audio for each message in the subset of messages to calculate an audio fingerprint for each message in the subset of messages by extracting features from a spectrogram corresponding the audio for each message in the subset of messages.

10. The method of claim 1, wherein the continuous audio stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

11. A server computer comprising:

a processor; and

a computer readable medium coupled with the processor, the computer readable medium comprising instructions stored thereon that are executable by the processor to cause a computing device to perform operations comprising;

receiving a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting an audio fingerprint for each message in the subset of messages;

grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining a set of messages of the subset of messages associated with a common audio timeline based on the match score for each pair of messages;

generating a stitched data stream from the data streams of each message of the set of messages by performing operations comprising;

determining for each message, a time period corresponding to the common audio timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

12. The server computer of claim 11, wherein determining the subset of messages of the plurality of messages associated with the similar geolocation comprises determining that a geolocation for each of the subset of messages is associated with the same predetermined region of global positioning system (GPS) coordinates.

13. The server computer of claim 11, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

14. The server computer of claim 11, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.

15. The server computer of claim 11, wherein the stitched data stream comprises messages that are prioritized based on a quality score for an order of display in the timeline.

16. The server computer of claim 11, wherein the stitched data stream comprises messages that are prioritized based on a random selection for an order of display in the timeline.

17. The server computer of claim 11, wherein audio associated with the stitched data stream comprises audio that is an average of audio associated with each data stream in the stitched data stream.

18. A non-transitory computer readable medium comprising instructions stored thereon that are executable by at least one processor to cause a computing device to perform operations comprising:

receiving a plurality of messages from a plurality of user computing devices, each message of the plurality of messages comprising a data stream;

determining a subset of messages of the plurality of messages associated with a similar geolocation and time period;

extracting an audio fingerprint for each message in the subset of messages;

grouping the subset of messages into a plurality of pairs of messages, each pair of messages comprising a first message and a second message;

comparing audio fingerprints of the first message and the second message in each pair of messages to determine a match score for each pair of messages;

determining a set of messages of the subset of messages associated with a con on audio timeline based on the match score for each pair of messages;

generating a stitched data stream from the data streams of each message of the set of messages by performing operations comprising;

determining, for each message, a time period corresponding to the common timeline;

determining, a start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline; and

stitching together the set of messages starting with the start message to generate the stitched data stream from the data streams for each message of the set of messages based on the time period for each message, wherein the stitched data stream comprises messages with data streams that overlap in time periods such that there is more than one data stream for at least one time period in the stitched data stream; and

providing the stitched data stream to one or more user computing devices, to be displayed on the one or more computing devices as a continuous data stream that transitions from one message data stream to a next message data stream to a continuous common audio stream.

19. The non-transitory computer readable medium of claim 18, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises selecting the start message from a plurality of messages with similar periods of time in the common audio timeline based on a quality score.

20. The non-transitory computer readable medium of claim 18, wherein determining the start message for the stitched data stream based on a message with a time period occurring first in the common audio timeline for the subset of messages comprises randomly selecting the start message from a plurality of messages with similar periods of time in the common audio timeline.