Ingesting streaming signals

• US 10,552,683 B2
• Filed: 07/15/2019
• Issued: 02/04/2020
• Est. Priority Date: 04/27/2018
• Status: Expired due to Fees

• Current Assignee: Banjo, Inc.
• Original Assignee: Banjo, Inc.
• Inventors: Leung, KW Justin, Urbach, Michael Avner, Guereca-Pinuelas, Armando, Latko, Christopher E., Patton, Damien, Mehta, Rish
• Primary Examiner(s): Zong, Ruolei
• Application Number:
  US16/511,720
• Publication Number:
  US 20190340438A1
• Time in Patent Office: 204 Days
• Field of Search: 709224
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/765 intermediate

Abstract

The present invention extends to methods, systems, and computer program products for ingesting streaming signals. Signal ingestion modules sample a frame from a raw streaming signal. A preliminary severity or possible event type is computed from the sampled frame. A deeper inspection request is triggered of the raw streaming signal or of another raw streaming signal. Segments of content from the raw streaming signal or the other raw streaming signal are inspected. An actual severity or actual event type is computed from the inspected content. The actual severity or actual event type can be included in a normalized signal.

Claims

1. A method comprising:

accessing a raw streaming signal;

ingesting the raw streaming signal including;

a first processing pipeline sampling a plurality of single frames from the raw streaming signal; and

a second processing pipeline storing a plurality of segments separated from at least a portion of the raw streaming signal;

utilizing the first processing pipeline at a first resource allocation level to analyze the plurality of single frames to determine an at least approximate likelihood that the raw streaming signal indicates a possible event of a real-world event type;

determining that the at least approximate likelihood indicates a chance of identification of an actual event that exceeds a threshold;

based on the determining, triggering an inspection of at least a portion of the stored plurality of segments utilizing the second processing pipeline at an increased resource allocation level as compared to the first resource allocation level, including at least;

playing a stored segment that is related to the possible event of the real-world event type, from among the plurality of segments;

inspecting the content of the stored segment during play of the stored segment; and

verifying a real-world event of the real-world event type based on the possible event and the inspected content; and

updating a normalized Time, Location, Context (TLC) signal corresponding to the raw streaming signal to include the verified real-world event and real-world event type.

2. The method of claim 1, wherein ingesting the raw streaming signal comprises ingesting one or more of a video bitstream or an audio bitstream.

3. The method of claim 1, wherein sampling the plurality of single frames comprises sampling the raw streaming signal in accordance with a sampling frequency.

4. The method of claim 1, wherein storing the plurality of segments comprises storing the plurality of segments in a memory buffer.

5. The method of claim 1, wherein storing the plurality of segments comprises storing the plurality of segments at a storage device configured to provide multi consumer access by time and range indexing.

6. The method of claim 1, wherein playing a stored segment comprises feeding the stored segment to a content classifier.

7. The method of claim 6, further comprising:

receiving a context annotation from the content classifier; and

looking up a classification tag corresponding to the context annotation.

8. The method of claim 7, further comprising normalizing the raw streaming signal into the Time, Location, Context (TLC) signal, including:

determining a time associated with the streaming signal;

determining a location associated with the streaming signal; and

including the time, the location, the classification tag, and the plurality of segments in the TLC signal; and

further comprising sending the TLC signal to an event detection infrastructure.

9. The method of claim 1, wherein utilizing the first processing pipeline at a first resource allocation level to analyze the plurality of single frames comprises preliminarily classifying the raw streaming signal as indicative of a real-world event type based on metadata associated with the raw streaming signal.

10. The method of claim 1, wherein ingesting the raw streaming signal comprises ingesting the raw streaming signal selected from among:

a social broadcast or a camera signal.

11. The method of claim 1, wherein ingesting the raw streaming signal comprises the first processing pipeline and the second processing pipeline operating concurrently to ingest the raw streaming signal.

12. A computer system comprising:

a processor;

system memory coupled to the processor and storing instructions configured to cause the processor to;

access a raw streaming signal;

ingest the raw streaming signal using a first processing pipeline;

sample, using the first processing pipeline, a plurality of single frames from the raw streaming signal at a first resource allocation level;

analyze the plurality of single frames to at least approximate a likelihood that the raw streaming signal indicates a possible event of a real-world event type;

determine that the at least approximate likelihood indicates a chance of identification of an actual event that exceeds a threshold;

based on the determination, trigger an inspection of the raw streaming signal utilizing a second processing pipeline at an increased resource allocation level as compared to the first resource allocation level, the inspection causing the second processing pipeline to;

access a segment of the raw streaming signal related to the possible event of the real-world event type;

inspect the content of the segment that is related to the possible event during access of the segment; and

verify a real-world event of the real-world event type based on the possible event and the inspected content; and

update a normalized Time, Location, Context (TLC) signal corresponding to the raw streaming signal to include the verified real-world event and real-world event type.

13. The computer system of claim 12, wherein the instructions configured to sample the plurality of single frames comprise instructions configured to sample a single frame from a video bitstream;

and

wherein the instructions configured to analyze the plurality of single frames comprise instructions configured to analyze pixels in the single frame sampled from the video stream.

14. The computer system of claim 12, wherein the instructions configured to analyze the plurality of single frames comprise instructions configured to use artificial intelligence to analyze the plurality of frames.

15. The computer system of claim 12, further comprising instructions configured to store the accessed segment at a storage device providing multi consumer access by time and range indexing.

16. The computer system of claim 12, wherein the instructions configured to inspect the content of the segment comprise instructions configured to cause a content classifier to determine a context of the segment content.

17. The computer system of claim 16, further comprising instructions configured to cause the processor to:

receive a context annotation from the content classifier; and

look up a classification tag corresponding to the context annotation.

18. The computer system of claim 17, further comprising instructions configured to cause the processor to normalize the raw streaming signal into the TLC signal, including causing the processor to:

determine a time associated with the streaming signal;

determine a location associated with the streaming signal; and

include the time, the location, the classification tag, and the plurality of segments in the TLC signal; and

further comprising instructions configured to cause the processor to send the TLC signal to an event detection infrastructure.

19. The computer system of claim 12, wherein instructions configured to analyze the plurality of single frames comprises instructions configured to preliminarily classify the raw streaming signal based on metadata associated with the raw streaming signal.

20. The computer system of claim 15, wherein instructions configured to store the accessed segment comprises instructions configured to store the accessed segment in a memory buffer.

21. A computer system comprising:

a processor;

system memory coupled to the processor and storing instructions configured to cause the processor to;

access a raw streaming signal;

ingest the raw streaming signal including;

sample, by a first processing pipeline, a plurality of single frames from the raw streaming signal; and

store, by a second processing pipeline, a plurality of segments separated from at least a portion of the raw streaming signal;

utilize the first processing pipeline at a first resource allocation level to analyze the plurality of single frames to determine an at least approximate likelihood that the raw streaming signal indicates a possible event of a real-world event type;

determine that the at least approximate likelihood indicates a chance of identification of an actual event that exceeds a threshold;

based on the determining, trigger an inspection of at least a portion of the stored plurality of segments utilizing the second processing pipeline at an increased resource allocation level as compared to the first resource allocation level, including at least;

play a stored segment that is related to the possible event of the real-world event type, from among the plurality of segments;

inspect the content of the stored segment during play of the stored segment; and

verify a real-world event of the real-world event type based on the possible event and the inspected content; and

update a normalized Time, Location, Context (TLC) signal corresponding to the raw streaming signal to include the verified real-world event and real-world event type.

22. The computer system of claim 21, wherein instructions configured to ingest the raw streaming signal comprise instructions configured to ingest one or more of a video bitstream or an audio bitstream.

23. The computer system of claim 21, wherein instructions configured to sample a plurality of single frames comprise instructions configured to sample the raw streaming signal in accordance with a sampling frequency.

24. The computer system of claim 21, wherein instructions configured to play a stored segment comprise instructions configured to send the stored segment to content classier;

and further comprising instructions configured to;

receive a context annotation from the content classifier; and

looking up a classification tag corresponding to the context annotation.

25. The computer system of claim 21, wherein instructions configured to utilize the first processing pipeline at a first resource allocation level to analyze the plurality of single frames comprise instructions configured to preliminarily classify the raw streaming signal as indicative of a real-world event type based on metadata associated with the raw streaming signal.

26. A method comprising:

accessing a raw streaming signal;

ingesting the raw streaming signal using a first processing pipeline;

sampling, using the first processing pipeline, a plurality of single frames from the raw streaming signal at a first resource allocation level;

analyzing the plurality of single frames to at least approximate a likelihood that the raw streaming signal indicates a possible event of a real-world event type;

determining that the at least approximate likelihood indicates a chance of identification of an actual event that exceeds a threshold;

based on the determination, triggering an inspection of the raw streaming signal utilizing a second processing pipeline at an increased resource allocation level as compared to the first resource allocation level, the inspection causing the second processing pipeline to;

accessing a segment of the raw streaming signal related to the possible event of the real-world event type;

inspecting the content of the segment that is related to the possible event during access of the segment; and

verifying a real-world event of the real-world event type based on the possible event and the inspected content; and

updating a normalized Time, Location, Context (TLC) signal corresponding to the raw streaming signal to include the verified real-world event and real-world event type.

27. The method of claim 26, wherein sampling a plurality of single frames comprises sampling a single frame from a video bitstream.

28. The method of claim 26, wherein analyzing the plurality of single frames comprises analyzing pixels included in a frame of a video stream.

29. The method of claim 26, wherein analyzing the plurality of signal frames comprises preliminarily classifying the raw streaming signal based on metadata associated with the raw streaming signal.

30. The method of claim 26, wherein inspecting the content of the segment comprises determining a context of the segment content.