Real-time anomaly mitigation in a cloud-based video streaming system

• US 10,164,853 B2
• Filed: 05/29/2015
• Issued: 12/25/2018
• Est. Priority Date: 05/29/2015
• Status: Active Grant

• Current Assignee: iStreamPlanet Co. LLC (Warner Bros. Discovery, Inc.)
• Original Assignee: iStreamPlanet Co. LLC (Warner Bros. Discovery, Inc.)
• Inventors: VanAntwerp, Mark Daniel, Wang, Weimin Mark
• Primary Examiner(s): Bayard, Djenane M
• Application Number:
  US14/725,919
• Publication Number:
  US 20160352603A1
• Time in Patent Office: 1,306 Days
• CPC Class Codes: H04L 43/0847 Transmission error
• CPC Class Codes: H04L 43/55 Testing of service level quality, e.g. simulating service usage
• CPC Class Codes: H04L 65/70 Media network packetisation
• CPC Class Codes: H04L 65/765 intermediate

Abstract

A method for detect and mitigate anomaly in video streaming platforms is disclosed. In one embodiment, performance data from a set of workers is received at a central telemetry system (CTS), where the performance data is indicative of operational status of the set of workers. The CTS processes the performance data, including generating task-specific monitoring data based on the performance data, and it identifies whether the performance data or the task-specific monitoring data contains any anomaly. Upon an anomaly being identified, the CTS mitigates the anomaly by interacting with the set of workers.

Claims

1. A method executed by an electronic device in a video streaming platform including at least a central telemetry system, the method comprising:

receiving at the central telemetry system performance data from at least one local telemetry agent of a worker, where the worker is from a set of workers of the video streaming platform, wherein each worker in the set of workers executes tasks in a task graph of a media workflow created for a video source, where each worker in the set of workers is a processing unit in the video streaming platform, wherein the performance data from each worker is generated during execution of the worker, and wherein the performance data is indicative of operational status of the set of workers;

processing the performance data at the central telemetry system, wherein the processing includes generating task-specific monitoring data based on the performance data;

identifying at the central telemetry system whether the performance data or the task-specific monitoring data contains an anomaly, where the anomaly includes a failure of a component of the video streaming platform which the media workflow utilizes; and

upon the anomaly being identified, mitigating the anomaly by interacting with the set of workers by moving tasks from a worker associated with the anomaly to other workers of the streaming platform and preventing all tasks from running on the worker.

2. The method of claim 1, wherein the performance data for each worker is derived from execution of the tasks in the task graph, wherein the task graph is a directed acyclic graph of tasks with each node of the task graph representing a media processing task and each edge of the task graph representing a data flow across two tasks and a corresponding input and output of each task, and wherein the task-specific monitoring data is indicative of operational status of the tasks.

3. The method of claim 1, wherein the at least one local telemetry agent collects performance data for the worker from different processing blocks at different processing stages within the worker.

4. The method of claim 1, wherein generation of the task-specific monitoring data includes integrating historical performance data with the performance data.

5. The method of claim 1, wherein identifying whether the task-specific monitoring data contains the anomaly includes comparing the task-specific monitoring data with stored historical task-specific monitoring data.

6. The method of claim 1, further comprising:

detecting another anomaly where a value of the task-specific monitoring data crosses a threshold, which predicts failure of a component of the video streaming platform that currently does not impact performance of the media workflow.

7. The method of claim 1, wherein mitigating the anomaly further includes at least one of restarting a task of the task graph, restarting the media workflow, restarting a worker, each of which is associated with the anomaly.

8. The method of claim 1, further comprising:

upon the anomaly being identified, providing a notification to an operator of the video streaming platform, wherein the notification indicates the severity of the anomaly, the task experiencing the anomaly, and the worker experiencing the anomaly.

9. An electronic device to serve as a central telemetry system in a video streaming platform, the electronic device comprising:

a processor and a non-transitory machine-readable storage medium coupled to the processor, the non-transitory machine-readable storage medium containing operations executable by the processor, wherein the electronic device is operative to;

receive at the central telemetry system performance data from at least one local telemetry agent of a worker, where the worker is from a set of workers of the video streaming platform, wherein each worker in the set of workers executes tasks in a task graph of a media workflow created for a video source, where each worker in the set of workers is a processing unit in the video streaming platform, wherein the performance data from each worker is generated during execution of the worker, and wherein the performance data is indicative of operational status of the set of workers;

process the performance data at the central telemetry system, wherein the processing includes generating task-specific monitoring data based on the performance data;

identify at the central telemetry system whether the performance data or the task-specific monitoring data contains an anomaly, where the anomaly includes a failure of a component of the video streaming platform that the media workflow utilizes; and

upon the anomaly being identified, mitigate the anomaly by interacting with the set of workers by moving tasks from a worker associated with the anomaly to other workers of the streaming platform and preventing all tasks from running on the worker.

10. The electronic device of claim 9, wherein the at least one local telemetry agent collects performance data for the worker from different processing blocks at different processing stages within the worker.

11. The electronic device of claim 9, generation of the task-specific monitoring data is to include integrating historical performance data with the performance data.

12. The electronic device of claim 9, wherein the at least one local telemetry agent detects another anomaly that predicts failure of a component of the video streaming platform that currently does not impact performance of the media workflow.

13. The electronic device of claim 9, wherein the mitigation of the anomaly further includes at least one of restarting a task of the task graph, restarting the media workflow, restarting a worker, each of which is associated with the anomaly.

14. The electronic device of claim 9, wherein the electronic device is further operative to:

upon the anomaly being identified, provide a notification to an operator of the video streaming platform, wherein the notification indicates the of severity of the anomaly, the task experiencing the anomaly, and the worker experiencing the anomaly.

15. A non-transitory machine-readable storage medium having instructions stored therein, which when executed by a processor, cause the processor to perform operations in an electronic device serving as a central telemetry system in a video streaming platform, the operations comprising:

receiving at the central telemetry system performance data from at least one local telemetry agent of a worker, where the worker is from a set of workers of the video streaming platform, wherein each worker in the set of workers executes tasks in a task graph of a media workflow created for a video source, where each worker in the set of workers is a processing unit in the video streaming platform, wherein the performance data from each worker is generated during execution of the worker, and wherein the performance data is indicative of operational status of the set of workers;

processing the performance data at the central telemetry system, wherein the processing includes generating task-specific monitoring data based on the performance data;

identifying at the central telemetry system whether the performance data or the task-specific monitoring data contains an anomaly, where the anomaly includes a failure of a component of the video streaming platform which the media workflow utilizes; and

upon the anomaly being identified, mitigating the anomaly by interacting with the set of workers by moving tasks from a worker associated with the anomaly to other workers of the streaming platform and preventing all tasks from running on the worker.

16. The non-transitory machine-readable storage medium of claim 15, wherein the at least one local telemetry agent collects performance data for the worker from different processing blocks at different processing stages within the worker.

17. The non-transitory machine-readable storage medium of claim 15, wherein the at least one local telemetry agent detects another anomaly that predicts failure of a component of the video streaming platform that currently does not impact performance of the media workflow.

18. The non-transitory machine-readable storage medium of claim 15, wherein the operations further comprise:

upon the anomaly being identified, providing a notification to an operator of the video streaming platform, wherein the notification indicates the severity of the anomaly, the task experiencing the anomaly, and the worker experiencing the anomaly.