Web-based configurable pipeline for media processing

• US 8,856,212 B1
• Filed: 08/10/2011
• Issued: 10/07/2014
• Est. Priority Date: 02/08/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Jia, Wei
• Primary Examiner(s): YI, DAVID
• Application Number:
  US13/206,996
• Time in Patent Office: 1,154 Days
• Field of Search: 709/203, 709/212
• US Class Current: 709/203
• CPC Class Codes: G06F 9/54 Interprogram communication

Abstract

An apparatus and method for a web-based configurable pipeline for media processing is disclosed. The disclosed embodiments include a web browser implemented on a computing device. The web browser includes a web application processor configured to execute instructions included in a web application, and one or more configurable pipelines, wherein a configurable pipeline of the one or more configurable pipelines is created using at least some of the instructions included in the web application, the configurable pipeline including a plurality of components for processing one or more media streams, each component including at least one input or at least one output, and the configurable pipeline is adapted to establish data connections between the plurality of components.

Claims

1. An apparatus, comprising:

a processor configured to execute instructions included in a web application accessible through a web browser, the instructions including;

instructions to select a plurality of executable software components, each of the plurality of executable software components including at least one of an input or an output;

instructions to establish data connections with the plurality of executable software components so as to establish a processing order for the plurality of executable software components, wherein each component includes at least one of an input configured to receive input data in the processing order or an output configured to provide output data in the processing order, and the plurality of executable software components and the processing order forming a configurable pipeline, the data connections including;

a plurality of control channels for the plurality of executable software components, each control channel established between the web application and an executable software component of the plurality of executable software components, the plurality of control channels enabling the web application to control the plurality of executable software components; and

a plurality of data channels for the plurality of executable software components, each data channel established between one of the web application and an input or an output of each executable software component, or an input of one executable software component and an output of another executable software component, to pass data between the data channels to form the processing order; and

instructions to process a media stream through the configurable pipeline by decoding or encoding the media stream.

2. The apparatus of claim 1, wherein the configurable pipeline is configured to decode an encoded media stream, wherein:

one or more components of the plurality of executable software components are configured as source inputs for accepting the encoded media stream;

one or more components of the plurality of executable software components are configured for rendering a decoded media stream after processing; and

the remaining components of the plurality of executable software components are configured to decode the encoded media stream into the decoded media stream by processing the encoded media stream.

3. The apparatus of claim 1, wherein the media stream includes audio and video streams and the configurable pipeline includes a demuxer to split the video and audio streams for separate processing.

4. The apparatus of claim 1, wherein the configurable pipeline is configured to encode a raw media stream, wherein:

one or more components of the plurality of executable software components are configured as source inputs for accepting the raw media stream;

one or more components of the plurality of executable software components are configured to output an encoded media stream after processing; and

the remaining components of the plurality of executable software components are configured to encode the raw media stream into the encoded media stream by processing the raw media stream.

5. The apparatus of claim 1, wherein the plurality of executable software components are selected from the group consisting of built-in components, binary add-in components, and interpretive add-in components.

6. The apparatus of claim 1, further comprising:

a web application receiver configured to receive the web application from a web server.

7. The apparatus of claim 1, wherein:

each data channel is established between the web application and the input or the output of each executable software component, the web application configured to pass data between the data channels to form the processing order.

8. The apparatus of claim 1, wherein the instructions include:

instructions to establish a pipeline controller having a pipeline control channel to the web application, the pipeline controller configured to control the configurable pipeline responsive to control signals sent to the pipeline controller from the web application via the pipeline control channel;

and wherein

each data channel is established between the pipeline controller and the input or the output of each executable software component, the pipeline controller configured to pass data between the data channels to form the processing order.

9. The apparatus of claim 1, wherein an intermediate executable software component of the one or more executable software components is an interpretive add-in executable software component, the intermediate executable software component included as instructions and executed within the web application, the intermediate executable software component configured to accept a stream of data output via a source input or component data channel, process the stream of data, and output the processed stream of data to a data channel connected to an input of a executable software component or renderer.

10. The apparatus of claim 1, wherein:

each data channel is established between the input of one executable software component and the output of another executable software component, the plurality of executable software components configured to pass data through the data channels to form the processing order.

11. The apparatus of claim 1, wherein the plurality of components includes a mixer component having a plurality of inputs and configured to combine media from the plurality of inputs into a single output.

12. The apparatus of claim 1, wherein the plurality of executable software components includes a decoder.

13. The apparatus of claim 1, wherein at least one of the plurality of executable software components is an add-in component specified by the instructions included in the web application.

14. The apparatus of claim 13, wherein the plurality of executable software components is provided with an interface to the web browser that is limited to functionality related to processing the media stream in the configurable pipeline.

15. An apparatus, comprising:

a processor configured to execute instructions included in a web application accessible through a web browser, the instructions including;

instructions to establish at least one configurable decoder pipeline, each configurable decoder pipeline including;

a source input for accepting encoded media specified by the web application,

a processing component for processing the accepted encoded media, and

a renderer for rendering the accepted encoded media to one or more output devices;

a plurality of control channels, each control channel established between the web application and one of the source input, the processing component or the renderer, the plurality of control channels enabling the web application to control the source input, the processing component or the renderer; and

a plurality of data channels to pass data through the configurable decoder pipeline, each data channel established between one of;

the web application and an input or an output of the source input, the processing component or the renderer;

an input of the processing component and an output of the source input;

or

an input of the renderer and an output of the processing component; and

instructions to process a media stream through the at least one configurable decoder pipeline, wherein the media stream includes audio and video streams and a first configurable decoder pipeline includes a demuxer to split the video and audio streams for separate processing.

16. A method, comprising:

obtaining a web application by a web browser;

executing, using a processor, instructions of the web application to;

select a plurality of executable software components, each of the plurality of executable software components including at least one of an input or an output;

create at least one configurable pipeline that includes the plurality of executable software components by establishing a plurality of control channels, each control channel established between the web application and an executable software component of the plurality of executable software components, the plurality of control channels enabling the web application to control the plurality of executable software components to process a media stream in a processing order; and

configure the plurality of executable software components to pass data by establishing a plurality of data channels, each data channel established between one of the web application and an input or an output of each executable software component, or an input of one executable software component and an output of another executable software component;

obtaining a media stream; and

processing the media stream using the at least one configurable pipeline in the processing order.

17. The method of claim 16, wherein processing the media stream comprises:

controlling each of the plurality of executable software components in the configurable pipeline directly from the web application.

18. The method of claim 17, wherein processing the media stream comprises:

transferring data between the plurality of executable software components in the configurable pipeline via the plurality of data channels, each of the data channels directly connecting an input of a first component of the plurality of executable software components and an output of a second component of the plurality of executable software components.

19. The method of claim 17, wherein processing the media stream comprises:

transferring data between the plurality of executable software components in the configurable pipeline via the plurality of data channels, each of the data channels connecting the input or the output of each executable software component of the plurality of executable software components to the web application, the web application configured to transfer data received from the plurality of data channels between the plurality of executable software components using the plurality of data channels.

20. The method of claim 16, wherein processing the media stream comprises:

controlling each of the plurality of executable software components in the configurable pipeline directly from a pipeline control object; and

controlling the pipeline control object directly from the web application, wherein the pipeline control object is configured to control the plurality of executable software components responsive to the control of the web application.

21. An apparatus, comprising:

a memory; and

a processor configured to execute instructions stored in the memory to;

obtain a web application by a web browser,

select a plurality of executable software components based on instructions of the web application, each of the plurality of executable software components including at least one of an input or an output;

create at least one configurable pipeline that includes the plurality of executable software components by establishing a plurality of control channels, each control channel established between the web application and an executable software component of the plurality of executable software components, the plurality of control channels enabling the web application to control the plurality of executable software components to process a media stream in a processing order;

configure the plurality of executable software components to pass data by establishing a plurality of data channels, each data channel established between one of the web application and an input or an output of each executable software component, or an input of one executable software component and an output of another executable software component;

obtain a media stream; and

process the media stream using the at least one configurable pipeline in the processing order.