Securely and efficiently extending data processing pipeline functionality
First Claim
1. At a computer system, the computer system including a processor and system memory, the computer system also including a data pipeline comprising a sequential arrangement of a plurality of pipeline modules, an electronic message received at the data pipeline being sequentially processed by pipeline modules such that the output of at least one pipeline module is the input to the next sequential pipeline module, a method for extending the functionality of the data processing pipeline, the method comprising:
- an act of an agent engine that is external to the data pipeline detecting an occurrence of an event within the data pipeline;
an act of mapping the detected event to a specified topic string representing an electronic messaging command;
an act of matching the specified topic string to the topic string for a plurality of data processing agents previously stored at a specified known location at the computer system so as to determine that the plurality of data processing agents have registered for the detected event, the plurality of data processing agents being external to the data pipeline;
an act of the processor loading each of the plurality of data processing agents into a secure zone such that execution of each data processing agent in the plurality of data processing agents is isolated from execution of all the other data processing agents in the plurality of data processing agents;
for each of the plurality of data processing agents;
an act of the agent engine external to the data pipeline diverting pipeline data related to the represented electronic messaging command out of the data pipeline from a first pipeline module and to the data processing agent to extend the functionality of the data pipeline subsequent to loading the data processing agent, wherein the pipeline data is diverted by the external agent engine when the pipeline data is directed from the first pipeline module to a second pipeline module, such that the external agent engine at least temporarily prevents the pipeline data from reaching its intended destination, namely the second pipeline module, the second pipeline module being a next sequential pipeline module following the first pipeline module; and
an act of the agent engine external to the data pipeline injecting results from the data processing agent back into the data pipeline for further processing, the results being injected at the second pipeline module such that sequential processing continues at the next sequential pipeline module following the first pipeline module, namely the second pipeline module to which the pipeline data was directed prior to such pipeline data being diverted by the agent engine;
the agent engine accessing a plurality of application programming interfaces (API), including at least;
an Initialize API configured to transfer a server value, configuration settings values, and an agent group value for initializing the agent engine;
an Invoke API configured to transfer an execution context value, an event identifier value, and a list of other context values for invoking the plurality of data processing agents;
an Asynchronous Invoke API configured to transfer the execution context value, the event identifier value, a time out value, a callback value, a state value, and the list of other context values for asynchronously invoking the plurality of data processing agents; and
an Event Handler API configured to transfer an event source value and event argument values for executing a specified agent handler within one or more of the plurality of data processing agents; and
the specified known location at the computer system storing data processing agents mapped to each of the following topic strings and Extended Simple Mail Transfer Protocol keywords and verbs;
OnDataCommand, OnEhloCommand, OnEndOfData, OnEndOfHeaders, OnHeloCommand, OnHelpCommand, OnMailCommand, OnNoopCommand, OnRcptCommand, OnReject, OnRsetCommand, OnSessionStopEvent, OnCategorization, OnDeleteMailItem, OnPostCategorization, OnSubmit, OnPerRecipient, OnAuthCommand, SMTP HELLO, and SMTP AUTH.
2 Assignments
0 Petitions
Accused Products
Abstract
The present invention extends to methods, systems, and computer program products for securely and efficiently extending data processing pipeline functionality. Data processing agents register, potentially based solely on configuration data contained in the data processing agents, to manipulate data of a data processing pipeline. Configuration data can identify a specified event (e.g., an SMTP verb) the data processing agent is to hook into. Upon detection of a specified event, one or more agents that have registered for the specified event are loaded into secure zones. Process boundaries separate agents from one another and from other processes such that execution of each agent is isolated from other processes. Pipeline related data (e.g., electronic message data) is transferred to each loaded agent. Application Program Interfaces (“APIs”) can be used to transfer data between modules that implement embodiments of the present invention.
29 Citations
14 Claims
-
1. At a computer system, the computer system including a processor and system memory, the computer system also including a data pipeline comprising a sequential arrangement of a plurality of pipeline modules, an electronic message received at the data pipeline being sequentially processed by pipeline modules such that the output of at least one pipeline module is the input to the next sequential pipeline module, a method for extending the functionality of the data processing pipeline, the method comprising:
-
an act of an agent engine that is external to the data pipeline detecting an occurrence of an event within the data pipeline; an act of mapping the detected event to a specified topic string representing an electronic messaging command; an act of matching the specified topic string to the topic string for a plurality of data processing agents previously stored at a specified known location at the computer system so as to determine that the plurality of data processing agents have registered for the detected event, the plurality of data processing agents being external to the data pipeline; an act of the processor loading each of the plurality of data processing agents into a secure zone such that execution of each data processing agent in the plurality of data processing agents is isolated from execution of all the other data processing agents in the plurality of data processing agents; for each of the plurality of data processing agents; an act of the agent engine external to the data pipeline diverting pipeline data related to the represented electronic messaging command out of the data pipeline from a first pipeline module and to the data processing agent to extend the functionality of the data pipeline subsequent to loading the data processing agent, wherein the pipeline data is diverted by the external agent engine when the pipeline data is directed from the first pipeline module to a second pipeline module, such that the external agent engine at least temporarily prevents the pipeline data from reaching its intended destination, namely the second pipeline module, the second pipeline module being a next sequential pipeline module following the first pipeline module; and an act of the agent engine external to the data pipeline injecting results from the data processing agent back into the data pipeline for further processing, the results being injected at the second pipeline module such that sequential processing continues at the next sequential pipeline module following the first pipeline module, namely the second pipeline module to which the pipeline data was directed prior to such pipeline data being diverted by the agent engine; the agent engine accessing a plurality of application programming interfaces (API), including at least; an Initialize API configured to transfer a server value, configuration settings values, and an agent group value for initializing the agent engine; an Invoke API configured to transfer an execution context value, an event identifier value, and a list of other context values for invoking the plurality of data processing agents; an Asynchronous Invoke API configured to transfer the execution context value, the event identifier value, a time out value, a callback value, a state value, and the list of other context values for asynchronously invoking the plurality of data processing agents; and an Event Handler API configured to transfer an event source value and event argument values for executing a specified agent handler within one or more of the plurality of data processing agents; and the specified known location at the computer system storing data processing agents mapped to each of the following topic strings and Extended Simple Mail Transfer Protocol keywords and verbs; OnDataCommand, OnEhloCommand, OnEndOfData, OnEndOfHeaders, OnHeloCommand, OnHelpCommand, OnMailCommand, OnNoopCommand, OnRcptCommand, OnReject, OnRsetCommand, OnSessionStopEvent, OnCategorization, OnDeleteMailItem, OnPostCategorization, OnSubmit, OnPerRecipient, OnAuthCommand, SMTP HELLO, and SMTP AUTH. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A computer program product for use at a computer system, the computer system including a processor and system memory, the computer system also including a data pipeline comprising a sequential arrangement of a plurality of pipeline modules, an electronic message received at the data pipeline being sequentially processed by pipeline modules such that the output of at least one pipeline module is the input to the next sequential pipeline module, the computer program product for implementing a method for extending the functionality of the data processing pipeline, the computer program product comprising one or more physical computer storage devices having stored thereon computer-executable instructions that, when executed by a processor, cause the computer system to perform the following:
-
at an agent engine that is external to the data pipeline, detect an occurrence of an event within the data pipeline; map the detected event to a specified topic string representing an electronic messaging command match the specified topic string to the topic string for a plurality of data processing agents previously stored at a specified known location at the computer system so as to determine that a plurality of data processing agents have registered for the detected event, the plurality of data processing agents being external to the data pipeline; load each of the plurality of data processing agents into a secure zone such that execution of each data processing agent in the plurality of data processing agents is isolated from execution all other data processing agents in the plurality of data processing agents; for each of the plurality of data processing agents; at the agent engine external to the data pipeline, and acting independent of directives from the data pipeline, divert pipeline data related to the represented electronic messaging command out of the data pipeline from a first pipeline module to the data processing agent to extend the functionality of the data pipeline subsequent to loading the data processing agent, wherein the pipeline data is diverted by the external agent engine when the pipeline data is directed from the first pipeline module to a second pipeline module, such that the external agent engine at least temporarily prevents the pipeline data from reaching its intended destination, namely the second pipeline module, the first pipeline module being configured to maintain the pipeline data within the data pipeline by directing the pipeline data to the second pipeline module and the second pipeline module being a next sequential pipeline module following the first pipeline module; and at the agent engine external to the data pipeline, inject results from the data processing agent back into the data pipeline for further processing, the results being injected at the second pipeline module such that sequential processing continues at the next sequential pipeline module following the first pipeline module, namely the second pipeline module to which the pipeline data was directed prior to such pipeline data being diverted by the agent engine; the agent engine accesses a plurality of application programming interfaces (API), including at least; an Initialize API configured to transfer a server value, configuration settings values, and an agent group value for initializing the agent engine; an Invoke API configured to transfer an execution context value, an event identifier value, and a list of other context values for invoking the plurality of data processing agents; an Asynchronous Invoke API configured to transfer the execution context value, the event identifier value, a time out value, a callback value, a state value, and the list of other context values for asynchronously invoking the plurality of data processing agents; and an Event Handler API configured to transfer an event source value and event argument values for executing a specified agent handler within one or more of the plurality of data processing agents; and the specified known location at the computer system stores data processing agents mapped to each of the following topic strings and Extended Simple Mail Transfer Protocol keywords and verbs; OnDataCommand, OnEhloCommand, OnEndOfData, OnEndOfHeaders, OnHeloCommand, OnHelpCommand, OnMailCommand, OnNoopCommand, OnRcptCommand, OnReject, OnRsetCommand, OnSessionStopEvent, OnCategorization, OnDeleteMailItem, OnPostCategorization, OnSubmit, OnPerRecipient, OnAuthCommand, SMTP HELO, and SMTP AUTH. - View Dependent Claims (11, 12, 13, 14)
-
Specification