System And Method For Integration Of Streaming And Static Data

• US 20090037514A1
• Filed: 03/14/2007
• Published: 02/05/2009
• Est. Priority Date: 03/18/2006
• Status: Active Grant

• Current Assignee: Metafluent LLC
• Original Assignee: Metafluent LLC
• Inventors: MacGaffey, Andrew, Lankford, Peter
• Granted Patent: US 8,281,026 B2
• Time in Patent Office: Days
• US Class Current: 709/201
• CPC Class Codes: H04L 45/00 Routing or path finding of packets in data switching networks routing or path finding in wireless networks H04W40/00
• CPC Class Codes: H04L 45/24 Multipath
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/1008 based on parameters of servers, e.g. available memory or workload monitoring of computer activity G06F11/30
• CPC Class Codes: H04L 67/14 Session management for real-time applications in data packet communications networks H04L65/1066

Abstract

The invention relates to integration of streaming data. The invention provides a means for integrating both high-throughput streaming and static content originating from multiple types and/or instances of data sources into discrete units of data. The invention provides integration of a wide variety of disparate streaming and static data sources as well as deployment of content in multiple protocols, either static or streaming or both. In the preferred embodiment, the invention provides a method and system for the insertion of domain-specific implementations of the JMS (Java Message System) messaging paradigm into a server-based JMS provider infrastructure. A Content Integration Framework provides access to Content Instances. The Content Integration Framework is used by Distribution Adapters that expose content from Content Instances cached in the system using a variety of protocols.

Claims

1. (canceled)

2. A system for data integration from multiple data sources and for data output to applications and users, said architecture of said system including:

a Component Framework, which Component Framework includes Distribution Adapters and

a Content Integration Framework operable to integrate streaming and static content originating from multiple types and instances of data sources into Content Instances, and

Distribution Adapters, operable to utilize the Content Integration Framework such that Content Instances, by means of Distribution Adapters, are accessible to applications and users, wherein said Distribution Adapters utilize said Content Integration Framework so as to provide a mechanism by which events can be exchanged between the Content Integration Framework and applications and users.

3. A system as in claim 2, said Content Integration Framework further comprising:

Content Templates, wherein said Content Templates, provide a mechanism for specifying parameterized templates derived from a Content Specification such that a single template can be used to create multiple Content Instances of structurally similar data.

4. A system as in claim 2 wherein said Content Integration Framework further includes:

a set of software interfaces and protocols that support creation of the particular Content Adapters that function to manage the content for a particular name-space in Content Specifications.

5. A system as in claim 2, wherein said Component Framework supports Content Linking, wherein said Content Linking provides a mechanism by which one Content Specification may link to another Content Specification such that any Content Instances resulting from Content Specifications are likewise linked.

6. A system as in claim 2, wherein said Content Integration Framework supports Chained-Event propagation, where said Chained-Event propagation is enabled through a set of software interfaces and primitive constructs that support the propagation of events from Content Instances, where said Content Instances may contain integrated value-added data, where said value-added data is generated by a form of Content Adapter which does not access data from an external source but rather derives data from other data sourced within the same Content Instance or a Linked Content Instance, and where said Chained-Event Propagation, preserves Atomicity of data changes to value-added data with the context of events generated by the original data.

7. A system as in claim 2, wherein said system further includes:

Content-Based Permissioning, wherein a set of software interfaces and protocols support authentication and authorization of applications and users with respect to data subscription and publication using modular authorization mechanisms that can vary from one type of data to another, and whereby the Content Integration Framework supports authentication of application and user and permits access on the basis of data within Content Instances rather than relying on Content Instance names or other types of related identifiers.

8. A computer implemented system capable of integrating data within a server-based infrastructure such that, conformant with a client-controlled Data Model, Content Instances can be retrieved, or events or notifications received, irrespective of the type of data source, said system further comprising:

A) message-oriented middleware, further comprising;

a) Component Framework, comprising;

1) Content Integration Framework, comprising;

Content Adapters;

Content Specifications;

Content Templates;

Content Instances;

2) Distribution Adapters;

3) Distribution Protocol Plug-ins; and

B) an application program interface (API) compatible with the message oriented middleware.

9. A computer implemented system as in claim 8 where the message-oriented middleware is a JMS Provider.

10. A computer implemented system as in claim 8 wherein the message oriented middleware is one of:

RMDS, Wombat, IBM WebSphere Front Office.

11. A computer implemented system as in claim 8 wherein the message-oriented middleware is TIBCO Rendezvous.

12. A computer implemented system as in claim 8 wherein the message oriented middleware is IBM MQ series.

13. A computer implemented system as in claim 8 where the server-based infrastructure is not session oriented.

14. A computer implemented system as in claim 13 wherein the Content Integration Framework is accessed through use of the FAST protocol.

15. A computer implemented system as in claim 13 wherein the Content Integration Framework is accessed via a SOAP Distribution Adapter.

16. A computer implemented system as in claim 13 wherein the Content Integration Framework is accessed via the SMS protocol.

17. A computer implemented system as in claim 13 wherein the Content Integration Framework is accessed via SMTP protocol.

18. A computer implemented system as in claim 13 wherein the Component Framework is compatible with a hybrid system.

19. A computer implemented system as in claim 9 wherein said Content Integration Framework further includes a method for creating, for use by a Distribution Adapter, a Content Instance, said method comprising the steps of:

a) locating a Template in Template Cache;

b) if Template found in Template Cache, then proceed to step c;

if not found, then execute the substeps of;

b1 loading Template;

b.2 allocating Content Adapter nodes;

b.3 verifying/initializing parameter substitution nodes;

b.4 finding adapter pre and post event initializers;

b.5 cache Template; and

c) accessing Content Instance.

20. A computer implemented system as in claim 19 where said Content Integration Framework further includes a method for accessing a Content Instance for use by a Distribution Adapter, said method comprising the steps of:

a) building, by Content Template, of identifier from parameters;

b) determining if cached;

if cached, go to step c;

if not cached, perform the substeps of;

b1 substituting parameters;

b2 allocating content;

b3 populating content;

b4 activating resources;

b5 caching content;

and using, by a Distribution Adapter, Content Instance, followed by the additional steps of;

c) releasing content;

d) determining if done;

if no, proceed to step e;

if yes, performing the substeps of;

d1 deactivating resources;

d2 dropping content from Cache; and

e) finishing.

21. A computer implemented system as in claim 20, wherein said Content Integration Framework directs set up and Content Adapter event handling by causing the performance of the steps of:

1. receiving, by Content Adapter, of data event from external source;

2. generating “

pre-event”

;

3. passing pre-event to “

registered listeners,”

so that event listener prepares for event handling;

4. modifying field values;

5. setting values and generate events as required for “

registered listeners”

; and

, after event

6. generating post-event; and

7. passing post-event to registered-listeners.

22. A computer implemented system as in claim 21, wherein, by performance of said steps Chained-Event propagation is enabled, such that after the completion of event handling for external source by Content Adapter, and after the generation of a post event, the Content Instance then handles all events waiting, including change events for value added content such that registered event listeners detect a single sequence of pre-event, field-level events and post-event, including field-level events for value-added content.

23. A computer implemented system as in claim 22 further including a Managed Content Plug-in, causing performance of the steps of:

1. getting Destination from factory (Server Network Session);

2. checking Cache (XML topic factory) and if not cached;

2a. getting Template parameters from Topic name (XML topic factory);

2b. obtaining Content Instance from Template (XML topic factory);

2c. allocating Distribution Adapter for this Topic (MapMessage Adapter Factory);

2d. initializing Adapter with content (XML topic factory);

2e. initializing field adapters (Map Message Adapter);

2f. allocating XML topic (XML topic factory);

2g. starting Adapter (XML Topic Factory);

2h. registering for Content Events (Map Message Adapter);

3. obtaining subscriber reference (Server Network Session);

4. allocating Topic subscriber (XML Topic);

5. determining session initialized (Map Message Adapter);

5a. if not initialized, Initializing Session (Map Message adapter);

6. Session Initialized (Map Message Adapter); and

7. registering Session Subscriber (Server 5 Network Session).

24. A computer implemented system as in claim 23 further including instructions sufficient to implement method for transmission of data between a client-side of the JMS provider (Client) and a server-side JMS provider (Server), where said data is represented to JMS client applications as JMS MapMessages, said method comprising the steps of:

Step 1 Client requests [Server] permission to publish/subscribe to Destination;

Step 2 Server determines

a) permission and

b) eligibility of Destination for Optimization;

Step 3 if not eligible;

Client transmits message to Server w/o optimization;

if eligible, go to step 4;

Step 4 MapMessage is optimized by the sub steps of;

k) Server determines that Client should use a particular destination-specific dictionary [DSD];

l) Server sends DSD to Client Session;

m) Client Session saves Destination specific dictionary for session-life;

n) Server sends client a reference to DSD for Destination;

o) Client uses reference to associate DSD with Destination;

p) Server sends message to Client using coding implied by dictionary as appropriate to DSD for the particular Destination;

q) Client interprets Server message using meta-data from DSD;

r) Client represents meta-data message to JMS API as MapMessageObjects;

s) Client transmits optimized message to Server; and

t) Server may send to client a new DSD followed by a reference to that DSD for some Destination.

25. A computer implemented system as in claim 24 further including instructions sufficient to implement the use of a Destination-Specific Dictionary enabling message optimization, where such instructions cause the performance of the steps such that:

j) for each field that may be present in a MapMessage, the field is assigned an index starting from zero;

k) the dictionary describes the number of fields and the name and type of each field (e.g. string, floating point, long integer, boolean etc);

l) the data portion of an encoded MapMessage is preceded by a variable-length bitmap (a sequence of binary bits);

m) the first bit in each byte of the bitmap is used as a stop bit wherein the presence of a 1 indicates no further bytes in the bitmap;

n) a value of 1 at a particular point or index in the bitmap indicates the presence of an encoded value for the field having the corresponding index in the dictionary;

o) the specific encoding technique and the number of bytes required to represent a particular value will vary depending on the type of the field;

the Dictionary is used to determine the decoding technique that must be applied for each value;

p) when full “

image”

messages are sent, all bits in the bitmap are fully populated, i.e. all bits (except stop bits) being set to 1;

q) when “

update”

messages are sent the bitmap is sparsely populated, having bits set to 1 only for those indices corresponding to fields in the message; and

r) the number of bytes in the bitmap transferred need only be large enough to accommodate the highest field index in the update, rather than the full number of indices in the image.