Method and system for processing raw financial data streams to produce and distribute structured and validated product offering objects

• US 7,958,025 B2
• Filed: 05/31/2001
• Issued: 06/07/2011
• Est. Priority Date: 08/04/2000
• Status: Active Grant

• Current Assignee: Goldman Sachs & Co. LLC (Goldman Sachs Group Incorporated)
• Original Assignee: Goldman Sachs & Company (Goldman Sachs Group Incorporated)
• Inventors: Smith, Andrew J. R., Thomas, Deane, Cowan, Randy
• Primary Examiner(s): Hamilton, Lalita M
• Application Number:
  US09/871,427
• Publication Number:
  US 20020046043A1
• Time in Patent Office: 3,659 Days
• Field of Search: 705/35, 705/36.R
• US Class Current: 705/35
• CPC Class Codes: G06F 16/95 Retrieval from the web
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes
• CPC Class Codes: G06Q 40/04 Trading; Exchange, e.g. stocks, commodities, derivatives or currency exchange
• CPC Class Codes: G06Q 40/06 Asset management; Financial planning or analysis
• CPC Class Codes: H04L 12/1859 adapted to provide push services, e.g. data channels
• CPC Class Codes: H04L 12/1895 for short real-time information, e.g. alarms, notifications, alerts, updates
• CPC Class Codes: H04L 41/0604 using filtering, e.g. reduction of information by using priority, element types, position or time
• CPC Class Codes: H04L 67/55 Push-based network services

Abstract

A real time information manager which processes raw data from data providers, such as feeds of current information about financial product offerings, is provided. Raw data objects received from various data providers are analyzed, formatted, and a current version of each unique data object is stored in an object storage pool. Information about the state of each objects is transmitted on a particular output data channel in accordance with the type of object. A subscriber process can access one or more of the data channels and utilize the broadcast data. In one configuration, one or more client managers are provided which subscribe to particular data channels and serve as an intermediary between the real time information manager and sets of clients.

Claims

1. A processor-implemented method for processing a raw data stream generated by a financial product provider and carrying a plurality of raw data objects containing information related to financial product offerings and subject to periodic updates, the method comprising:

receiving a raw data object carried on a raw data stream input;

determining an object type of the raw data object;

generating via a processor a formatted data object based on the contents of the raw data object by applying a set of formatting rules selected in accordance with the determined object type;

determining if a prior version of the formatted data object is present in an object storage pool;

if a prior version is present in the object storage pool;

determining a data differential between the formatted data object and the prior version;

updating the prior version of the formatted data object to correspond with the generated formatted data object; and

incrementing a sequence number associated with the formatted data object; and

broadcasting the data differential and the sequence number on an output broadcast data channel selected from a plurality of broadcast channels according to the object type;

otherwise, if no prior version is present in the object storage pool;

providing the formatted data object to subscriber processes; and

storing the formatted data object and an associated initial sequence number in the object storage pool.

2. The method of claim 1, wherein providing the formatted data object comprises broadcasting the formatted data object on the output broadcast channel.

3. The method of claim 1, wherein the formatted data object stored in the object storage pool has an associated sequence number;

wherein updating comprises incrementing the sequence number of the formatted data object; and

broadcasting the data differential including broadcasting the sequence number associated with the formatted data object.

4. The method of claim 1, further comprising broadcasting the formatted data object on the output broadcast channel.

5. The method of claim 1, wherein the broadcast channel is selected from a plurality of broadcast channels according to the object type.

6. The method of claim 1, wherein the applied formatting rules are selected in accordance with the determined object type.

7. The method of claim 1, further comprising validating the contents of the raw data object in accordance with selected validation conditions.

8. The method of claim 1, wherein the object typing and formatting rules are stored in a tree format;

the tree having a root node and at least one descendant stream nodes, each stream node being associated with a specific raw data stream;

each stream node further having at least one type leaf node descending therefrom, each type leaf node being associated with a specific object type carried by the raw data stream associated with the respective stream node;

each type leaf node having at least one associated object typing rule for identifying objects of the type associated with the respective type leaf node.

9. The method of claim 8, wherein each type leaf node further has at least one associated formatting rule.

10. The method of claim 1, further comprising:

receiving a particular formatted data object at a subscriber process;

storing the particular formatted data object in a database associated with the subscriber process;

receiving a data differential for the particular formatted object via the broadcast data channel; and

update the stored formatted data object in accordance with the received data differential.

11. The method of claim 10, further comprising communicating information about the formatted data object stored in the subscriber database to a remote location.

12. The method of claim 1, wherein the raw data object comprises information related to a financial product offering.

13. A system for processing raw data streams carrying a plurality of raw events containing information which is subject to periodic updates, the system comprising processor-implemented components:

an information manager component receiving the raw data streams as input;

a processing database component connected to the information manager and having object typing and formatting rules stored therein; and

an object storage pool component connected to the information manager;

the information manager component configured to;

receive a raw data object carried on a particular raw data stream input;

identify an object type of the raw data object in accordance with the object typing rules;

generate a formatted data object based on the contents of the raw data object by in accordance with the formatting rules;

determine if a prior version of the formatted data object is present in the object storage pool and in response to a negative determination;

provide the formatted data object to subscriber processes, and store the formatted data object in the object storage pool;

otherwise, in response to a negative determination;

update the prior version of the formatted data object to correspond with the generated formatted data object, identify a data differential between the formatted data object and the prior version, and broadcast the data differential on an output broadcast data channel.

14. The system of claim 13, wherein the information manager is further configured to:

associate a respective sequence number with formatted data objects stored in the object storage pool;

increment the respective sequence number during an update to the formatted data object in the storage pool; and

broadcast the respective sequence number with the data differential.

15. The system of claim 13, wherein the formatted data objects are provided to subscriber processes via the output broadcast channel.

16. The system of claim 13, wherein the information manager is further configured to select the broadcast channel from a plurality of broadcast channels according to the identified object type.

17. The system of claim 13, wherein the information manager is further configured to apply particular formatting rules selected in accordance with the identified object type.

18. The system of claim 13, further comprising at least one client manager connected to the broadcast channel;

the client manager configured to;

receive a particular formatted data object;

store the received formatted data object in a database associated with the client manager;

receive a data differential for the particular formatted object via the broadcast data channel; and

update the stored formatted data object in accordance with the received data differential.

19. The system of claim 18, wherein the client manager is further configured to provide information related to data objects stored in the associated database to a plurality of subscribing clients.

20. The system of claim 18, wherein a plurality of client managers are connected to the broadcast data channel.

21. The system of claim 13, wherein the raw data objects comprise information related to a financial product offerings.

22. A system for processing raw data streams generated by respective financial product providers, each stream carrying a plurality of raw data objects containing information related to financial product offerings and subject to periodic updates, the system comprising processor-implemented components:

a translator component configured to receive the raw data objects as input and generate raw events comprising a set of name-value pairs derived from data in the raw data objects;

a processing database component having object typing and formatting rules stored therein;

an offer processor component connected to the processing database and configured to, in response to the receipt of a raw event;

determine an object type associated with the raw event; and

generate an external event containing a formatted data object derived from the contents of the received raw event in accordance with the formatting rules;

an object storage pool component configured to store a plurality of formatted data objects therein;

an offer pool manager component connected to the object storage pool and configured to, upon receipt of an external event;

determine if a prior version of the formatted data object is present in an object storage pool;

if a prior version is present in the object storage pool;

determine a data differential between the formatted data object and the prior version, update the prior version of the formatted data object to correspond with the generated formatted data object, and broadcast the data differential on an output broadcast data channel;

otherwise, if no prior version is present in the object storage pool;

provide the formatted data object to subscriber processes, and store the formatted data object in the object storage pool.

23. The system of claim 22, wherein the offer pool manager is further configured to:

associate a sequence number with the formatted data object stored in the object storage pool;

increment the sequence number upon updating the formatted data object; and

broadcast the sequence number associated with the formatted data object with the data differential.

24. The system of claim 22, wherein the offer pool manager is configured to provide the formatted data object by broadcasting the formatted data object on the output broadcast channel.

25. The system of claim 22, wherein the offer pool manager is configured to select the broadcast channel from a plurality of broadcast channels in accordance with the object type.

26. The system of claim 22, wherein the offer processor is configured to apply specific formatting rules in accordance with the determined object type.

27. The system of claim 22, the processing database further having object validation conditions stored therein;

the offer processor being further configured to validate the contents of the raw data object in accordance with selected validation conditions.

28. The system of claim 22, wherein the object typing and formatting rules are stored in a tree format;

the tree having a root node and at least one descendant stream nodes, each stream node being associated with a specific raw data stream;

each stream node further having at least one type leaf node descending therefrom, each type leaf node being associated with a specific object type carried by the raw data stream associated with the respective stream node;

each type leaf node having at least one associated object typing rule for identifying objects of the type associated with the respective type leaf node.

29. The system of claim 28, wherein each type leaf node further has at least one associated formatting rule.

30. The system of claim 22, further comprising at least one client manager in communication with the information manager;

the client manager configured to;

receive a particular formatted data object from the offer pool manager;

store the particular formatted data object in an associated client manager database;

receive a data differential for the particular formatted object via the broadcast data channel; and

update the stored formatted data object in accordance with the received data differential.

31. The system of claim 30, wherein the client manager is further configured to communicate information about the formatted data object stored in the subscriber database to a remote location.

32. The system of claim 22, wherein the raw data object comprises information related to a financial product offering.

33. A processor-implemented method for processing raw data streams carrying a plurality of raw data objects containing information which is subject to periodic updates, the method comprising:

receiving a raw data object carried on a raw data stream input;

determining an object type of the raw data object;

generating via a processor a formatted data object based on the contents of the raw data object by applying a set of formatting rules;

determining if a prior version of the formatted data object is present in an object storage pool;

if a prior version is present in the object storage pool;

determining a data differential between the formatted data object and the prior version;

updating the prior version of the formatted data object to correspond with the generated formatted data object; and

broadcasting the data differential on an output broadcast data channel;

otherwise, if no prior version is present in the object storage pool;

providing the formatted data object to subscriber processes; and

storing the formatted data object in the object storage pool.

34. The method of claim 33, wherein the formatted data object stored in the object storage pool has an associated sequence number;

wherein updating comprises incrementing the sequence number of the formatted data object; and

broadcasting the data differential includes broadcasting the sequence number associated with the formatted data object.

35. The method of claim 33, wherein providing the formatted data object comprises broadcasting the formatted data object on the output broadcast channel.

36. The method of claim 33, wherein the broadcast channel is selected from a plurality of broadcast channels according to the object type.

37. The method of claim 33, wherein the applied formatting rules are selected in accordance with the determined object type.

38. The method of claim 33, further comprising translating the raw data object into a raw event comprising at least one name-value pair prior to determining an object type of the raw data object and generating a formatted data object.

39. The method of claim 33, further comprising the steps of:

validating the contents of the raw data object; and

upon a failed validation, preventing subsequent broadcast of the data differential or formatted data object derived from the respective raw data object.

40. The method of claim 33, further comprising:

receiving a particular formatted data object at a subscriber process;

storing the received formatted data object in a database associated with the subscriber process;

receiving at the subscriber process a data differential for the particular formatted object via the broadcast data channel; and

updating the stored formatted data object in accordance with the received data differential.

41. The method of claim 40, wherein the broadcast data channel is received by a plurality of subscriber processes.

42. The method of claim 40, wherein the subscriber processes receives a plurality of broadcast channels.

43. The method of claim 40, further comprising communicating information about the formatted data object stored in the subscriber database to a remote location.

44. The method of claim 33, wherein the raw data object comprises information related to a financial product offering.