SYSTEMS AND METHODS FOR SCALABLE STRUCTURED DATA DISTRIBUTION

• US 20130275612A1
• Filed: 04/15/2013
• Published: 10/17/2013
• Est. Priority Date: 04/13/2012
• Status: Abandoned Application

• Current Assignee: Goldman Sachs & Co. LLC (Goldman Sachs Group Incorporated)
• Original Assignee: Goldman Sachs & Company (Goldman Sachs Group Incorporated)
• Inventors: Voss, Matthew, Mavuram, Vishnu, Cohen, Scott
• Application Number:
  US13/863,248
• Publication Number:
  US 20130275612A1
• Time in Patent Office: Days
• US Class Current: 709/231
• CPC Class Codes: G06F 16/113 Details of archiving lifecycle management in storage systems G06F3/0649; backup systems G06F11/1446
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24561 Intermediate data storage techniques for performance improvement
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/25 Integrating or interfacing systems involving database management systems
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04L 67/1001 for accessing one among a plurality of replicated servers
• CPC Class Codes: H04L 67/561 Adding application-functional data or data for application control, e.g. adding metadata
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

Systems and methods for efficiently absorbing, archiving, and distributing any size data sets are provided. Some embodiments provide flexible, policy-based distribution of high volume data through real time streaming as well as past data replay. In addition, some embodiments provide for a foundation of solid and unambiguous consistency across any vendor system through advanced version features. This consistency is particularly valuable to the financial industry, but also extremely useful to any company that manages multiple data distribution points for improved and reliable data availability.

Claims

1. A method comprising:

receiving streaming data from a data producer;

determining business aligned archive sequence from the streaming data that should be bundled together in accordance with a set of bundling parameters;

bundling the data into packages of data having a standard format;

ordering each of the packages of data using a series of consecutive integers produced by a master clock;

publishing metadata regarding availability of the packages of data on a control channel; and

delivering the packages of data to data consumers which have subscribed to the data producer.

2. The method of claim 1, wherein the bundling parameters include declarative rules specified by a business.

3. The method of claim 1, wherein the standard format is a standard neutral format biased for movement and compression of the streaming data.

4. The method of claim 1, further comprising replaying the packages of data based on the ordering upon a request from a data consumer.

5. The method of claim 1, further comprising archiving the packages of data in a platform independent manner.

6. The method of claim 1, wherein the metadata that is published on the control channel includes indexes.

7. The method of claim 1, wherein bundling the data includes identifying data that when compressed will result in each package in the packages of data having a desired size.

8. The method of claim 1, wherein bundling the data includes associating new metadata with each of the bundled packages, wherein the new meta data comprises at least one of summary data, quality data, index data, or checksum data.

9. The method of claim 1, wherein the delivery of the packages of data to the consumers comprises parallel delivery.

10. The method of claim 1, further comprising using columnar checksums for verifying the data.

11. The method of claim 10, wherein the columnar checksums allow for rounding errors with a specified tolerance.

12. A system comprising:

a bundler configured to receive streaming raw data from a data producer and bundle the raw data into a series of data packages and associate with each of the data packages a unique identifier having a monotonically increasing order based on upload from the data producer;

a transformer to receive the data packages having the associated unique identifier and generate loadable data structures for a reporting store associated with a data subscriber; and

a loader to receive and store the loadable data structures into a storage device associated with the data subscriber based on the monotonically increasing order.

13. The system of claim 12, wherein the streaming raw data comprises multiple streams.

14. The system of claim 13, wherein the data packages from each of the multiple streams are assigned different sets of unique identifiers.

15. The system of claim 13, wherein each of the multiple streams of streaming raw data are assigned a flow priority.

16. The system of claim 12, further comprising an identification module to receive a logical series of integers from the stream clock and generate the unique identifier having the logical ordering.

17. The system of claim 16, further comprising a stream clock configured to generate the logical series of integers, and wherein a single integer is the unique identifier associated with a single data package in the series of data packages.

18. The system of claim 12, further comprising:

a data channel allowing data from a data producer to be continuously streamed to the data subscriber through the bundler;

a messaging channel to provide a current status of the data being continuously streamed from the data producer to the data subscriber; and

a control channel separate from the data channel to allow the data subscriber to request replay of the data.

19. The system of claim 18, wherein the control channel is running at a faster rate than the data channel.

20. The system of claim 18, wherein the control channel recursively publishes metadata regarding the data packages.

21. The system of claim 12, further comprising an archiving service to archive the data packages.

22. A method comprising:

receiving a request to replay data bundled into data packages having a logical ordering assigned to the data packages before being stored in an archive, wherein the request includes a logical bound on the data to be replayed and identifies a format for the data subscriber;

retrieving, from the archive, data consistent with the logical bound; and

transforming the data packages into the loadable format identified in the request to replay the data.

23. The method of claim 22, wherein data packages after the logical bound are ignored.

24. The method of claim 22, further comprising:

receiving a selection of an archiving strategy; and

archiving the data in accordance with the archiving strategy.

25. The method of claim 24, wherein the archiving strategy is a dimension-based archiving strategy.

26. The method of claim 22, wherein the data packages are compressed using columnar compression.

27. The method of claim 22, wherein the data packages include metadata with each providing summary data, quality data, index data, or checksum data.