Method and system for aggregating and managing data from disparate sources in consolidated storage
First Claim
1. A distributed computing platform comprising:
- a first computer system that includes one or more data hardware processors coupled to a transceiver, wherein the transceiver is configured to;
receive a first plurality of electronic data transaction messages from a plurality of different exchange computer systems, each one of the plurality of electronic data transaction messages including data fields that reflect how a corresponding one of the plurality of different exchange computer systems processed a corresponding electronic order submitted thereto, andreceive a second plurality of electronic data messages from a plurality of different participant computer systems that each submit orders to different ones of the plurality of different exchange computer systems for execution thereby, wherein the first plurality of electronic data transaction messages and the second plurality of electronic data messages are in different data formats;
wherein the one or more hardware processors is configured to;
generate a combined first data set that is in JavaScript Object Notation (JSON) format by parsing securities transactions from the first plurality of electronic data transaction messages and the second plurality of electronic data messages to identify specific data fields for each security transaction, and then mapping those data fields to data elements of the combined first data set,partition the combined first data set into a plurality files, wherein orders that are part of the same order chain are determined to be partitioned into the same file,encrypt the plurality of files using a first encryption key, andtransmit the encrypted plurality of files for storage on another computer system;
a cloud-based computer system that includes a plurality of computing nodes, with each comprising hardware processing resources, each one of the plurality of computing nodes executing one or more virtual machines, the cloud-based computer system providing for dynamic scaling of the computing nodes that are allocated for handling processing and/or storage of data communicated from the first computer system;
a secure storage system under control of at least one of the plurality of computing nodes of the cloud-based computer system, the cloud-based computer system comprising;
a first data store configured to accept data that is written thereto, wherein once data is written to the first data store it can only be read, but not deleted or modified until being automatically deleted in response to expiration of a retention period,a plurality of participant data stores that are each respectively associated with a corresponding one of a plurality of participant accounts of a plurality of participants; and
wherein the hardware processing resources of the cloud-based computer system are configured to;
receive the transmitted encrypted plurality of files from the first computer system,write the encrypted plurality of files that are in JSON to the first data store, wherein the plurality of files that are written to the first data store are readable a plurality of times after the initial writing, but after the initial writing of the corresponding plurality of files cannot be deleted or modified until being automatically deleted in response to expiration of a retention period,receive a search query for data that is contained in the first data set that is spread across the plurality of files that are in JSON,execute the search query in parallel across multiple ones of the plurality of computing nodes and on each of those nodes decrypt a corresponding subset of the plurality of encrypted files that are stored to the first data store, wherein individual files of the plurality of encrypted files are assigned to the same computing node for processing, wherein the files are only decrypted in memory of the corresponding computing node and not persisted to non-volatile storage,determine, for each computing node that is executing the search query, whether data in the unencrypted files is responsive to the search query,compile responsive data to the search query to a result file and encrypt the result file, anddeliver the result file to the participant data store that corresponds to the participant that submitted the search query.
1 Assignment
0 Petitions
Accused Products
Abstract
Non-limiting example embodiments include methods and systems for acquiring private financial data from multiple disparate sources. The private financial data is normalized, aggregated, preferably enhanced, and stored in secure storage. Entitled entities may retrieve selected private financial data from that secure storage efficiently, flexibility, and rapidly. Examples of financial private data include non-liquidity destination related sources of private data as well as liquidity destination related sources. A non-limiting example of a computer-implemented, consolidated, private financial data service is based on a secure, permission-based, aggregated and consolidated data cloud, which enables provision/distribution to one or more authorized parties with legitimate interests selected portions of the consolidated, private financial data.
22 Citations
16 Claims
-
1. A distributed computing platform comprising:
-
a first computer system that includes one or more data hardware processors coupled to a transceiver, wherein the transceiver is configured to; receive a first plurality of electronic data transaction messages from a plurality of different exchange computer systems, each one of the plurality of electronic data transaction messages including data fields that reflect how a corresponding one of the plurality of different exchange computer systems processed a corresponding electronic order submitted thereto, and receive a second plurality of electronic data messages from a plurality of different participant computer systems that each submit orders to different ones of the plurality of different exchange computer systems for execution thereby, wherein the first plurality of electronic data transaction messages and the second plurality of electronic data messages are in different data formats; wherein the one or more hardware processors is configured to; generate a combined first data set that is in JavaScript Object Notation (JSON) format by parsing securities transactions from the first plurality of electronic data transaction messages and the second plurality of electronic data messages to identify specific data fields for each security transaction, and then mapping those data fields to data elements of the combined first data set, partition the combined first data set into a plurality files, wherein orders that are part of the same order chain are determined to be partitioned into the same file, encrypt the plurality of files using a first encryption key, and transmit the encrypted plurality of files for storage on another computer system; a cloud-based computer system that includes a plurality of computing nodes, with each comprising hardware processing resources, each one of the plurality of computing nodes executing one or more virtual machines, the cloud-based computer system providing for dynamic scaling of the computing nodes that are allocated for handling processing and/or storage of data communicated from the first computer system; a secure storage system under control of at least one of the plurality of computing nodes of the cloud-based computer system, the cloud-based computer system comprising; a first data store configured to accept data that is written thereto, wherein once data is written to the first data store it can only be read, but not deleted or modified until being automatically deleted in response to expiration of a retention period, a plurality of participant data stores that are each respectively associated with a corresponding one of a plurality of participant accounts of a plurality of participants; and wherein the hardware processing resources of the cloud-based computer system are configured to; receive the transmitted encrypted plurality of files from the first computer system, write the encrypted plurality of files that are in JSON to the first data store, wherein the plurality of files that are written to the first data store are readable a plurality of times after the initial writing, but after the initial writing of the corresponding plurality of files cannot be deleted or modified until being automatically deleted in response to expiration of a retention period, receive a search query for data that is contained in the first data set that is spread across the plurality of files that are in JSON, execute the search query in parallel across multiple ones of the plurality of computing nodes and on each of those nodes decrypt a corresponding subset of the plurality of encrypted files that are stored to the first data store, wherein individual files of the plurality of encrypted files are assigned to the same computing node for processing, wherein the files are only decrypted in memory of the corresponding computing node and not persisted to non-volatile storage, determine, for each computing node that is executing the search query, whether data in the unencrypted files is responsive to the search query, compile responsive data to the search query to a result file and encrypt the result file, and deliver the result file to the participant data store that corresponds to the participant that submitted the search query. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
-
-
9. A method implemented on a distributed computing platform that includes first computer system, a cloud-based computer system, and a secure storage system under control of at least one of a plurality of computing nodes of the cloud-based computer system, the secure storage system including a first data store configured to accept data that is written thereto, wherein once data is written to the first data store it can only be read, but not deleted or modified until being automatically deleted in response to expiration of a retention period, a plurality of participant data stores that are each respectively associated with a corresponding one of a plurality of participant accounts of a plurality of participants, the method comprising:
-
receiving a first plurality of electronic data transaction messages from a plurality of different exchange computer systems, each one of the plurality of electronic data transaction messages including data fields that reflect how a corresponding one of the plurality of different exchange computer systems processed a corresponding electronic order submitted thereto; receiving a second plurality of electronic data messages from a plurality of different participant computer systems that each submit orders to different ones of the plurality of different exchange computer systems for execution thereby, wherein the first plurality of electronic data transaction messages and the second plurality of electronic data messages are in different data formats; generating a combined first data set that is in JavaScript Object Notation (JSON) format by parsing securities transactions from the first plurality of electronic data transaction messages and the second plurality of electronic data messages to identify specific data fields for each security transaction, and then mapping those data fields to data elements of the combined first data set; partitioning the combined first data set into a plurality files, wherein orders that are part of the same order chain are determined to be partitioned into the same file; encrypting the plurality of files using a first encryption key; writing the encrypted plurality of files that are in JSON to the first data store, wherein the plurality of files that are written to the first data store are readable a plurality of times after the initial writing, but after the initial writing of the corresponding plurality of files cannot be deleted or modified until being automatically deleted in response to expiration of a retention period; executing a search query in parallel across multiple ones of the plurality of computing nodes and on each of those nodes decrypt a corresponding subset of the plurality of encrypted files that are stored to the first data store, wherein individual files of the plurality of encrypted files are assigned to the same computing node for processing, wherein the files are only decrypted in memory of the corresponding computing node and not persisted to non-volatile storage; determining, for each computing node that is executing the search query, whether data in the unencrypted files is responsive to the search query; compiling responsive data to the search query to a result file and encrypt the result file; and delivering the result file to the participant data store that corresponds to the participant that submitted the search query. - View Dependent Claims (10, 11, 12, 13, 14, 15, 16)
-
Specification