System and method for processing large amounts of transactional data

• US 8,301,798 B2
• Filed: 06/12/2009
• Issued: 10/30/2012
• Est. Priority Date: 06/12/2009
• Status: Active Grant

• Current Assignee: Barclays Capital Incorporated (Barclays PLC)
• Original Assignee: Barclays Capital Incorporated (Barclays PLC)
• Inventors: Agrawal, Amit, Mond, Amar
• Primary Examiner(s): Swearingen, Jeffrey R
• Application Number:
  US12/483,983
• Publication Number:
  US 20100318674A1
• Time in Patent Office: 1,236 Days
• Field of Search: None
• US Class Current: 709/234
• CPC Class Codes: G06Q 10/00 Administration; Management
• CPC Class Codes: G06Q 40/00 Finance; Insurance; Tax strategies; Processing of corporate or income taxes

Abstract

A system including a reference data server that stores a first set of data used in the plurality of processes, a load balancer that reconfigures the transactional data, a first stage processing system and a second stage processing system. The first stage processing system includes one or more first processing modules that execute at least one process of a first set of the plurality of processes on the reconfigured transactional data to generate first stage processed transactional data, each of the one or more first processing modules comprising an in-memory cache that stores a second set of data used in the at least one process, and a first stage data storage system that stores the first stage processed transactional data. The second stage processing system includes one or more second processing modules that execute at least one process of a second set of the plurality of processes on the first stage processed transactional data to generate second stage processed transactional data, each of the one or more second processing modules comprising an in-memory cache that dynamically stores a third set of data related to the at least one process, and a second stage data storage system that stores the second stage processed transactional data.

Claims

1. A system comprising:

a configuration service system that generates information related to configuration of a plurality of specific instances of a generic process to be performed on transactional data, each of the specific instances comprising a plurality of processes;

a reference data server that stores a first set of data used in the generic process;

a first stage processing system comprising;

one or more first processing modules that, for each specific instance, execute at least one process of the plurality of processes that make up the specific instance based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

each of the one or more first processing modules comprising a first task executor, the first task executors being re-initialized based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one process so as to generate first stage processed transactional data, each of the one or more first processing modules comprising an in-memory cache that stores a second set of data used in the at least one process; and

a first stage data storage system that stores the first stage processed transactional data;

a second stage processing system comprising;

one or more second processing modules that, for each specific instance, execute at least one process of the plurality of processes on the first stage processed transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

each of the one or more second processing modules comprising a second task executor, the second task executors being re-initialized based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one process so as to generate second stage processed transactional data, each of the one or more second processing modules comprising an in-memory cache that stores a third set of data related to the at least one process; and

a second stage data storage system that stores the second stage processed transactional data;

a load balancer that distributes processing load within each of the first and second stage processing systems; and

a flow controller that manages flow of data between the first and second stage processing systems.

2. The system of claim 1, wherein each of the one or more first and second processing modules comprise a rule executor that applies an associated set of rules.

3. The system of claim 1, wherein distribution of the processing load is performed by the load balancer by eliminating interdependencies within the transactional data.

4. The system of claim 1, wherein the first set of data comprises reference data required for processing the transactional data.

5. The system of claim 1, wherein each of the one or more first and second processing modules comprises one or more adapters for interfacing with various types of data.

6. The system of claim 5, wherein the types of data are selected from the following:

database data, messaging system data, file system data, and e-mail system data.

7. The system of claim 1, wherein the first and second stage data storage systems are selected from the following type of data storage systems:

databases, messaging systems, files systems and e-mail systems.

8. A method of processing large amounts of transactional data, comprising the steps of:

generating at a configuration service system information related to configuration of a plurality of specific instances of a generic process to be performed on the transactional data, each of the specific instances comprising a plurality of processes;

storing a first set of data used in the generic process at a reference data server;

at one or more processing modules of a first stage processing system, for each specific instance, executing at least one first process of the plurality of processes that make up the specific instance on the transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

the step of executing comprising;

at each of the one or more processing modules of the first stage processing system, storing a second set of data used in the at least one first process at an in-memory cache; and

at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one first process so as to generate first stage processed transactional data;

at one or more processing modules of a second stage processing system, for each specific instance, executing at least one second process of the plurality of processes that make up the specific instance on the first stage processed transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

the step of executing comprising;

at each of the one or more processing modules of the second stage processing system, storing a third set of data used in the at least one second process at an in-memory cache; and

at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one second, process so as to generate second stage processed transactional data;

managing using a flow controller flow of data between the first and second stage processing systems; and

distributing using a load balancer processing load within each of the first and second stage processing systems.

9. The method of claim 8, wherein the steps of executing are performed using a multithreading technique.

10. The method of claim 8, wherein the steps of executing comprise application by the first and second processing modules of an associated set of rules.

11. The method of claim 8, wherein the step of distributing comprises at least one of eliminating and reducing interdependencies within the transactional data.

12. The method of claim 8, further comprising modifying format of data at data adapters associated with each of the first and second processing modules so that the first and second processing modules are capable of interfacing with the reference data server.

13. The method of claim 12, wherein the step of modifying comprises converting the data to one or more of the following formats:

database data, messaging system data, file system data and e-mail system data.

14. A computer system comprising one or more processors, one or more communication devices operatively connected to the one or more processors, and one or more computer-readable media containing computer-readable instructions executable on the one or more processors, the instructions being related to performance of a method for processing large amounts of transactional data, comprising the steps of:

generating at a configuration service system information related to configuration of a plurality of specific instances of a generic process to be performed on the transactional data, each of the specific instances comprising a plurality of processes;

storing a first set of data used in the generic process at a reference data server;

at one or more processing modules of a first stage processing system, for each specific instance, executing at least one first process of the plurality of processes that make up the specific instance on the transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

the step of executing comprising;

at each of the one or more processing modules of the first stage processing system, storing a second set of data used in the at least one first process at an in-memory cache; and

at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one first process so as to generate first stage processed transactional data;

at one or more processing modules of a second stage processing system, for each specific instance, executing at least one second process of the plurality of processes that make up the specific instance on the first stage processed transactional data based on a corresponding one of a plurality of instance specific configurations generated by the configuration service system;

the step of executing comprising;

at each of the one or more processing modules of the second stage processing system, storing a third set of data used in the at least one second process at an in-memory cache; and

at each of the one or more processing modules, re-initializing one or more first task executors based on the corresponding one of the plurality of instance specific configurations to perform one or more tasks associated with the at least one second process so as to generate second stage processed transactional data;

managing using a flow controller flow of data between the first and second stage processing systems; and

distributing using a load balancer processing load within each of the first and second stage processing systems.

15. The computer system of claim 14, wherein the steps of executing are performed using a multithreading technique.

16. The computer system of claim 14, wherein the information generated by the configuration service system comprises rules related to the plurality of processes, and the steps of executing comprise application by the first and second processing modules of an associated set of rules.

17. The computer system of claim 14, wherein the step of distributing comprises at least one of eliminating and reducing interdependencies within the transactional data.

18. The computer system of claim 14, further comprising modifying format of data at data adapters associated with each of the first and second processing modules so that the first and second processing modules are capable of interfacing with the reference data server.

19. The computer system of claim 18, wherein the step of modifying comprises converting the data to one or more of the following formats:

database data, messaging system data, file system data and e-mail system data.