Real-time processing of event-based streaming with NoSQL

• US 10,445,322 B1
• Filed: 11/29/2018
• Issued: 10/15/2019
• Est. Priority Date: 11/29/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Kommera, Raveender, Gloier, Nathan E., Sathraboina, Rajesh, Sathish Kumaran, Vairavelu, Meda, Venu, Zhao, Jingjie
• Primary Examiner(s): Cheema, Azam M
• Application Number:
  US16/204,128
• Time in Patent Office: 320 Days
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/284 Relational databases
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

Systems, methods, and media for real-time processing of event-based streaming with NoSQL databases are provided. For example, disclosed embodiments may include receiving an event associated with an update to an entry in a first database. Also, disclose embodiments may include identifying, based on the event, relevant data in the first database associated with the update. Further, disclose embodiments may include generating a plurality of queries to the first database to request the identified relevant data. Disclose embodiments may also include receiving, in response to the plurality of queries, raw data from the first database. Additionally, disclose embodiments may include processing the raw data to be compatible with a second database. Disclose embodiments may also include storing the processed raw data in the second database. And, disclosed embodiments may include receiving data requests for the second database to distribute the processed data to at least one computer terminal.

Claims

1. A computing system, comprising:

at least one processor;

a storage medium storing instructions that, when executed, configure the at least one processor to perform operations comprising;

receiving an event associated with an update to an entry in a first database, wherein the first database is a relational database;

adding the event to an event queue;

identifying, based on the event, relevant data in the first database associated with the update by retrieving the event from the event queue;

generating a plurality of queries to the first database to request the identified relevant data;

receiving, in response to the plurality of queries, raw data from the first database;

processing the raw data to be compatible with a second database by transforming the raw data into a new format, wherein the second database is a nonrelational database;

storing the processed raw data in the second database; and

receiving data requests for the second database to distribute the processed data to at least one computer terminal.

2. The computing system of claim 1, wherein generating a plurality of queries comprises generating a plurality of concurrent calls to the first database.

3. The computing system of claim 1, wherein receiving raw data from the first database comprises receiving a stream of data at an associated cloud-based storage device.

4. The computing system of claim 1, wherein processing the raw data utilizes a multithreaded processing architecture.

5. The computing system of claim 4, wherein the multithreaded processing architecture simultaneously computes and persist the raw data.

6. The computing system of claim 1, wherein the relational database is a SQL database.

7. The computing system of claim 1, wherein the nonrelational database is a wide-column-store NoSQL database.

8. The computing system of claim 1, wherein the new format is a JavaScript Object Notation (JSON) format.

9. The computing system of claim 1, wherein processing the raw data further comprises manipulating the raw data to be organized based on a field associated with the raw data.

10. The computing system of claim 1, wherein the event corresponds to an update to a field of a data entry in a relational database.

11. The computing system of claim 1, wherein the event corresponds to an update to data associated with a mortgage application.

12. The computing system of claim 11, wherein the update to the data associated with the mortgage application comprises credit report results for an applicant of the mortgage application.

13. The computing system of claim 12, wherein processing the data requests for the second database causes an alert to be generated at a terminal that instructs a user to contact the applicant of the mortgage application responsive to determining that the credit report results match predetermined criteria.

14. The computing system of claim 11, wherein processing the data requests for the second database facilitates providing an alert at a user terminal responsive to determining that the update to the data associated with the mortgage application matches predetermined criteria.

15. A method for real-time processing of event-based streaming with NoSQL databases comprising:

receiving, by at least one processor, an event associated with an update to an entry in a first database, wherein the first database is a relational database;

adding, by at least one processor, the event to an event queue;

identifying, by the at least one processor and based on the event, relevant data in the first database associated with the update by retrieving the event from the event queue;

generating, by the at least one processor, a plurality of queries to the first database to request the identified relevant data;

receiving, by the at least one processor and in response to the plurality of queries, raw data from the first database;

processing, by the at least one processor, the raw data to be compatible with a second database by transforming the raw data into a new format, wherein the second database is a nonrelational database;

storing, by the at least one processor, the processed raw data in the second database; and

processing, by the at least one processor, data requests for the second database to distribute the processed data to at least one computer terminal.

16. A method for real-time processing of event-based streaming with NoSQL databases comprising:

receiving an event associated with an update to an entry in a relational database;

adding the event to an event queue;

identifying, based on the event, relevant data in the first database associated with the update by retrieving the event from the event queue;

generating a plurality of queries to the relational database to request the identified relevant data;

receiving, in response to the plurality of queries, data from the relational database;

updating nonrelational data in a nonrelational database based on the received data; and

processing data requests for the nonrelational database to distribute the updated nonrelational data to at least one computer terminal, wherein updating the nonrelational data comprises transforming the received data into a new format compatible with the nonrelational data base.