Systems and methods for expedited large file processing

• US 10,191,952 B1
• Filed: 02/26/2018
• Issued: 01/29/2019
• Est. Priority Date: 07/25/2017
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Bhatt, Japan
• Primary Examiner(s): Cao, Phuong Thao
• Application Number:
  US15/905,163
• Publication Number:
  US 20190034495A1
• Time in Patent Office: 337 Days
• Field of Search: 707737
• CPC Class Codes: G06F 16/188 Virtual file systems
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2458 Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
• CPC Class Codes: G06F 16/285 Clustering or classification

Abstract

A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform steps of a method for processing a large file. The system may receive record data comprising a plurality of records having an identification value in a common field having a data format. The system may determine a plurality of focus values based on the data format and create a plurality of virtual processing units based on the plurality of focus values. Each of the plurality of virtual processing units may process a sub-group of the plurality of records that corresponds to the focus value associated with the respective virtual processing unit.

Claims

1. A system for processing a large file, comprising:

one or more processors; and

a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to;

receive record data comprising a plurality of records, each of the plurality of records comprising an identification value in a common field, the common field having a data format comprising a sequence of characters and each of the identification values corresponding to a name;

determine, based on a comparison of a size of the record data to a predetermined size threshold, a focus value order of magnitude;

determine, based on the data format and the focus value order of magnitude, a plurality of unique focus values, each of the plurality of focus values corresponding to a sub-group of the plurality of records, wherein the plurality of focus values comprise a set of letters from A to Z with each of the plurality of focus values corresponding to a specified portion of the sequence of characters in the data format;

create a plurality of virtual processing units, each associated with a unique one of the plurality of focus values; and

responsive to searching the record data, process, by each of the plurality of virtual processing units, the respective sub-group of the plurality of records that corresponds to the focus value associated with the respective virtual processing unit.

2. The system of claim 1, wherein the data format of the common field comprises a sequence of characters associated with a letter data type such that each character of the sequence of characters comprises either a letter or a null.

3. The system of claim 1, wherein the sequence of characters has a predetermined maximum length.

4. The system of claim 2, wherein the specified portion of the sequence of characters of the data format is a last letter of the sequence of characters.

5. The system of claim 2, wherein the specified portion of the sequence of characters of the data format is a first letter of the sequence of characters.

6. The system of claim 4, wherein each of the plurality of focus values corresponds to a sub-group of the plurality of records that include names sharing a common last letter that corresponds to the respective focus value.

7. The system of claim 1, wherein each of the virtual processing units comprises a virtual machine or container.

8. The system of claim 1, wherein each of the plurality of virtual processing units is configured to search the record data to identify the respective sub-group of the plurality of records that corresponds to the respective virtual processing unit.

9. The system of claim 1, wherein the processing by each of the plurality of virtual processing units occurs in parallel.

10. A system for processing a large file, comprising:

one or more processors; and

a memory in communication with the one or more processors and storing instructions that, when executed by the one or more processors, are configured to cause the system to;

receive record data comprising a plurality of records, each of the plurality of records comprising an identification value in a common field, the common field having a data format comprising a sequence of characters and each of the identification values corresponding to a name;

determine, based on a comparison of a size of the record data to a predetermined size threshold, a focus value order of magnitude;

determine, based on the data format and the focus value order of magnitude, a plurality of focus values comprising at least a first focus value and a second focus value, each of the plurality of focus values corresponding to a sub-group of the plurality of records such that the first focus value corresponds to a first sub-group of the plurality of records and the second focus value corresponds to a second sub-group of the plurality of records, wherein the plurality of focus values comprise a set of unique two-character sequences where each unique two-character sequence comprises a first letter followed by a second letter with each of the plurality of focus values corresponding to a specified portion of the sequence of characters in the data format;

create a first virtual processing unit for processing the first sub-group of the plurality of records corresponding to the first focus value;

create a second virtual processing unit for processing the second sub-group of the plurality of records corresponding to the second focus value; and

process the first and second sub-groups of the plurality of records via the first and second virtual processing units, respectively.

11. The system of claim 10, wherein the sequence of characters has a predetermined maximum length.

12. The system of claim 11, wherein each character of the sequence of characters is associated with a letter data type such that each character of the sequence of characters comprises either a letter or a null.

13. The system of claim 12, wherein each of the plurality of focus values corresponds to a sub-group of the plurality of records that include names sharing a common two-character sequence that corresponds to the respective focus value.

14. The system of claim 13, wherein the specified portion of the data format is the last two letters of the sequence of characters.

15. The system of claim 13, wherein the specified portion of the data format is the first two letters of the sequence of characters.

16. A method of processing a large file comprising:

receiving record data comprising a plurality of records, each of the plurality of records comprising an identification value in a common field, the common field having a data format comprising a multi-digit sequence;

determining, based on a comparison of a size of the record data to a predetermined size threshold, a focus value order of magnitude;

determining, based on the data format and the focus value order of magnitude, a plurality of unique focus values, each of the plurality of focus values corresponding to a sub-group of the plurality of records, wherein the plurality of unique focus values comprise a set of unique two-digit numbers ranging from 00 to 99 with each of the plurality of focus values corresponding to a specified portion of the multi-digit sequence in the data format;

creating a plurality of virtual processing units, each associated with a unique one of the plurality of focus values; and

responsive to searching the record data, processing, by each of the plurality of virtual processing units, the respective sub-group of the plurality of records that corresponds to the focus value associated with the respective virtual processing unit.

17. The method of claim 16, wherein each of the identification values in the common field correspond to a social security number.

18. The method of claim 17, wherein the data format is a nine-digit number.

19. The method of claim 18, wherein the specified portion of the multi-digit sequence of the data format is the last two numbers of the multi-digit sequence.

20. The method of claim 17, further comprising determining a number of records in each sub-group of the plurality of records in response to searching the record data.