Efficient Data Processing

• US 20190318023A1
• Filed: 04/16/2018
• Published: 10/17/2019
• Est. Priority Date: 04/16/2018
• Status: Active Grant

• Current Assignee: Computational Systems Incorporated (Emerson Electric Company)
• Original Assignee: Computational Systems Incorporated (Emerson Electric Company)
• Inventors: Hayzen, Anthony J.
• Granted Patent: US 10,713,232 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06
• CPC Class Codes: G06F 16/214 Database migration support
• CPC Class Codes: G06F 16/2272 Management thereof
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 7/08 Sorting, i.e. grouping record carriers in numerical or other ordered sequence according to the classification of at least some of the information they carry by merging two or more sets of carriers in ordered sequence G06F7/16

Abstract

A method for improving the operation of a computer by transferring data from an SQL database in a storage unit to a memory by copying the data from the SQL database to a binary file in the storage unit, where the binary file having a data structure and a data type. A data file having the data structure and the data type is configured within the memory, and the binary file in the storage unit is copied to the data file in the memory with a single read/write operation.

Claims

1. A method for improving the operation of a computer by transferring data from an SQL database in a storage unit to a memory, the method comprising the steps of:

copying the data from the SQL database to a binary file in the storage unit, the binary file having a data structure and a data type,

configuring within the memory a data file having the data structure and the data type, and

copying the binary file in the storage unit to the data file in the memory with a single read/write operation.

2. The method of claim 1, wherein the data structure comprises records and fields, with a number of records and a number of fields for each record.

3. The method of claim 1, wherein the data type comprises at least one of real, integer, and word length.

4. A method for reading the data file produced by the method of claim 1, when the records of the data file are sorted, comprising:

reading through the entire data file only once to find record positions for data containing predetermined data characteristics,

creating a pointer table in the memory, the pointer table including a single record position for a first instance of each of the predetermined data characteristics within the data file, and

reading the data file starting at the record position in the pointer table that corresponds to a desired one of the predetermined data characteristics.

5. A method for reading the data file produced by the method of claim 1 to locate desired data, when the records of the data file are unsorted, comprising:

reading through the entire data file only once to find record positions for data containing predetermined data characteristics,

creating a first pointer table in the memory, the first pointer table including a single record position for a first instance of each of the predetermined data characteristics within the data file,

creating a second pointer table in the memory, the second pointer table including, in association with the record position for the first instance of each of the predetermined data characteristics, a subsequent record position for a next subsequent instance of each of the predetermined data characteristics within the data file, and

reading the data file starting at the record position in the first pointer table that corresponds to a desired one of the predetermined data characteristics, and if the data in the record position does not contain the desired data, iteratively reading the data file at the record position associated with the next subsequent instance in the second pointer table, until the desired data is found.

6. A method for correlating variables in the data file produced by the method of claim 1, the method comprising the steps of:

reading the data file a first time to determine averages x and y for each x and y variable pair to be correlated,

reading the data file a second time to determine for each x_i and y_i variable the following three values, where i represents the number of variables,

Σ

_i(x_i−

x)(y_i−

y),

√

{square root over (Σ

_i(x_i−

x)²)},

√

{square root over (Σ

_i(y_i−

y)²)},

storing each of the three values in a cell of an array designated for the x_i and y_i variable pair, and

using the three values in the array for each combination of x_i and y_i variables, computing the correlation value r according to the formula

7. A method for improving the operation of a computer by reading a data file of records in a memory, the method comprising:

reading through the entire data file only once to find record positions for data containing predetermined data characteristics,

creating a pointer table in the memory, the pointer table including a single record position for a first instance of each of the predetermined data characteristics within the data file, and

reading the data file starting at the record position in the pointer table that corresponds to a desired one of the predetermined data characteristics.

8. The method of claim 7, wherein each record in the data file comprises a plurality of fields, and all the fields are sorted according to a predetermined order.

9. The method of claim 7, wherein each record of the data file comprises a plurality of fields, all the fields are sorted according to a predetermined order, and the pointer table includes, for each of the fields, a single record position associated with a first instance for each of the predetermined data characteristics for each field.

10. A method for improving the operation of a computer by reading a data file of unsorted records to locate desired data, the method comprising:

reading through the entire data file only once to find record positions for data containing predetermined data characteristics,

creating a first pointer table in the memory, the first pointer table including a single record position for a first instance of each of the predetermined data characteristics within the data file,

creating a second pointer table in the memory, the second pointer table including, in association with the record position for the first instance of each of the predetermined data characteristics, a subsequent record position for a next subsequent instance of each of the predetermined data characteristics within the data file, and

reading the data file starting at the record position in the first pointer table that corresponds to a desired one of the predetermined data characteristics, and if the data in the record position does not contain the desired data, iteratively reading the data file at the record position associated with the next subsequent instance in the second pointer table, until the desired data is found.

11. The method of claim 11, wherein a first pointer table and a second pointer table are created for each field within the data file.

12. A method for improving the operation of a computer by correlating variables in a data file, the method comprising the steps of:

reading the data file a first time to determine averages x and y for each x and y variable pair to be correlated,

reading the data file a second time to determine for each x_i and y_i variable the following three values, where i represents the number of variables,

Σ

_i(x_i−

x)(y_i−

y),

√

{square root over (Σ

_i(x_i−

x)²)},

√

{square root over (Σ

_i(y_i−

y)²)},

storing each of the three values in a cell of an array designated for the x_i and y_i variable pair, and

using the three values in the array for each combination of x_i and y_i variables, computing the correlation value r according to the formula,

13. The method of claim 12, wherein the correlation value r is only calculated for unique combinations of the x_i and y_i variables.