AUTOMATIC GENERATION OF STRUCTURED DATA FROM SEMI-STRUCTURED DATA

• US 20170316070A1
• Filed: 05/01/2017
• Published: 11/02/2017
• Est. Priority Date: 04/29/2016
• Status: Active Grant

• Current Assignee: Boomi, Inc. (Boomi TopCo, LP)
• Original Assignee: UNIFI Software, Inc. (Boomi TopCo, LP)
• Inventors: KRISHNAN, Ravikiran, PARASHAR, Ayush, SARKAR, Sudeep
• Granted Patent: US 10,467,244 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/211 Schema design and management
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/86 Mapping to a database

Abstract

A method and system for generating structured data from semi-structured data are provided. The method includes reading a plurality of records from a data file including semi-structured data. Further, the method includes obtaining aligned delimiters in a list for every record that has been read. The method also includes selecting a most occurring delimiter from the list. The method then includes constructing a regular expression using the selected delimiter to split the records into different fields. The method also includes reconstructing the records for the regular expression to fit and split into fields. In addition, the method includes displaying the records split into the fields.

Claims

1. A method for generating structured data from semi-structured data, the method comprising:

reading a plurality of records from a data file comprising semi-structured data;

obtaining aligned delimiters in a list for every record that has been read;

selecting a most occurring delimiter from the list;

constructing a regular expression using the selected delimiter to split the records into different fields;

reconstructing the records for the regular expression to fit and split into fields; and

displaying the records split into the fields.

2. The method as claimed in claim 1, wherein the obtaining comprises:

aligning read records using dynamic time warping.

3. The method as claimed in claim 2, wherein the obtaining further comprises:

identifying matching non-alpha characters as initial delimiters list D1.

4. The method as claimed in claim 3, wherein the obtaining further comprises iteratively performing:

picking a test record (test);

aligning the test record with aligned record (A);

obtaining common delimiters list D;

aligning the common delimiters list D with initial delimiters list D1 using dynamic time warping;

obtaining aligned records delimiters list D2;

adding D2 to a delimiter set S; and

repeating above steps till N number of test records are processed.

5. The method as claimed in claim 1, wherein the semi-structured data comprises at least one of:

end of line delimiter as a new line character;

column delimiter as alpha numeric;

column delimiter as multi-byte or multi-character;

different column delimiter for every column; and

any number of column delimiters in every row.

6. The method as claimed in claim 1 and further comprising:

identifying missing delimiters and missing values.

7. The method as claimed in claim 6, wherein the reconstructing comprises:

reconstructing the records for the regular expression to fit and subsequently filling in NULL for the missing values.

8. A method for generating structured data from semi-structured data, the method comprising:

reading a plurality of records from a data file comprising semi-structured data;

obtaining aligned delimiters in a list for every record that has been read;

selecting a most occurring delimiter from the list;

constructing a regular expression using the selected delimiter to split the records into different fields;

identifying missing delimiters and missing values;

reconstructing the records for the regular expression to fit and split into fields, and subsequently filling in NULL for the missing values; and

displaying the records in a split tabulated form.

9. The method as claimed in claim 8, wherein the obtaining comprises:

aligning read records using dynamic time warping.

10. The method as claimed in claim 9, wherein the obtaining further comprises:

identifying matching non-alpha characters as initial delimiters list D1.

11. The method as claimed in claim 10, wherein the obtaining further comprises iteratively performing:

picking a test record (test);

aligning the test record with aligned record (A);

obtaining common delimiters list D;

aligning the common delimiters list D with initial delimiters list D1 using dynamic time warping;

obtaining aligned records delimiters list D2;

adding D2 to a delimiter set S; and

repeating above steps till N number of test records are processed.

12. The method as claimed in claim 8, wherein the semi-structured data comprises at least one of:

end of line delimiter as a new line character;

column delimiter as alphanumeric;

column delimiter as multi-byte or multi-character;

different column delimiter for every column; and

any number of column delimiters in every row.

13. A system comprising:

a processor; and

a memory coupled to the processor, the memory storing instructions which when executed by the processor cause the system to perform a method for providing information to a user, the method comprising

reading a plurality of records from a data file comprising semi-structured data;

obtaining aligned delimiters in a list for every record that has been read;

selecting a most occurring delimiter from the list;

constructing a regular expression using the selected delimiter to split the records into different fields;

reconstructing the records for the regular expression to fit and split into fields; and

displaying the records split into the fields.

14. The system as claimed in claim 13, wherein the obtaining comprises:

aligning read records using dynamic time warping.

15. The system as claimed in claim 14, wherein the obtaining further comprises:

identifying matching non-alpha characters as initial delimiters list D1.

16. The system as claimed in claim 15, wherein the obtaining further comprises iteratively performing:

picking a test record (test);

aligning the test record with aligned record (A);

obtaining common delimiters list D;

aligning the common delimiters list D with initial delimiters list D1 using dynamic time warping;

obtaining aligned records delimiters list D2;

adding D2 to a delimiter set S; and

repeating above steps till N number of test records are processed.

17. The system as claimed in claim 13, wherein the semi-structured data comprises at least one of:

end of line delimiter as a new line character;

column delimiter as alpha numeric;

column delimiter as multi-byte or multi-character;

different column delimiter for every column; and

any number of column delimiters in every row.

18. The system as claimed in claim 13 and further comprising:

identifying missing delimiters and missing values.

19. The system as claimed in claim 18, wherein the reconstructing comprises:

reconstructing the records for the regular expression to fit and subsequently filling in NULL for the missing values.

20. The system as claimed in claim 13, wherein the reconstructing comprises:

applying symmetric delimiters to split the records into fields.