DATA INGESTION TO GENERATE LAYERED DATASET INTERRELATIONS TO FORM A SYSTEM OF NETWORKED COLLABORATIVE DATASETS

• US 20180314705A1
• Filed: 03/20/2018
• Published: 11/01/2018
• Est. Priority Date: 06/19/2016
• Status: Active Grant

• Current Assignee: Data.World, Inc.
• Original Assignee: Data.World, Inc.
• Inventors: Griffith, David Lee, Jacob, Bryon Kristen, Reynolds, Shad William
• Granted Patent: US 11,016,931 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/116 Details of conversion of file system types or formats
• CPC Class Codes: G06F 16/148 File search processing
• CPC Class Codes: G06F 16/185 Hierarchical storage management [HSM] systems, e.g. file migration or policies thereof details of archiving G06F16/11

Abstract

Various embodiments relate generally to data science and data analysis, and computer software and systems to provide an interface between repositories of disparate datasets and computing machine-based entities that seek access to the datasets, and, more specifically, to a computing and data storage platform that facilitates consolidation of one or more datasets, whereby data ingestion is performed to form data representing layered data files and data arrangements to facilitate, for example, interrelations among a system of networked collaborative datasets. In some examples, a method may include forming a first layer data file and a second layer data file, assigning addressable identifiers to uniquely identify units of data and data units to facilitate the linking of data, and implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request for a collaborative dataset.

Claims

1. A method comprising:

receiving data representing a set of data formatted in a data arrangement;

adapting the data arrangement for the set of data to form a dataset having a first data format;

forming one or more first layer data files to reference the set of data associated with a second data format in which units of data in the set of data are configured to link with other layer data files;

forming one or more second layer data files that include a subset of data based on the set of data in the second data format, data units of the subset of data in the second data format being configured to link to the units of data in the first data format;

assigning addressable identifiers to uniquely identify the units of data and the data units to facilitate linking data between the set of data in the first format and the dataset in the second data format; and

implementing selectively one or more of a unit of data and a data unit as a function of a context of a data access request.

2. The method of claim 1 further comprising:

forming a plurality of hierarchical layer data files including a hierarchy of subsets of data each of which is configured to link to the units of data in the first data format,

wherein the hierarchy of subsets of data are configured to link to original data of the set of data to provide access to the original data.

3. The method of claim 2 wherein implementing selectively the one or more of the unit of data and the data unit comprises:

determining data representing the context of the data access request;

selecting one or more files of the first layer data files, the second layer data files, and the hierarchical layer data files based on the context; and

omitting at least a group of layer files a function of the data access request.

4. The method of claim 2 further comprising:

associating a first subset of nodes and a second subset of nodes to the dataset;

associating at least a third subset of nodes to the subset of data, the third subset of nodes being linked to either the first subset of nodes or the second subset of nodes, or both; and

associating a plurality of subsets of nodes to the hierarchy of subsets of data; and

implementing selectively one or more nodes of the first subset of nodes, the second subset of nodes, the third subset of nodes, and the plurality of subsets of nodes as a function of the context of the data access request.

5. The method of claim 1 wherein forming the one or more first layer data files comprises:

transforming the set of data from a first format to a dataset having the second data format in which the data of the dataset includes linked data; and

associating a first subset of nodes and a second subset of nodes to the dataset, at least one node from each of the first subset and the second subset of nodes identifying a unit of data.

6. The method of claim 5 wherein associating the first subset of nodes and the second subset of nodes to the dataset comprises:

associating row nodes and column nodes, respectively, to the dataset.

7. The method of claim 1 wherein forming the one or more second layer data files comprises:

forming the subset of data based on the set of data; and

associating at least a third subset of nodes to the first subset of nodes and the second subset of nodes.

8. The method of claim 7 wherein forming the subset of data based on the set of data comprises:

deriving data of the subset of data.

9. The method of claim 8 wherein deriving data of the subset of data comprises:

deriving a column of derived data representing a categorical variable.

10. The method of claim 1 wherein assigning addressable identifiers comprises:

assigning addressable identifiers to uniquely identify nodes in the first subset and the second subset of nodes to facilitate linking data between the set of data in the first format and the dataset in the second data format.

11. The method of claim 10 wherein assigning addressable identifiers comprises:

implementing an Internationalized Resource Identifier (“

IRI”

) identify a node.

12. The method of claim 1 wherein adapting the data arrangement to form the dataset having the first data format comprises:

forming a tabular data arrangement format as the first data format; and

associating the set of data in the tabular data arrangement.

13. The method of claim 1 wherein forming the one or more first layer data files and forming the one or more second layer data files comprises:

transforming the one or more first layer data files and the one or more second layer data files into an atomized dataset format.

14. The method of claim 13 wherein the atomized dataset format include linked data points.

15. The method of claim 14 wherein linked data points comprise triples.

16. The method of claim 15 wherein at least one triple of the triples are formatted to comply with a Resource Description Framework (“

RDF”

) data model.

17. The method of claim 1 further comprising:

formatting the second data format to store the data of the dataset in the graph database.

18. The method of claim 17 wherein the data access request comprises a query.

19. An apparatus comprising:

a memory including executable instructions; and

a processor, responsive to executing the instructions, is configured to;

receive data representing a set of data formatted in a data arrangement;

adapt the data arrangement for the set of data to form a dataset having a first data format;

form one or more first layer data files that include the set of data disposed in a second data format in which units of data in the set of data are configured to link with other layer data files;

form one or more second layer data files that include a subset of data based on the set of data in the second data format, data units of the subset of data in the second data format being configured to link to the units of data in the first data format;

assign addressable identifiers to uniquely identify the units of data and the data units to facilitate linking data between the set of data in the first format and the dataset in the second data format; and

implement selectively one or more of a unit of data and a data unit as a function of a context of a data access request.

20. The apparatus of claim 19, wherein the processor is further configured to:

transform the one or more first layer data files and the one or more first layer data files into an atomized dataset format including linked data points comprising triplets formatted to comply with a Resource Description Framework (“

RDF”

) data model.