Dynamic distributed generation of data representations from highly condensed data

• US 9,811,575 B2
• Filed: 03/30/2017
• Issued: 11/07/2017
• Est. Priority Date: 03/30/2016
• Status: Active Grant

• Current Assignee: SAS Institute Incorporated
• Original Assignee: SAS Institute Incorporated
• Inventors: Christian, Stacey Michelle, Whitcher, Michael Stephen, McAlister, Donald Kent, Hanna, Phillip Elliot
• Primary Examiner(s): Yen, Syling
• Application Number:
  US15/474,832
• Publication Number:
  US 20170286509A1
• Time in Patent Office: 222 Days
• Field of Search: 707791, 707802, 707803, 707809, 707812, 7079991
• CPC Class Codes: G06F 16/1744 using compression, e.g. sparse files
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/24532 of parallel queries
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/27 Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence

Abstract

An apparatus includes a processor and storage storing instructions causing the processor to store, at a node device of a grid of node devices, a data block of a data blob, receive data blob metadata indicative of an organization of data items within the data blob, receive a command to generate, from the data block, at least one row of a representation portion of a plurality of representation portions from which a 2D representation of the data blob is to be generated, use the data blob metadata and a node block map indicative of which data items of the data block are required to generate the representation portion to derive one or more transforms to be performed with the data block to generate the at least one row of the representation portion, and perform the one or more transforms with the data block to generate the at least one row.

Claims

1. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmit, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receive, at the node device, a command to store the data block from the coordinating device;

receive, at the node device, the data block from the at least one source device;

receive, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and

transmit, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

2. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;

select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

3. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

receive, at the node device, at least a portion of the representation data from the coordinating device;

receive, at the node device, a command from the coordinating device to store the portion of the representation data; and

store, at the node device, the received portion of the representation data.

4. The apparatus of claim 1, wherein the processor is caused to perform operations comprising:

use the node bock map and an identifier of the requesting device or an entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

5. The apparatus of claim 1, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

6. The apparatus of claim 1, wherein a transform of the one or more transforms is selected from a group consisting of aggregation of data items of the data blob, averaging of data items of the data blob, sorting data items of the data blob, calculating a value of at least one column of the at least one row from a value of at least one other column of the at least one row, decompressing data items of the data blob, and decrypting data items of the data blob.

7. An apparatus comprising a processor and a storage to store instructions that, when executed by the processor, cause the processor to perform operations comprising:

store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmit, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receive, at the node device, a command to store the data block from the coordinating device;

receive, at the node device, the data block from the at least one source device;

receive, at the node device, another command to generate at least one more row of the representation portion from the data block;

retrieve first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;

use the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;

store, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and

transmit, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

8. The apparatus of claim 7, wherein the processor is caused to perform operations comprising:

store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;

select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

9. The apparatus of claim 7, wherein the processor is caused to perform operations comprising:

use the node bock map and an identifier of the requesting device or an entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

10. The apparatus of claim 7, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

11. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmit, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receive, at the node device, a command to store the data block from the coordinating device;

receive, at the node device, the data block from the at least one source device;

receive, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and

transmit, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

12. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;

select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

13. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

receive, at the node device, at least a portion of the representation data from the coordinating device;

receive, at the node device, a command from the coordinating device to store the portion of the representation data; and

store, at the node device, the received portion of the representation data.

14. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising:

use the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

15. The computer-program product of claim 11, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

16. The computer-program product of claim 15, wherein the at least one operation is selected from a group consisting of aggregation of values of the at least one representation portion, averaging of data items of the at least one representation portion, sorting data items of the at least one representation portion, calculating a value of at least one column of the at least one row from a value of at least one other column of the at least one row, and transmitting the at least one row of the representation data to the grid of node devices for storage.

17. A computer-program product tangibly embodied in a non-transitory machine-readable storage medium, the computer-program product including instructions operable to cause a processor to perform operations comprising:

store, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receive, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receive, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

use the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

perform, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmit, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receive, at the node device, a command to store the data block from the coordinating device;

receive, at the node device, the data block from the at least one source device;

receive, at the node device, another command to generate at least one more row of the representation portion from the data block;

retrieve first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;

use the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;

store, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and

transmit, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

18. The computer-program product of claim 17, wherein the processor is caused to perform operations comprising:

store, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by the processor to perform a different type of transform from others of the multiple reader routines;

select one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

use the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

19. The computer-program product of claim 17, wherein the processor is caused to perform operations comprising:

use the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refrain from using the portion of the data items to generate the at least one row of the representation portion.

20. The computer-program product of claim 17, wherein the processor is caused to perform operations comprising use representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

21. A computer-implemented method comprising:

storing, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receiving, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receiving, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

using the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

performing, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmitting, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receiving, at the node device, a command to store the data block from the coordinating device;

receiving, at the node device, the data block from the at least one source device;

receiving, at the node device, the command to generate at least one row of the representation portion from the coordinating device; and

transmitting, from the node device, the at least one row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one row of the representation data from at least one row of each representation portion of the plurality of representation portions.

22. The computer-implemented method of claim 21, comprising:

storing, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by a processor of the node device to perform a different type of transform from others of the multiple reader routines;

selecting one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

using the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

23. The computer-implemented method of claim 21, comprising:

receiving, at the node device, at least a portion of the representation data from the coordinating device;

receiving, at the node device, a command from the coordinating device to store the portion of the representation data; and

storing, at the node device, the received portion of the representation data.

24. The computer-implemented method of claim 21, comprising:

using the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refraining from using the portion of the data items to generate the at least one row of the representation portion.

25. The computer-implemented method of claim 21, comprising using representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.

26. The computer-implemented method of claim 21, wherein:

the data blob comprises a hypercube that employs an indexing scheme of at least 3 dimensions; and

the data blob does not comprise the 2D representation of data provided by the representation data as a slice of the hypercube.

27. A computer-implemented method comprising:

storing, at a node device of a grid of node devices, a data block of a plurality of data blocks of a data blob, wherein;

the plurality of data blocks of the data blob are distributed among the grid of node devices to be stored by the grid of node devices;

the data blob comprises multiple data items; and

each data block of the plurality of data blocks comprises a subset of the multiple data items;

receiving, at the node device, data blob metadata that comprises an indication of organization of the multiple data items within the data blob;

receiving, at the node device, a command that originates from a requesting device to generate at least one row of a representation portion of a plurality of representation portions from the data block, wherein;

a representation data that provides a two-dimensional (2D) representation of data items of the data blob is to be assembled from the plurality of representation portions;

the plurality of representation portions is to be generated from a subset of data blocks of the plurality of data blocks by a subset of node devices of the grid of node devices that store the subset of data blocks; and

the subset of node devices comprises the node device;

using the data blob metadata and a node block map of a plurality of node block maps to derive one or more transforms to be performed at the node device with the data block to generate the at least one row of the representation portion, wherein;

the plurality of node block maps is provided to the subset of node devices to generate the plurality of representation portions from the subset of data blocks; and

the node block map comprises an indication of which data items of the data block are required to generate the representation portion, what data values are to be generated from the required data items, and an organization of the generated data values within the representation portion;

performing, at the node device, the one or more transforms with the data block to generate the at least one row of the representation portion;

transmitting, from the node device to a coordinating device of the grid of node devices, node data comprising an indication of availability of at least storage resources of the node device to enable the coordinating device to coordinate provision of the plurality of data blocks by at least one source device to the grid of node devices, and to coordinate storage of the plurality of data blocks by the grid of node devices;

receiving, at the node device, a command to store the data block from the coordinating device;

receiving, at the node device, the data block from the at least one source device;

receiving, at the node device, another command to generate at least one more row of the representation portion from the data block;

retrieving first state data indicative of a state of the node device after the performance of the one or more transforms to generate the at least one row of the representation portion;

using the first state data to prepare the node device for another performance of the one or more transforms with the data block to generate the at least one more row of the representation portion;

storing, at the node device, second state data indicative of a state of the node device after the another performance of the one or more transforms to generate the at least one more row of the representation portion; and

transmitting, from the node device, the at least one more row of the representation portion to the coordinating device to enable generation, at the coordinating device, of at least one more row of the representation data from at least one more row of each representation portion of the plurality of representation portions.

28. The computer-implemented method of claim 27, comprising:

storing, at the node device, multiple reader routines, wherein each reader routine comprises other instructions executable by a processor of the node device to perform a different type of transform from others of the multiple reader routines;

selecting one of the multiple reader routines to use in deriving and performing the one or more transforms based on the node block map; and

using the selected reader routine to derive and perform the one or more transforms with the data block to generate the at least one row of the representation portion.

29. The computer-implemented method of claim 27, comprising:

using the node bock map and an identifier of the requesting device or a entity associated with the requesting device to determine whether the requesting device or the entity is authorized to access data items of the data block that are required to generate the at least one row of the representation portion; and

in response to a determination that either the requesting device or the entity is not authorized to access a portion of the data items of the data block that are required to generate the at least one row of the representation portion, refraining from using the portion of the data items to generate the at least one row of the representation portion.

30. The computer-implemented method of claim 27, comprising using representation metadata comprising an indication of features of the representation data and the data blob metadata to derive at least one operation to perform with at least one row of at least one representation portion of the plurality of representation portions to generate the at least one row of the representation data.