Automated data flows using flow-based data processor blocks

• US 10,541,930 B2
• Filed: 05/07/2018
• Issued: 01/21/2020
• Est. Priority Date: 08/28/2015
• Status: Active Grant

• Current Assignee: Buurst Incorporated
• Original Assignee: SoftNas Operating Inc.
• Inventors: Braddy, Rick Gene, Ferrentino, Pasqualino
• Primary Examiner(s): Winder, Patrice L
• Application Number:
  US15/973,278
• Publication Number:
  US 20180254989A1
• Time in Patent Office: 624 Days
• CPC Class Codes: H04L 43/0829 Packet loss
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/14 using software, i.e. software packages network security related monitoring H04L63/1408
• CPC Class Codes: H04L 43/50 Testing arrangements
• CPC Class Codes: H04L 47/25 with rate being modified by the source upon detecting a change of network conditions
• CPC Class Codes: H04L 47/32 by discarding or delaying data units, e.g. packets or frames
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/56 Provisioning of proxy services store-and-forward switching systems in data switching networks H04L12/54
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561
• CPC Class Codes: H04L 69/16 Implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP]

Abstract

Systems and methods for a flow-based data processing can begin with receiving a selection of a plurality of data processor blocks and a configuration mapping corresponding to the inputs and outputs of each data processor block. At a first compute node associated with a selected data source, an automated data flow can be initiated, wherein the automated data flow is generated from the configuration mapping and comprises a continuous sequence of one or more of the selected plurality of data processor blocks. Data from the selected data source is ingested into the automated data flow and is transformed by the continuous sequence of data processor blocks. The transformed ingested data is transmitted from the automated data flow to a second compute node associated with a selected data destination, via a data accelerator.

Claims

1. A method comprising:

receiving a selection of a plurality of data processor blocks and a configuration mapping, the configuration mapping corresponding to one or more inputs and outputs of each data processor block;

initiating, at a first compute node associated with a selected data source, an automated data flow, the automated data flow generated from the configuration mapping and comprising a continuous sequence of the selected plurality of data processor blocks arranged in an order specified by the input and output relationships contained in the configuration mapping;

detecting, using one or more of the selected plurality of data processor blocks, any changes in the data stored in the selected data source;

ingesting, from the selected data source and into a first data processor block of the automated data flow, the detected changes in the data stored in the selected data source;

transforming the ingested data with the continuous sequence of data processor blocks, such that;

the first data processor block applies a first transformation operation to the ingested data, thereby generating and outputting a first transformed data;

a second data processor block of the continuous sequence receives the first transformed data output by the first data processor block and applies a second transformation operation to the first transformed data, thereby generating and outputting a second transformed data, wherein;

the second data processor block is coupled to the first data processor block according to the input and output relationships contained in the configuration mapping;

the second transformation operation is different from the first transformation operation; and

neither the first nor second transformation operation comprises data compression or acceleration;

transmitting, from the automated data flow via a data accelerator and to a second compute node associated with a selected data destination, the transformed ingested data, wherein the second compute node thereby maintains a synchronized cloud copy of the data stored in the selected data source.

2. The method of claim 1, further comprising receiving an input specifying one or more of the selected data source, the selected data destination, and a data transform parameter.

3. The method of claim 1, wherein the selection of the plurality of data processor blocks and the configuration mapping comprises a selection of a pre-defined template.

4. The method of claim 1, wherein the selection of the plurality of data processor blocks and the configuration mapping comprises receiving a plurality of user inputs into a drag and drop graphical user interface (GUI).

5. The method of claim 1, wherein the plurality of data processor blocks are flow-based programming processor blocks.

6. The method of claim 1, wherein the plurality of data processor blocks are Apache NiFi FlowFile processors.

7. The method of claim 1, wherein the automated data flow comprises:

the first data processor block at an input end of the continuous sequence;

a final data processor block at an output end of the continuous sequence; and

one or more data transform data processor blocks disposed between the input and output ends of the continuous sequence.

8. The method of claim 7, wherein the first data processor block is configured to:

couple to the selected data source and ingest the data;

normalize the ingested data into a data object; and

transmit the data object to the one or more data transform data processor blocks.

9. The method of claim 1, wherein:

one or more of the first compute node, the second compute node, and the selected data source comprises one or more of;

a cloud NAS (Network-Attached Storage), a NAS, a SAN (Storage Area Network), a SQL (Structured Query Language) database, a local environment, and a file server.

10. The method of claim 1, wherein the automated data flow is stored as a pre-defined template for a one-to-one data migration, wherein the pre-defined template comprises at least the plurality of data processor blocks and a configuration mapping for the plurality of data processor blocks.

11. The method of claim 1, wherein the automated data flow is stored as a pre-defined template comprising at least the plurality of data processor blocks and a configuration mapping for the plurality of data processor blocks, and wherein the automated data flow comprises one or more of:

a one-to-many data transfer;

a many-to-one data transfer;

a spoke-and-hub data transfer; and

a graph topology based data transfer.

12. The method of claim 1, wherein the data accelerator comprises a data storage accelerator and a data transport accelerator and wherein the data acceleration is based at least in part on one or more of measured network latency characteristics and measured network packet loss characteristics.

13. The method of claim 12, wherein the data accelerator acts as a proxy to intercept one or more TCP packets transmitted by the data source and to perform a data accelerated transmission of the one or more TCP packets to the data destination.

14. The method of claim 13, wherein the one or more TCP packets are intercepted based on at least their port number.

15. The method of claim 1, wherein the data accelerator performs data acceleration by:

initiating a generation of a plurality of gamblers;

initiating a gambler of the generation;

transmitting one or more packets for the gambler over a network;

in response to a determination the gambler is finished, waiting for the network to discharge; and

in response to a determination the generation is finished, determining a gain-loss of the generation and adjust the plurality of gamblers.

16. The method of claim 15, further comprising:

in response to a determination that the gambler is not finished, transmitting additional packets over the network.

17. The method of claim 15, further comprising:

in response to a determination that the generation is not finished, incrementing to a next gambler.

18. The method of claim 17, further comprising

initiating the next gambler of the generation;

transmitting one or more packets for the next gambler over the network;

in response to a determination that the gambler is finished, waiting for the network to discharge; and

in response to a determination that the gambler is finished, determining a gain-loss of the generation and adjusting the plurality of gamblers.

19. The method of claim 15, wherein the one or more packets are transmitting at a gambler specified transmission rate.

20. The method of claim 15, wherein the plurality of gamblers are adjusted by incrementing a transmission rate to be closer to an inherent network loss rate.

21. The method of claim 20, wherein the inherent network loss rate is determined by transmitting a probe loss packet after the network is discharged.

22. The method of claim 15, wherein the data accelerator comprises one or more hardware circuits for performing the data acceleration.