Method and apparatus for optimizing data transfers utilizing machine learning

• US 10,348,828 B2
• Filed: 06/20/2016
• Issued: 07/09/2019
• Est. Priority Date: 06/20/2016
• Status: Active Grant

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Inventors: Bahl, Rohit, Williams, Stephen, Parandekar, Harsh
• Primary Examiner(s): Hussain, Imad
• Application Number:
  US15/187,594
• Publication Number:
  US 20170366611A1
• Time in Patent Office: 1,114 Days
• Field of Search: 709217
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: H04L 41/16 using machine learning or artificial intelligence
• CPC Class Codes: H04L 43/0852 Delays
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 67/1097 for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests

Abstract

In one embodiment, a method includes receiving at a network device comprising a data transfer optimization module, input identifying a source directory comprising data and a target directory at a storage device, splitting the data into a plurality of data groups for transfer to the storage device, transmitting the data groups concurrently to the storage device on data paths wherein transfer times of the data groups are monitored, receiving at the data transfer optimization module, identification of a data group with a longest transfer time, splitting the data group with the longest transfer time, and placing the data from the data group into at least two of the data groups for transfer to the storage device. An apparatus and logic are also disclosed herein.

Claims

1. A method comprising:

receiving at a network device comprising a data transfer optimization module, input identifying a source directory comprising data and a target directory at a storage device;

splitting said data into a plurality of data groups at the data transfer optimization module for transfer to the storage device;

transmitting the data groups concurrently to the storage device on data paths wherein completed transfer times of the data groups are monitored at one or more nodes located within one or more of the data paths;

receiving at the data transfer optimization module, identification of a data group with a longest transfer time; and

splitting the data group with said longest transfer time at the data transfer optimization module and placing said data from the data group into at least two of the data groups for future transfer to the storage device.

2. The method of claim 1 further comprising receiving at the network device, a seed value indicating an initial number of the data groups.

3. The method of claim 2 wherein the number of the data groups may exceed said seed value for further optimization at the data transfer optimization module.

4. The method of claim 1 wherein said source directory is located at a build server and comprises a tree hierarchy with an unbalanced structure, each of the data groups associated with at least one subtree of said tree hierarchy.

5. The method of claim 1 wherein said data transfer times are monitored by at least one observer node operable to identify said transfer times for the data groups.

6. The method of claim 5 wherein the observer node comprises a learning server module operable to store performance metrics comprising said transfer times in performance storage.

7. The method of claim 6 wherein the performance storage is operable to classify and categorize said performance metrics and store optimization recommendations for the data transfer optimization module based on said transfer times.

8. The method of claim 1 further comprising repeating splitting the data group with said longest transfer time until said longest transfer time is sufficiently close to said transfer times for other of the data groups to indicate optimization of the data transfer.

9. The method of claim 1 wherein the data transfer optimization module utilizes machine learning based on performance metrics received from learning server modules within said data paths for use in optimization of periodic data transfer from the network device to the storage device, wherein said machine learning for a data build utilizes data from a previous similarly classified data build transfer.

10. An apparatus comprising:

a processor operable to process input identifying a source directory comprising data and a target directory at a storage device, and information identifying a data group with a longest completed transfer time;

a learning client module comprising logic encoded on one or more non-transitory computer readable media for execution and when executed operable to split said data into a plurality of data groups for transfer from the apparatus to the storage device, split the data group with said longest transfer time, and place said data from the data group into at least two of the data groups for transfer to the storage device; and

a plurality of ports operable to transmit the data groups concurrently from the apparatus to the storage device on data paths wherein transfer times of the data groups are monitored at one or more nodes located within one or more of the data paths.

11. The apparatus of claim 10 wherein the learning client module is configured for receiving a seed value indicating an initial number of the data groups.

12. The apparatus of claim 11 wherein the number of the data groups may exceed said seed value for further optimization at the learning client module.

13. The apparatus of claim 10 wherein said source directory is located at a build server and comprises a tree with an unbalanced hierarchy structure, each subtree of the tree associated with at least one of the data groups.

14. The apparatus of claim 10 wherein said data transfer times are monitored by at least one observer node operable to identify said transfer times for the data groups.

15. The apparatus of claim 14 wherein the observer node comprises a learning server module operable to store performance metrics comprising said transfer times in storage and wherein the performance storage is operable to store optimization recommendations for the data transfer optimization module based on said transfer times.

16. The apparatus of claim 10 wherein the learning client module is further configured to repeat splitting of the data group with said longest transfer time until said longest transfer time is sufficiently close to said transfer times for other of the data groups to indicate optimization of the data transfer.

17. The apparatus of claim 10 wherein the learning client module utilizes machine learning based on performance metrics received from learning server modules within said data paths.

18. Logic encoded on one or more non-transitory computer readable media for execution and when executed operable to:

process input identifying a source directory comprising data at a server and a target directory at a storage device;

split said data into a plurality of data groups for transfer from the server to the storage device;

initiate transfer of the data groups concurrently from the server to the storage device on data paths wherein completed transfer times of the data groups are monitored at one or more nodes located within one or more of the data paths;

process information identifying a data group with a longest transfer time; and

split the data group with said longest transfer time and place said data from the data group into at least two of the data groups for future transfer to the storage device.

19. The logic of claim 18 further operable to create the data groups based on a seed value indicating an initial number of the data groups.

20. The logic of claim 18 wherein said data transfer times are monitored by at least one observer node operable to identify a transfer time for each of data groups, the observer node comprising a learning server module operable to store performance metrics comprising said transfer times in performance storage.