Performing direct data manipulation on a storage device

• US 8,768,898 B1
• Filed: 04/26/2007
• Issued: 07/01/2014
• Est. Priority Date: 04/26/2007
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Trimmer, Don Alvin, Yadav, Sandeep, Singh, Pratap
• Primary Examiner(s): Young, Kevin L
• Application Number:
  US11/740,471
• Time in Patent Office: 2,623 Days
• Field of Search: None
• US Class Current: 707/693
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/1727 Details of free space management performed by the file system saving storage space on storage systems G06F3/0608; management of blocks in storage devices G06F3/064
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067

Abstract

A method and system for performing data manipulation on a storage device is disclosed. A data manipulation command is created on a computing device, wherein the computing device is separate from the storage device. The computing device is a client or a server that requests services of a storage system to store data on a storage medium. The computing device and the storage device are connected over a network. The computing device executes a host application, and its data is stored on the medium. The computing device issues a command to the storage device to be performed on the data. The storage device executes the command and sends the result to the computing device. As a result, the data is not sent to the computing device for manipulation.

Claims

1. A method, comprising:

receiving at a storage server a repacking command in a network storage communication protocol, from a client device executing a host application, the repacking command having been generated at the client device by the client device having incorporated a plurality of ordered instructions into the repacking command, the repacking command identifying a source data set and a destination data set stored at the storage server, the source data set occupying a plurality of file system level segments,

wherein each of the plurality of instructions includes instructions to manipulate the destination data set using at least one of the plurality of file system level segments of the source data set at the storage server, and each instruction includes information identifying at least one of the file system level segments in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments of the source data set from the repacking command;

wherein the destination data set is a copy of the source data set and the repacking command causes the storage server to reclaim unused storage space, such that the destination data set occupies a smaller number of segments than the source data set;

accessing at the storage server the plurality of instructions in the repacking command;

reordering the plurality of instructions in the storage server to improve a performance of the plurality of instructions;

manipulating the destination data set using at least one of the file system level segments of the source data set at the storage server in response to receiving the repacking command, without transferring data of the destination data set or the source data set to the client device, by executing the plurality of instructions in the storage server, wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination data set, removing the file system level segment from the destination data set stored in the storage server; and

sending a result of the repacking command from the storage server to the client device.

2. The method according to claim 1, wherein the storage server is a file server.

3. The method according to claim 1, wherein the plurality of instructions are included in the repacking command in an order, the method further comprising:

determining a reordering of the plurality of instructions by the storage server to improve a performance of the plurality of instructions, prior to executing the instructions according to the reordering.

4. The method according to claim 1, further comprising sending a result of the copying step from the storage server to the client.

5. The method of claim 1, wherein the source data set contains a first number of file system level segments, and wherein the repacking command is for repacking the destination data set at the storage server into a second number of file system level segments smaller than the first number of file system level segments.

6. The method according to claim 1, wherein each of said file system level segments of the source data set contains data of a portion of the source data set, but not data of the entire source data set.

7. A method, comprising:

receiving at a storage server a repacking command from a client over a network, the repacking command having been generated by the client and including information identifying a source file at the storage server and a destination file stored at the storage server, the repacking command further including a plurality of ordered instructions;

wherein each of the plurality of instructions includes instructions to manipulate the destination file using at least one of a plurality of file system level segments for the storage server, and each instruction includes information identifying at least one of the plurality of file system level segments to copy from the source file to the destination file in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments from the repacking command;

wherein the destination file is a copy of the source file and the repacking command causes the storage server to reclaim unused storage space, such that the destination file occupies a smaller number of segments than the source file; and

at the storage server, copying file system level segments identified in the information from the source file to the destination file, in response to the plurality of instructions, without transferring any portion of the source file or the destination file to the client, wherein the copying comprises;

accessing at the storage server the plurality of instructions in the repacking command;

reordering the plurality of instructions in the storage server to improve a performance of the plurality of instructions;

manipulating the destination file using at least one of the file system level segments of the source file at the storage server in response to receiving the repacking command, without transferring data of the destination file or the source file to the client device, by executing the plurality of instructions in the storage server, wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source file from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source file to the destination file stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination file, removing the file system level segment from the destination file stored in the storage server; and

sending a result of the repacking command from the storage server to the client.

8. The method according to claim 7, wherein:

the repacking command further includes an exclusion list of file system level segments in the source file to exclude from the destination file; and

the copying step includes copying all file system level segments from the source file to the destination file except for those file system level segments in the exclusion list.

9. The method according to claim 7, wherein:

the copying step includes unpacking the information from the repacking command.

10. The method according to claim 7, wherein the repacking command further includes data to be inserted into the destination file, the method further comprising the step of:

writing the data to the destination file, the writing step being performed by the storage server.

11. The method according to claim 7, wherein the storage server includes a storage manager and a storage media and the copying step includes:

requesting the source file from the storage media by the storage manager;

retrieving the source file by the storage media;

reading the source file by the storage manager;

copying the file system level segments indicated in a list of file system level segments included in the repacking command by the storage manager from the source file to the destination file; and

writing the destination file from the storage manager to the storage media.

12. The method according to claim 7, wherein the plurality of instructions are included in the repacking command in an order, the method further comprising:

determining, by the storage server, a reordering of the plurality of instructions in the repacking command, and executing the plurality of instructions in the storage server according to the reordering.

13. The method according to claim 12, wherein the repacking command includes parameters to execute the repacking command on the storage server, and the performing step includes:

accessing the parameters after unpacking the repacking command; and

using the parameters to execute the repacking command.

14. The method according to claim 12, wherein the repacking command specifies the data set, and the manipulating step includes:

retrieving the source data set from the storage media; and

execute the repacking command on the destination data set.

15. The method according to claim 12, wherein the storage server includes a storage manager and a data storage media, the manipulating step including:

manipulating the source data set by the storage manager, the storage manager accessing the storage media to manipulate the source data set.

16. The method according to claim 15, wherein the manipulating step includes:

requesting the source data set from the storage media by the storage manager;

retrieving the source data set by the storage media;

reading the source data set by the storage manager;

manipulating the destination data set from the file by the storage manager according to the plurality of instructions in the repacking command;

writing the manipulated destination data set back to the storage media by the storage manager; and

sending a result of the manipulating step from the storage manager to the client.

17. The method of claim 7, the repacking command having been generated by the client prior to receiving the repacking command being received at the storage server, by the client having autonomously determined the plurality of instructions to incorporate into the repacking command, without the plurality of instructions having been specified for the repacking command by a human user.

18. A storage server, comprising:

a storage adapter through which to access a set of storage media of the storage server;

a network adapter through which to receive a network file system protocol repacking command over a network from a client, the repacking command having been generated by the client to include information identifying a source file at the storage server and a destination file stored at the storage server, the repacking command further including a plurality of ordered instructions;

wherein each of the plurality of instructions includes instructions to manipulate the destination file using at least one of a plurality of file system level segments for the storage server, and each instruction includes information identifying at least one of the plurality of file system level segments for the storage server to copy from the source file to the destination file in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments from the repacking command,

wherein the destination file is a copy of the source file and the repacking command causes the storage server to reclaim unused storage space, such that the destination file occupies a smaller number of segments than the source file; and

a processor coupled to the storage adapter and the network adapter and configured to copy file system level segments identified in the information in the command from the source file to the destination file at the storage server, in response to the plurality of instructions without transferring any portion of the source file or the destination file to the client, wherein the copying comprises;

access at the storage server the plurality of instructions in the repacking command;

reorder the plurality of instructions in the storage server to improve a performance of the plurality of instructions;

manipulate the destination file using at least one of the file system level segments of the source file at the storage server in response to receiving the repacking command, without transferring data of the destination file or the source file to the client, by executing the plurality of instructions in the storage server, wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination data set, removing the file system level segment from the destination data set stored in the storage server; and

send a result of copying the file system level segments identified in the list to said client.

19. The storage server according to claim 18, further comprising a storage manager and a data storage media, wherein said storage manager is configured to:

receive the identified information;

request the source file from said storage media;

read the source file from said storage media;

copy the list of file system level segments from the source file to the destination file;

write the destination file to said storage media; and

send the result of the copying to said client.

20. The storage server according to claim 19, wherein:

said repacking command further includes information identifying data to be inserted into the destination file; and

said processor is further configured to insert the identified data to the destination file.

21. A storage server, comprising:

a storage adapter through which to access a set of storage media of the storage server;

a network adapter through which to receive a repacking command identifying a source data set and a destination data set stored at the storage server from a client via a network, the repacking command having been generated by the client, by the client having incorporated a plurality of ordered instructions into the repacking command, the source data set occupying a plurality of file system level segments, and each instruction includes information identifying at least one of the file system level segments in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments of the source data set from the repacking command,

wherein the destination data set is a copy of the source data set and the repacking command causes the storage server to reclaim unused storage space, such that the destination data set occupies a smaller number of segments than the source data set; and

a processor coupled to the storage adapter and the network adapter and configured to control the storage server to provide network storage services to the client, including controlling the storage server to

accessing the plurality of instructions in the repacking command;

reordering the plurality of instructions in the storage server to improve a execution of the repacking command; and

manipulating the destination data set using at least one of the file system level segments of the source data set at the storage server in response to the repacking command, without transferring data of the destination data set or the source data set to the client, by executing the plurality of instructions, wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination data set, removing the file system level segment from the destination data set stored in the storage server; and

sending a result of the repacking command from the storage server to the client device.

22. The storage server according to claim 21, wherein the storage server is a file server.

23. The storage server according to claim 21, wherein the repacking command includes parameters to execute the repacking command, and the processor is further configured to:

access the parameters after unpacking the repacking command; and

use the parameters to execute the repacking command.

24. The storage server according to claim 21, wherein the command specifies the data set, and the processor executes the repacking command by:

retrieving the source data set from the storage media; and

executing the repacking command on the destination data set.

25. The storage server according to claim 21, wherein the processor is configured to:

request the source data set from the storage media by the storage manager;

retrieve the source data set by the storage media;

read the source data set by the storage manager;

manipulate the data from the destination data set by the storage manager according to the repacking command;

write the repacked data set back to the storage media by the storage manager; and

send a result of the repacking command from the storage manager to the client.

26. The storage server according to claim 21, wherein at least one of the plurality of instructions is from the group consisting of:

copying a file, reading a file, writing a file, resizing a file, deleting a file, renaming a file, and relocating a file.

27. The storage server of claim 21, wherein the source data set occupies a first number of file system level segments at the storage server, and wherein the repacking command is for repacking the destination data set at the storage server into a second number of file system level segments smaller than the first number of file system level segments.

28. The storage server according to claim 21, wherein the manipulating comprises:

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

if the repacking command includes an instruction to remove a file system level segment of destination data set;

remove the file system level segment from the destination data set stored in the storage server.

29. A method, comprising:

creating in a client device a network storage communication protocol repacking command for a storage server that provides network storage services to the client device, by the client device incorporating into the repacking command a plurality of ordered instructions to be executed by the storage server, the repacking command identifying a source data set and a destination data set managed by the storage server, the source data set occupying a plurality of file system level segments at the storage server,

wherein each instruction includes information identifying at least one of the file system level segments in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments of the source data set from the repacking command;

wherein the destination data set is a copy of the source data set and the repacking command causes the storage server to reclaim unused storage space, such that the destination data set occupies a smaller number of segments than the source data set;

sending the repacking command from the client device over a network to the storage server, to cause the storage server to access the plurality of instructions in the repacking command and manipulate the destination data set using at least one of the file system level segments according to the plurality of instructions;

wherein the plurality of instructions are reordered in the storage server to improve a performance of the plurality of instructions;

wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination data set, removing the file system level segment from the destination data set stored in the storage server; and

receiving at the client device from the storage server a result of the storage server having executed the repacking command.

30. The method according to claim 29, wherein the storage server is a file server.

31. The method according to claim 29, wherein creating the repacking command further comprises including parameters in the data manipulation repacking command, for use by the storage server in executing the repacking command.

32. The method according to claim 29, wherein the repacking command specifies a destination data set on which the data manipulation command is to be performed.

33. The method according to claim 29, wherein at least one of the plurality of instructions is from the group consisting of:

copying a file, reading a file, writing a file, resizing a file, deleting a file, renaming a file, and relocating a file.

34. The method of claim 29, wherein the source data set occupies a first number of file system level segments at the storage server, and wherein the repacking command is for repacking the destination data set at the storage server into a second number of file system level segments smaller than the first number of file system level segments.

35. A method, comprising:

receiving at a storage server a repacking command in accordance with a network storage communication protocol, from a client device executing a host application, the repacking command having been generated by the client device, by the client device having incorporated a plurality of ordered instructions into the repacking command, the repacking command identifying a source data set and a destination data set stored at the storage server, the source data set occupying a plurality of file system level segments at the storage server,

wherein each of the plurality of instructions includes instructions to manipulate the destination data set using at least one of the plurality of file system level segments of the source data set at the storage server, and each instruction includes information identifying at least one of the file system level segments in such a manner that no mapping function is needed to unambiguously ascertain each of said file system level segments of the source data set from the repacking command;

wherein the destination data set is a copy of the source data set and the repacking command causes the storage server to reclaim unused storage space, such that the destination data set occupies a smaller number of segments than the source data set;

accessing by the storage server the plurality of instructions in the repacking command;

determining by the storage server a reordering of the plurality of instructions relative to an ordering of the instructions in the repacking command;

reordering the plurality of instructions in the storage server according to the reordering to improve a performance of the plurality of instructions; and

manipulating the destination data set using at least one of the file system level segments of the source data set at the storage server in response to receiving the repacking command, without transferring data of the destination data set or the source data set to the client device, by executing the plurality of instructions in the storage server according to the reordering, wherein the manipulating comprises;

for each instruction;

retrieving the at least one of the file system level segments of the source data set from a storage media of the storage server;

inserting data of the at least one of the file system level segments of the source data set to the destination data set stored in the storage server; and

in an event the repacking command includes an instruction to remove a file system level segment of destination data set, removing the file system level segment from the destination data set stored in the storage server; and

sending a result of the repacking command from the storage server to the client device.

36. The method of claim 35, wherein the source data set occupies a first number of file system level segments at the storage server, and wherein the repacking command is for repacking the destination data set into a second number of file system level segments smaller than the first number of file system level segments.