On the fly summarization of file walk data

• US 7,630,994 B1
• Filed: 03/12/2004
• Issued: 12/08/2009
• Est. Priority Date: 03/12/2004
• Status: Active Grant

• Current Assignee: NetApp, Inc.
• Original Assignee: NetApp, Inc.
• Inventors: Thompson, Timothy J., Swartzlander, Benjamin, Deshmukh, Vijay
• Primary Examiner(s): Cottingham; John R.
• Assistant Examiner(s): WILSON, KIMBERLY LOVEL
• Application Number:
  US10/799,861
• Time in Patent Office: 2,097 Days
• Field of Search: 707/101
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1827 Management specifically adapted to NAS management of storage area networks [SAN] G06F3/067
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface

Abstract

According to an embodiment of the invention, a directory on a storage server is scanned. A number of children of the directory is determined, and the number is added to a reference count. A child is scanned to collect information about the child, and the information is combined into a summary of the directory. After the child is scanned, the reference count is reduced by one. According to another embodiment of the invention, the storage server is scanned by an agent separate from the storage server.

Claims

1. An apparatus comprising:

a storage server, coupled to a network, having a mass storage device;

a multi-appliance management application (MMA) coupled to the network to manage the storage server; and

a server computer including a first agent coupled to the storage server and the MMA via the network, the first agent to scan a first subset of a directory structure of the storage server to collect information about a file stored on the storage server, and to combine information collected into a summary of a directory in which the file is located, the summary being accessible to the MMA, wherein the server computer, the MMA, and the storage server are separate devices, wherein the first agent operates independently of the storage server and the MMA, and wherein the first agent uses a file system different from a file system that the storage server uses, wherein the first agent is configured to scan and summarize the first subset of the directory structure of the storage server by

scanning a directory in the storage server to identify child nodes in the directory;

scanning each of the child nodes in the directory in the storage server to collect information about the child nodes;

combining the collected information about the child nodes into the summary of the directory, wherein said combining occurs concurrently with said scanning, for different items of information; and

storing the summary of the directory in a storage facility accessible to the MMA;

the first agent including a directory thread to scan directories and a separate file thread to scan files, wherein the first agent is one of a plurality of agents on the network, a second agent of the plurality of agents being configured to scan and summarize a second subset of the directory structure of the storage server, wherein the second agent operates independently of the first agent, the storage server and the MMA.

2. The apparatus of claim 1, further comprising a graphical user interface (GUI) coupled to the MMA.

3. The apparatus of claim 2, further comprising a database coupled to the MMA, the database to store the summary.

4. The apparatus of claim 3, wherein the summary can be is retrievable via the GUI.

5. The apparatus of claim 1, wherein the first agent uses Common Internet File System (CIFS) or Network File System (NFS).

6. The apparatus of claim 1, wherein the summary is in the form of a histogram.

7. The apparatus of claim 1, wherein the summary of the directory comprises an indication of per-user total storage usage for the directory.

8. The apparatus of claim 1, wherein the summary of the directory comprises an indication of average last access time, for a plurality of child nodes in the directory.

9. A method comprising:

causing a first agent device, of a plurality of independently operable agent devices, to scan a file and directory structure of a storage server to collect information about files maintained by the storage server, including using the first agent to scan and summarize a first subset of a plurality of directories in the storage server, the first agent device including a directory thread to scan directories and a separate file thread to scan files, wherein scanning the file and directory structure of the storage server by the first agent includes

scanning a directory in the storage server to identify child nodes in the directory;

scanning each of the child nodes in the directory in the storage server to collect information about the child nodes;

combining, by the first agent device, information collected into a summary of a directory under which the files are stored, including combining the collected information about the child nodes into the summary of the directory, wherein said combining occurs concurrently with said scanning, for different items of information;

sending the summary from the first agent device to a multi-appliance management application (MMA), wherein the first agent device, the MMA, and the storage server are separate devices, and wherein the first agent device uses a file system different from a file system that the storage server uses; and

causing a second agent device, of the plurality of independently operable agent devices, to scan and summarize a second subset of the plurality of directories in the storage server, wherein the first agent device operates independently of the storage server and the MMA, and wherein the second agent device operates independently of the first agent device, the storage server and the MMA.

10. The method of claim 9, wherein the MMA sends the summary to a database server, which stores the summary as a table or a histogram.

11. The method of claim 9, wherein the first agent device uses Common Internet File System (CIFS) or Network File System (NFS).

12. A method comprising the steps of:

a) using a directory thread in a first agent, of a plurality of agents on a network, to scan and summarize a first subset of a plurality of directories in a storage server on the network, including using the first agent to scan a directory in the storage server to identify contents of the directory, the first agent being implemented in a server computer, separate from the storage server, on the network, wherein the first agent uses a file system different from a file system that the storage server uses;

b) determining, by the first agent, a number of child nodes in the directory in the storage server and incrementing a reference count by the number;

c) scanning, by the first agent, a child node in the directory in the storage server to collect information about the child node, wherein said scanning includes using a file thread in the first agent to scan and determine characteristics of a file in the directory;

d) combining, by the first agent, the collected information about the child node into a summary of the directory, the summary including a histogram;

e) decrementing the reference count after scanning the child node;

f) repeating said steps c) through e) for each of one or more additional child nodes in the directory until the reference count equals a predetermined value, wherein said combining occurs concurrently with said scanning, for different items of information;

g) storing, by the first agent, the summary of the directory in a storage facility accessible to a multi-appliance management application (MMA) configured to manage the storage server, wherein the first agent operates independently of the storage server and the MMA; and

h) using a second agent of the plurality of agents on the network, to scan and summarize a second subset of the plurality of directories in the storage server, wherein the second agent operates independently of the first agent, the storage server and the MMA.

13. The method of claim 12, wherein the summary of the directory comprises an indication of per-user total storage usage for the directory.

14. The method of claim 12, wherein the summary of the directory comprises an indication of average last access time, for a plurality of child nodes in the directory.

15. A computer system comprising:

a processor;

a communication interface, coupled to the processor, through which to communicate with a storage server on a network; and

a first agent, of a plurality of agents on the network, which configures the processor to execute a process that includes a set of steps, including

scanning and summarizing a first subset of a plurality of directories in the storage server, including

scanning a directory in the storage server to identify child nodes in the directory;

scanning each of the child nodes in the directory in the storage server to collect information about the child nodes; and

combining the collected information about the child nodes into a summary of the directory, wherein said combining occurs concurrently with said scanning, for different items of information;

storing the summary of the directory in a storage facility accessible to a storage management application configured to manage the storage server, wherein the agent is operable independently of the storage server and the storage management application,

wherein the first agent uses a file system different from a file system that the storage server uses; and

wherein the first agent includes a directory thread to scan a directory in the storage server to identify contents of the directory and a file thread to scan and determine characteristics of a file in the directory, wherein the first agent device operates independently of the storage server and the storage management application; and

a second agent of the plurality of agents configured to scan and summarize a second subset of a plurality of directories in the storage server.

16. The computer system of claim 15, wherein the summary of the directory is in the form of a histogram.

17. The computer system of claim 15, wherein the summary of the directory comprises an indication of per-user total storage usage for the directory.

18. The computer system of claim 15, wherein the summary of the directory comprises an indication of average last access time, for a plurality of child nodes in the directory.