Systems and methods for distributed system scanning
First Claim
1. A method for handling a failure in a distributed file system, the method comprising:
- for at least one file of a plurality of files, storing a the file across one or more memory devices of a plurality of nodes in a network, at least some of the files distributed in blocks across multiple nodes;
creating a data block location information structure associated with the file and listing, for all of the blocks in the file, the memory devices and physical addresses where the blocks are stored;
creating a consolidated device list which is associated with the file and which comprises an abbreviated listing of each of the one or more memory devices that are listed in the data block location information structure, but which does not include a duplicate listing of a particular memory device for all of the blocks in the file that are stored on the particular memory device, wherein the device list is separate from and includes less information than the data block location information structure;
storing metadata data structures across the plurality of nodes, at least two of the metadata data structures stored on different nodes, each of the metadata data structures referencing information about a corresponding one of the plurality of files, and the metadata data structures comprising the data block location information structure and the consolidated device list of the corresponding file;
detecting that one of the memory devices in the network has failed;
using one or more processors, commanding the nodes to search each consolidated device list stored in each of their respective metadata data structures to determine whether the failed memory device is used to store information referenced by their respective metadata data structures, wherein searching each consolidated device list allows the nodes to make the determination without searching through the data block location information structure, which includes more information than the consolidated device list; and
receiving an indication from the nodes of which metadata data structures reference information stored on the failed device.
12 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods are provided for scanning files and directories in a distributed file system on a network of nodes. The nodes include metadata with attribute information corresponding to files and directories distributed on the nodes. In one embodiment, the files and directories are scanned by commanding the nodes to search their respective metadata for a selected attribute. At least two of the nodes are capable of searching their respective metadata in parallel. In one embodiment, the distributed file system commands the nodes to search for metadata data structures having location information corresponding to a failed device on the network. The metadata data structures identified in the search may then be used to reconstruct lost data that was stored on the failed device.
-
Citations
12 Claims
-
1. A method for handling a failure in a distributed file system, the method comprising:
-
for at least one file of a plurality of files, storing a the file across one or more memory devices of a plurality of nodes in a network, at least some of the files distributed in blocks across multiple nodes; creating a data block location information structure associated with the file and listing, for all of the blocks in the file, the memory devices and physical addresses where the blocks are stored; creating a consolidated device list which is associated with the file and which comprises an abbreviated listing of each of the one or more memory devices that are listed in the data block location information structure, but which does not include a duplicate listing of a particular memory device for all of the blocks in the file that are stored on the particular memory device, wherein the device list is separate from and includes less information than the data block location information structure; storing metadata data structures across the plurality of nodes, at least two of the metadata data structures stored on different nodes, each of the metadata data structures referencing information about a corresponding one of the plurality of files, and the metadata data structures comprising the data block location information structure and the consolidated device list of the corresponding file; detecting that one of the memory devices in the network has failed; using one or more processors, commanding the nodes to search each consolidated device list stored in each of their respective metadata data structures to determine whether the failed memory device is used to store information referenced by their respective metadata data structures, wherein searching each consolidated device list allows the nodes to make the determination without searching through the data block location information structure, which includes more information than the consolidated device list; and receiving an indication from the nodes of which metadata data structures reference information stored on the failed device. - View Dependent Claims (2, 3, 4, 5, 6)
-
-
7. A distributed file system, comprising:
-
a plurality of nodes in a network, each comprising; one or more processors; and at least one memory device, wherein; a plurality of files is stored across the memory devices of the plurality of nodes, at least some of the files distributed in blocks across multiple nodes, and metadata data structures are stored across the memory devices of the plurality of nodes, at least two of the metadata data structures stored on different nodes, each of the metadata data structures referencing information about a corresponding one of the files, and the respective metadata data structures comprising; a data block location information structure associated with the file corresponding to the respective metadata data structure and listing, for all of the blocks in the file, the memory devices and physical addresses where the blocks in the file are stored; and a consolidated device list associated with the file corresponding to the respective metadata data structure and which comprises an abbreviated listing of each of one or more memory devices that are listed in the data block location information structure, wherein the consolidated device list does not include a duplicate listing of a particular memory device for all of the blocks in the file that are stored on the particular memory device, and is separate from and includes less information than the data block location information structure; and wherein the one or more processors of at least one of the nodes are configured to; detect that one of the memory devices in the network has failed; command the nodes to search each consolidated device list stored in each of their respective metadata data structures to determine whether the failed memory device is used to store information referenced by their respective metadata data structures, wherein searching each consolidated device list allows the nodes to make the determination without searching through the data block location information structure, which includes more information than the consolidated device list; and receive an indication from the nodes of which metadata data structures reference information stored on the failed device. - View Dependent Claims (8, 9, 10, 11, 12)
-
Specification