Partitioned performance tracking core resource consumption independently

• US 10,142,195 B1
• Filed: 09/30/2015
• Issued: 11/27/2018
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Inventors: Sledz, Dan, Walton, Jonathan, Powell, Daniel, Rang, Anton
• Primary Examiner(s): Meky, Moustafa M
• Assistant Examiner(s): Islam, Sm Z
• Application Number:
  US14/870,151
• Time in Patent Office: 1,154 Days
• CPC Class Codes: G06F 16/182 Distributed file systems
• CPC Class Codes: H04L 43/0817 by checking functioning

Abstract

Implementations are provided herein for offering partitioned performance within a distributed file system. Core resource consumption per work item can be tracked independently. Discriminative data already known by the file system surrounding the context of the work item can be used to determine a reference resource accounting specification applicable to the work item. When processing the work item, a detailed resource record can be generated that inventories the resources used in processing the work item. The resource record associated with the work item can be recorded into a set of resource accounting tokens that track activity resource consumption at a granular level. A universal table of resource accounting tokens can be dynamically updated upon the processing of work items and generation of associated resource records throughout the distributed file system.

Claims

1. A method to independently track resource consumption in a distributed file system operating as a cluster of a set of nodes comprising:

identifying a work item wherein the work item is associated with an access zone identifier of the distributed file system, a user identifier of the distributed file system, a group identifier of the distributed file system, a protocol of the distributed file system, a client internet protocol address, and a node identifier of the distributed file system;

determining a set of reference resource accounting token (“

RAT”

) specifications based on the access zone identifier, the user identifier, the group identifier, the protocol, the client internet protocol address, and the node identifier;

associating the work item with a set of resource accounting tokens (“

RATs”

) based on the set of reference RAT specifications;

generating a resource record for the work item wherein the resource record contains an accounting of a set of resources required to process the work item wherein the set of resources includes at least one of a resource consumption time duration, network bandwidth, disk activity, central processing unit (“

CPU”

) usage, and memory usage;

processing the work item; and

in response to the work item being processed, recording the resource record into the set of RATs.

2. The method of claim 1, further comprising:

dynamically storing and updating RATs to a central location within the cluster, wherein nodes in the set of nodes have access to read and write to the central location.

3. The method of claim 1, wherein the protocol is at least one of server message block, network file system, Hadoop distributed file system, file transfer protocol, object, hyper text transfer protocol, and a job engine of the distributed file system.

4. The method of claim 1, wherein the disk activity includes at least one of an indication of whether a disk cache was used to process the work item, a number of bytes read from cache, or a number of bytes written into cache.

5. The method of claim 1, wherein network bandwidth includes a number of bytes sent, a number of bytes received, mbufs, mbuf clusters, and mbuf bytes.

6. The method of claim 1, further comprising:

identifying a dependent work item associated with the work item;

generating a dependent resource record for the dependent work item;

in response to the dependent work item being processed, recording the dependent resource record into the set of RATs.

7. A system comprising at least one storage device and at least one hardware processor configured to:

identify a work item wherein the work item is associated with an access zone identifier of the distributed file system, a user identifier of the distributed file system, a group identifier of the distributed file system, a protocol of the distributed file system, a client internet protocol address, and a node identifier of the distributed file system;

determine a reference resource accounting token (“

RAT”

) specification based on the access zone identifier, the user identifier, the group identifier, the protocol, the client internet protocol address, and the node identifier;

associate the work item with a set of resource accounting tokens (“

RATs”

) based on the reference RAT specification;

generate a resource record for the work item wherein the resource record contains an accounting of a set of resources required to process the work item wherein the set of resources includes at least one of a resource consumption time duration, network bandwidth, disk activity, central processing unit (“

CPU”

) usage, and memory usage;

process the work item; and

in response to the work item being processed, record the resource record into the set of RATs.

8. The system of claim 7, further configured to:

dynamically store and update RATs to a central location within the cluster, wherein nodes in the set of nodes have access to read and write to the central location.

9. The system of claim 7, wherein the protocol is at least one of server message block, network file system, Hadoop distributed file system, file transfer protocol, object, hypertext transfer protocol, and a job engine of the distributed file system.

10. The system of claim 7, wherein the disk activity includes at least one of an indication of whether a disk cache was used to process the work item, a number of bytes read from cache, or a number of bytes written into cache.

11. The system of claim 7, wherein network bandwidth includes a number of bytes sent, a number of bytes received, mbufs, mbuf clusters, and mbuf bytes.

12. The system of claim 7, further configured to:

identify a dependent work item associated with the work item;

generate a dependent resource record for the dependent work item;

in response to the dependent work item being processed, record the dependent resource record into the set of RATs.

13. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:

identifying a work item wherein the work item is associated with an access zone identifier of the distributed file system, a user identifier of the distributed file system, a group identifier of the distributed file system, a protocol of the distributed file system, a client internet protocol address, and a node identifier of the distributed file system;

determining a reference resource accounting token (“

RAT”

) specification based on the access zone identifier, the user identifier, the group identifier, the protocol, the client internet protocol address, and the node identifier;

associating the work item with a set of resource accounting tokens (“

RATs”

) based on the reference RAT specification;

generating a resource record for the work item wherein the resource record contains an accounting of a set of resources required to process the work item wherein the set of resources includes at least one of a resource consumption time duration, network bandwidth, disk activity, central processing unit (“

CPU”

) usage, and memory usage;

processing the work item; and

in response to the work item being processed, recording the resource record into the set of RATs.

14. The non-transitory computer readable medium of claim 13, with program instructions stored thereon to further perform the following acts:

dynamically storing and updating RATs to a central location within the cluster, wherein nodes in the set of nodes have access to read and write to the central location.

15. The non-transitory computer readable medium of claim 13, wherein the protocol is at least one of server message block, network file system, Hadoop distributed file system, file transfer protocol, object, hyper text transfer protocol, and a job engine of the distributed file system.

16. The non-transitory computer readable medium of claim 13, wherein the disk activity includes at least one of an indication of whether a disk cache was used to process the work item, a number of bytes read from cache, or a number of bytes written into cache.

17. The non-transitory computer readable medium of claim 13, wherein network bandwidth includes a number of bytes sent, a number of bytes received, mbufs, mbuf clusters, and mbuf bytes.

18. The non-transitory computer readable medium of claim 13, with program instructions stored thereon to further perform the following acts:

identifying a dependent work item associated with the work item;

generating a dependent resource record for the dependent work item;

in response to the dependent work item being processed, recording the dependent resource record into the set of RATs.