Using a file path to determine file locality for applications

• US 10,169,348 B2
• Filed: 08/23/2012
• Issued: 01/01/2019
• Est. Priority Date: 08/23/2012
• Status: Active Grant

• Current Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Original Assignee: Red Hat, Inc. (International Business Machines Corporation)
• Inventors: Shankar, Venky, Bellur, Vijay
• Primary Examiner(s): Leroux, Etienne P
• Application Number:
  US13/592,673
• Publication Number:
  US 20140059094A1
• Time in Patent Office: 2,322 Days
• Field of Search: 707737, 707 10, 707100, 718 1, 719238
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06

Abstract

A processing device identifies a path name of a file that is stored in a file system and sends an extended attribute request comprising the path name to a server computer system to identify a physical location of the file in the file system. The file system includes a virtual extended attributes that identify the physical location of the file that corresponds to the path name in the request. The processing device receives a response from the server computer system indicating the physical location of the file in the file system.

Claims

1. A method comprising:

identifying a path name and a volume identifier of a file that is stored in a file system associated with a plurality of storage servers, the file being associated with a system call from a map/reduce-based application;

mounting the file system via a mount-point by using the volume identifier;

determining an extended attribute request by converting the system call from the map/reduce-based application to an executable routine to be used by the file system;

sending, by a processing device, the extended attribute request to the mount-point, the extended attribute request comprising the path name to a server computer system to identify a physical location of the file at a storage server of the plurality of storage servers in the file system, wherein the file system comprises a virtual extended attribute that identifies the physical location of the file in view of a hash value associated with the path name in the request;

receiving a response from the server computer system indicating the physical location of the file at the storage server in the file system in view of the hash value associated with the path name in the request, the received response comprising a directory name that is associated with the physical location of the file in the file system;

creating a job request for the file in view of a closeness of the physical location of the file in the file system from the received response to a physical location of another storage server to operate on the file at the storage server in the file system in view of the job request; and

sending the job request to a master storage server to combine results of an operation on the file with additional results associated with an additional job request.

2. The method of claim 1, wherein identifying the path name comprises:

receiving the path name from the map/reduce-based application.

3. The method of claim 1, wherein the response comprises a host identifier or a directory identifier.

4. The method of claim 1, wherein file comprises an extended attribute indicating the physical location of the file in the file system.

5. The method of claim 1, wherein the mounting of the file system via the mount point is further in view of the path name of the file that is stored in the file system.

7. The method of claim 1, wherein the response comprises a host identifier or a directory identifier.

8. The method of claim 1, wherein the file system comprises a plurality of virtual extended attributes indicating a physical location of a plurality of files in the file system.

6. A method comprising:

identifying a system call associated with a file from a map/reduce-based application;

receiving an extended attribute request from a client computer device, wherein the extended attribute request indicates a path name and a volume identifier of a file in a file system associated with a plurality of storage servers, the extended attribute request being in view of a conversion of the system call from the map/reduce-based application to an executable routine to be used by the file system;

mounting the file system via a mount-point by using the volume identifier;

determining a key using the path name;

generating, by a processing device, a value of a virtual extended attribute using the key and the mount-point, wherein the value of the virtual extended attribute indicates a physical location of the file at a storage server of the plurality of storage servers in the file system that is identified in view of a hash value associated with the path name;

sending a response indicating the physical location of the file at the storage server to the client computer device in view of the hash value associated with the path name, the response comprising a directory name that is associated with the physical location of the file in the file system;

creating a job request for the file in view of a closeness of the physical location of the file in the file system from the received response to a physical location of another storage server to operate on the file at the storage server in the file system in view of the job request; and

sending the job request to a master storage server to combine results of an operation on the file with additional results associated with an additional job request.

9. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to:

identify a path name and a volume identifier of a file that is stored in a file system associated with a plurality of storage servers, the file being associated with a system call from a map/reduce-based application;

mounting the file system via a mount-point by using the volume identifier;

determine an extended attribute request by converting the system call from the map/reduce-based application to an executable routine to be used by the file system;

send, by the processing device, the extended attribute request to the mount-point, the extended attribute request comprising the path name to a server computer system to identify a physical location of the file at a storage server of the plurality of storage servers in the file system, wherein the file system comprises a virtual extended attribute that identifies the physical location of the file in view of a hash value associated with the path name in the request;

receive a response from the server computer system indicating the physical location of the file at the storage server in the file system in view of the hash value associated with the path name in the request, the received response comprising a directory name that is associated with the physical location of the file in the file system;

create a job request for the file in view of a closeness of the physical location of the file in the file system from the received response to a physical location of another storage server to operate on the file at the storage server in the file system in view of the job request; and

send the job request to a master storage server to combine results of an operation on the file with additional results associated with an additional job request.

10. The non-transitory computer-readable storage medium of claim 9, wherein to identify the path name, the processing device is further to:

receive the path name from the map/reduce-based application.

11. The non-transitory computer-readable storage medium of claim 9, wherein the response comprises a host identifier or a directory identifier.

12. A system comprising:

a memory; and

a first processing device, operatively coupled to the memory, to;

identify a path name and a volume identifier of a file that is stored in a file system associated with a plurality of storage servers, the file being associated with a system call from a map/reduce-based application;

mount the file system via a mount-point by using the volume identifier;

determine an extended attribute request by converting the system call from the map/reduce-based application to an executable routine to be used by the file system;

send the extended attribute request to the mount-point, the extended attribute request comprising the path name to a second processing device to identify a physical location of the file at a storage server of the plurality of storage servers in the file system, wherein the file system comprises a virtual extended attribute that identifies the physical location of the file in view of a hash value associated with the path name in the request;

receive a response from the second processing device indicating the physical location of the file at the storage server in the file system in view of the hash value associated with the path name in the request, the received response comprising a directory name that is associated with the physical location of the file in the file system;

create a job request for the file in view of a closeness of the physical location of the file in the file system from the received response to a physical location of another storage server to operate on the file at the storage server in the file system in view of the job request; and

send the job request to a master storage server to combine results of an operation on the file with additional results associated with an additional job request.

13. The system of claim 12, wherein to identify the path name, the processing device is further to:

receive the path name from the map/reduce-based application.

14. The system of claim 12, wherein the response comprises a host identifier or a directory identifier.

15. The system of claim 12, wherein the second processing device is to:

receive the extended attribute request from the first processing device, wherein the extended attribute request indicates the path name of the file;

read a value of an extended attribute of the file that corresponds to the path name, wherein the value of the extended attribute indicates the physical location of the file; and

send the response indicating the physical location of the file to the first processing device.