Burst buffer appliance with small file aggregation

• US 8,972,465 B1
• Filed: 03/15/2013
• Issued: 03/03/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Emc IP Holding Company LLC (Dell Technologies Inc.)
• Original Assignee: EMC Corporation (Dell Technologies Inc.)
• Inventors: Faibish, Sorin, Bent, John M.
• Primary Examiner(s): NGUYEN, CAM LINH T
• Application Number:
  US13/840,274
• Time in Patent Office: 718 Days
• Field of Search: 707/609, 707/822, 707/823
• US Class Current: 707/822
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06

Abstract

A burst buffer appliance is adapted for coupling between a computer system and a file system. The burst buffer appliance comprises a flash memory or other high-speed memory having a substantially lower access time than the file system, and an analytics engine, and is configured to aggregate multiple data files having a size below a designated threshold into a container having at least one associated metadata file for processing by the analytics engine. The burst buffer appliance directs storage of the container in the file system. The analytics engine is configured to generate at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the metadata file(s) rather than the corresponding multiple data files.

Claims

1. An apparatus comprising:

a burst buffer appliance adapted for coupling between a computer system and a file system;

the burst buffer appliance comprising;

a high-speed memory having a substantially lower access time than the file system; and

an analytics engine;

the burst buffer appliance being configured to aggregate multiple data files having a size below a designated threshold into a container having at least one associated metadata file for processing by the analytics engine;

wherein the burst buffer appliance directs storage of the container in the file system;

wherein the analytics engine is configured to generate at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the one or more metadata files rather than the corresponding multiple data files.

2. The apparatus of claim 1 wherein the analytics engine is configured to perform a virus checking operation on the multiple data files of the container and to update the content-based portion of the one or more metadata files of the container to identify one or more of the multiple data files that are determined to be infected with a virus.

3. The apparatus of claim 1 wherein the analytics engine is configured to perform a map reduce operation on the multiple data files of the container and to update the content-based portion of the one or more metadata files of the container to identify one or more results of the map reduce operation.

4. The apparatus of claim 1 wherein the analytics engine is configured to determine keys associated with the multiple data files of the container and to update the content-based portion of the one or more metadata files of the container to identify one or more of the keys.

5. The apparatus of claim 1 wherein a given one of the one or more metadata files comprises for each of the multiple data files corresponding name list and index files.

6. The apparatus of claim 1 wherein the computer system comprises a plurality of compute nodes and the burst buffer appliance is adapted for coupling between the compute nodes and the file system.

7. The apparatus of claim 1 wherein the file system comprises a parallel log-structured file system.

8. The apparatus of claim 7 wherein the burst buffer appliance implements a virtual layer of the parallel log-structured file system.

9. The apparatus of claim 1 wherein the high-speed memory comprises a flash memory.

10. A processing platform that incorporates the apparatus of claim 1.

11. A method comprising:

configuring a burst buffer appliance adapted for coupling between a computer system and a file system to include a high-speed memory having a substantially lower access time than the file system; and

aggregating in the burst buffer appliance multiple files having a size below a designated threshold into a container having at least one associated metadata file;

processing the container in an analytics engine of the burst buffer appliance; and

storing the container in the file system;

wherein processing the container in the analytics engine comprises generating at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the one or more metadata files rather than the corresponding multiple data files.

12. The apparatus of claim 11 wherein processing the container in the analytics engine comprises:

performing a virus checking operation on the multiple data files of the container; and

updating the content-based portion of the one or more metadata files of the container to identify one or more of the multiple data files that are determined to be infected with a virus.

13. The apparatus of claim 11 wherein processing the container in the analytics engine comprises:

performing a map reduce operation on the multiple data files of the container; and

updating the content-based portion of the one or more metadata files of the container to identify one or more results of the map reduce operation.

14. The apparatus of claim 11 wherein processing the container in the analytics engine comprises:

determining keys associated with the multiple data files of the container; and

updating the content-based portion of the one or more metadata files of the container to identify one or more of the keys.

15. A computer program product comprising a processor-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed in at least one processing device cause the processing device to perform the method of claim 11.

16. An information processing system comprising:

a computer system;

a file system; and

a burst buffer appliance coupled between the computer system and the file system, the burst buffer appliance comprising;

a high-speed memory having a substantially lower access time than the file system; and

an analytics engine;

the burst buffer appliance being configured to aggregate multiple files having a size below a designated threshold into a container having at least one associated metadata file for processing by the analytics engine;

wherein the burst buffer appliance directs storage of the container in the file system;

wherein the analytics engine is configured to generate at least a content-based portion of the one or more metadata files of the container based at least in part on content of one or more of the multiple data files so as to permit one or more subsequent processing operations to be performed using the one or more metadata files rather than the corresponding multiple data files.

17. The system of claim 16 wherein the computer system comprises a plurality of compute nodes and the burst buffer appliance is coupled between the compute nodes and the file system.

18. The system of claim 16 wherein the file system comprises a parallel log-structured file system.