SYSTEM AND METHOD FOR EFFICIENT STORAGE OF SMALL FILES ON FILE-SYSTEM-BASED STORAGE DEVICES

• US 20160026645A1
• Filed: 07/23/2015
• Published: 01/28/2016
• Est. Priority Date: 07/24/2014
• Status: Active Grant

• Current Assignee: Cognyte Technologies Israel Ltd. (Cognyte Software Ltd.)
• Original Assignee: Verint Systems Limited (Verint Systems Incorporated)
• Inventors: Chai, Yossi
• Granted Patent: US 10,423,574 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/13 File access structures, e.g. distributed indices arrangements of input from, or output to, record carriers G06F3/06

Abstract

Methods and systems for storing and managing large numbers of small files. A data processing system includes clients that generate large numbers be stored on a storage device managed by a File System (FS). An Archive Server (AS) receives multiple files from the client, archives the files in larger archives, and sends the archives to the FS for storage. When requested to read a file, the AS retrieves the archive in which the file is stored, extracts the file and sends it to the requesting client. In other words, the AS communicates with the clients in individual file units, and with the storage device in archive units. The AS is typically constructed as an add-on layer on top of a conventional FS, which enables the FS to handle small files efficiently without modification.

Claims

1. A method for data storage, comprising:

receiving a plurality of files to be stored on a storage device that is managed by a File System (FS);

determining an association between at least first and second duplicate occurrences of a given file in the plurality;

assigning the files, including the given file, to one or more archives based on the association; and

sending the archives to the FS for storage on the storage device.

2. The method according to claim 1, wherein determining the association and assigning the files to the archives are performed transparently to the FS.

3. The method according to claim 1, wherein assigning the files to the archives comprises maintaining a catalog that specifies, for each file including the given file, a respective one of the archives in which the file is stored.

4. The method according to claim 3, wherein assigning the files to the archives comprises storing a single copy of the given file in a selected archive, and recording both the first occurrence and the second occurrence of the given file in the catalog as stored in the selected archive.

5. The method according to claim 3, wherein maintaining the catalog comprises maintaining respective separate first and second validity indications for the first and second occurrences of the given file.

6. The method according to claim 5, and comprising permitting deletion of an archive, in which the given file is stored, only when the first and second validity indications indicate that the first and second occurrences of the given file are both invalid.

7. The method according to claim 1, and comprising, upon receiving a request to read the given file, retrieving from the storage device an archive in which the given file is stored, extracting the given file from the retrieved archive, and serving the extracted given file in response to the request.

8. The method according to claim 1, wherein assigning the files to the archives comprises deciding to close a current archive and assign subsequently received files to a new archive based on a predefined criterion.

9. The method according to claim 8, wherein the criterion is based on at least one factor selected from a group of factors consisting of:

time that elapsed since the current archive was created;

a data size of the current archive;

a number of the files stored in the current archive;

names of the files stored in the current archive; and

types of the files stored in the current archive.

10. An archiving system, comprising:

an interface, which is configured to communicate with one or more client devices; and

one or more processors, which are configured to receive over the interface from the client devices a plurality of files to be stored on a storage device that is managed by a File System (FS), to determine an association between at least first and second duplicate occurrences of a given file in the plurality, to assign the files, including the given file, to one or more archives based on the association, and to send the archives to the FS for storage on the storage device.

11. The system according to claim 10, wherein the processors are configured to determine the association and assign the files to the archives transparently to the FS.

12. The system according to claim 10, wherein the processors are configured to maintain a catalog that specifies, for each file including the given file, a respective one of the archives in which the file is stored.

13. The system according to claim 12, wherein the processors are configured to store a single copy of the given file in a selected archive, and to record both the first occurrence and the second occurrence of the given file in the catalog as stored in the selected archive.

14. The system according to claim 12, wherein the processors are configured to maintain in the catalog respective separate first and second validity indications for the first and second occurrences of the given file.

15. The system according to claim 14, wherein the processors are configured to permit deletion of an archive, in which the given file is stored, only when the first and second validity indications indicate that the first and second occurrences of the given file are both invalid.

16. The system according to claim 10, wherein, upon receiving a request to read the given file, the processors are configured to retrieve from the storage device an archive in which the given file is stored, to extract the given file from the retrieved archive, and to serve the extracted given file in response to the request.

17. The system according to claim 10, wherein the processors are configured to decide to close a current archive and assign subsequently received files to a new archive based on a predefined criterion.

18. The system according to claim 17, wherein the criterion is based on at least one factor selected from a group of factors consisting of:

time that elapsed since the current archive was created;

a data size of the current archive;

a number of the files stored in the current archive;

names of the files stored in the current archive; and

types of the files stored in the current archive.

19. A data processing system, comprising:

a storage device that is managed by a File System (FS);

one or more client devices, which are configured to generate a plurality of files for storage on the storage device; and

one or more archiving processors, which are configured to determine an association between at least first and second duplicate occurrences of a given file in the plurality, to assign the files, including the given file, to one or more archives based on the association, and to send the archives to the FS for storage on the storage device.