File system queue

• US 9,037,559 B2
• Filed: 08/13/2012
• Issued: 05/19/2015
• Est. Priority Date: 08/13/2012
• Status: Active Grant

• Current Assignee: Transform Sr Brands, LLC (ESL Investments, Inc.)
• Original Assignee: Sears Brands LLC (Sears Holdings Corporation)
• Inventors: Harward, Nathaniel Distin, Semibratov, Dmitry
• Primary Examiner(s): Richardson, James E
• Application Number:
  US13/584,540
• Publication Number:
  US 20140046987A1
• Time in Patent Office: 1,009 Days
• Field of Search: 707/764, 707/799
• US Class Current: 707/704
• CPC Class Codes: G06F 2209/548 Queue
• CPC Class Codes: G06F 9/526 Mutual exclusion algorithms
• CPC Class Codes: G06F 9/546 Message passing systems or structures, e.g. queues

Abstract

A method and system that enables operation of a file system-based queue that supports the efficient and fair distribution of messages among a plurality of processors.

Claims

1. A method ensuring processing of each file of a first collection of files in a shared file system by only one computer, of a plurality of computers, processing files from the first collection, the method comprising:

in each computer of the plurality of computers,

(i) determining whether at least one file is available in the first collection of files by retrieving a list of files of the first collection of files from one location in the shared file system that is accessible to all computers of the plurality of computers;

(ii) selecting a first file from the list of files, if it is determined that at least one file is available in the first collection of files;

(iii) continuing to determine whether at least one file is available, if it is determined that no files are available in the first collection of files;

(iv) responsive to selecting the first file, attempting to move the first file from the first collection of files for processing, wherein the move is an indivisible operation that succeeds for only one computer of two or more computers of the plurality of computers that concurrently attempt to move any given file;

(v) responsive to attempting to move the first file, determining whether the attempt to move the first file was successful;

(vi) processing content of the first file, only if the move operation was successful; and

(vii) returning to step (i), if the move operation was not successful.

2. The method of claim 1, wherein the first file comprises metadata identifying a data file separate from the first file, and processing content comprises processing the data file identified by the metadata.

3. The method of claim 1, wherein the method further comprises:

in a second move operation upon the shared file system, attempting to move the first file to a collection of files separate from the first collection of files, based upon success of the processing of content of the first file.

4. The method of claim 3, wherein the second move operation locates the first file in a second collection of files in the shared file system separate from the first collection of files, if the processing is successful, and locates the first file in a third collection of files in the shared file system separate from the first collection of files and the second collection of files, if the processing is not successful.

5. The method of claim 1, wherein the shared file system is external to each of the plurality of computers.

6. The method of claim 1, wherein the shared file system is compliant with one or both of the Internet Engineering Task Force (IETF) specifications identified as RFC 1094 and RFC 1813.

7. The method of claim 1, wherein the first move operation and the second move operation are functionality provided by an operating system of each of the plurality of computers.

8. A system that ensures processing of each file of a first collection of files in a shared file system by only one computer, of a plurality of computers, processing files from the first collection, the system comprising:

in each computer of the plurality of computers, at least one processor enabled to communicatively couple to the shared file system, the at least one processor operable to, at least;

(i) determine whether at least one file is available in the first collection of files by retrieve a list of files of the first collection of files from one location in the shared file system that is accessible to all computers of the plurality of computers;

(ii) select a first file from the list of files, if it is determined that at least one file is available in the first collection of files;

(iii) continue to determine whether at least one file is available, if it is determined that no files are available in the first collection of files;

(iv) responsive to selecting the first file, attempt to move the first file from the first collection of files for processing, wherein the move is an indivisible operation that succeeds for only one computer of two or more computers of the plurality of computers that concurrently attempt to move any given file;

(v) responsive to the attempt to move the first file, determine whether the attempt to move the first file was successful;

(vi) process content of the first file, only if the move operation was successful; and

(vii) return to step (i), if the move operation was not successful.

9. The system of claim 8, wherein the first file comprises metadata identifying a data file separate from the first file, and processing content comprises processing the data file identified by the metadata.

10. The system of claim 8, wherein the at least one processor is operable to, at least:

in a second move operation upon the shared file system, attempt to move the first file to a collection of files separate from the first collection of files, based upon success of the processing of content of the first file.

11. The system of claim 10, wherein the second move operation locates the first file in a second collection of files in the shared file system separate from the first collection of files, if the processing is successful, and locates the first file in a third collection of files in the shared file system separate from the first collection of files and the second collection of files, if the processing is not successful.

12. The system of claim 8, wherein the shared file system is external to each of the plurality of computers.

13. The system of claim 8, wherein the shared file system is compliant with one or both of the Internet Engineering Task Force (IETF) specifications identified as RFC 1094 and RFC 1813.

14. The system of claim 8, wherein the first move operation and the second move operation are functionality provided by an operating system of each of the plurality of computers.

15. A non-transitory computer-readable medium having a plurality of code sections, each code section comprising instructions executable by at least one processor, the instructions causing the at least one processor to perform a method of processing each file of a first collection of files in a shared file system by only one computer, of a plurality of computers, processing files from the first collection, the method comprising:

(i) determining whether at least one file is available in the first collection of files by retrieving a list of files of the first collection of files from one location in the shared file system that is accessible to all computers of the plurality of computers;

(ii) selecting a first file from the list of files, if it is determined that at least one file is available in the first collection of files;

(iii) continuing to determine whether at least one file is available, if it is determined that no files are available in the first collection of files;

(iv) responsive to selecting the first file, attempting to move the first file from the first collection of files for processing, wherein the move is an indivisible operation that succeeds for only one computer of two or more computers of the plurality of computers that concurrently attempt to move any given file;

(v) responsive to attempting to move the first file, determining whether the attempt to move the first file was successful;

(vi) processing content of the first file, only if the move operation was successful; and

(vii) returning to step (i), if the move operation was not successful.

16. The computer-readable medium of claim 15, wherein the first file comprises metadata identifying a data file separate from the first file, and processing content comprises processing the data file identified by the metadata.

17. The computer-readable medium of claim 15, wherein the method further comprises:

in a second move operation upon the shared file system, attempting to move the first file to a collection of files separate from the first collection of files, based upon success of the processing of content of the first file.

18. The computer-readable medium of claim 17, wherein the second move operation locates the first file in a second collection of files in the shared file system separate from the first collection of files, if the processing is successful, and locates the first file in a third collection of files in the shared file system separate from the first collection of files and the second collection of files, if the processing is not successful.

19. The computer-readable medium of claim 15, wherein the shared file system is external to each of the plurality of computers.

20. The computer-readable medium of claim 15, wherein the shared file system is compliant with one or both of the Internet Engineering Task Force (IETF) specifications identified as RFC 1094 and RFC 1813.

21. The computer-readable medium of claim 15, wherein the first move operation and the second move operation are functionality provided by an operating system of each of the plurality of computers.