Mirrored file system

• US 8,286,127 B2
• Filed: 07/22/2010
• Issued: 10/09/2012
• Est. Priority Date: 04/14/2006
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bumgarner, William M., Swift, Adam C., Gross, Andrew
• Primary Examiner(s): Das, Chameli
• Application Number:
  US12/841,867
• Publication Number:
  US 20100287526A1
• Time in Patent Office: 810 Days
• Field of Search: None
• US Class Current: 717/103
• CPC Class Codes: G06F 11/1662 the resynchronized component or unit being a persistent storage device re-synchronization of failed mirror storage G06F11/2082; rebuild or reconstruction of parity RAID storage G06F11/1008
• CPC Class Codes: G06F 11/2097 maintaining the standby controller/processing unit updated initialisation or re-synchronisation thereof G06F11/1658 and subgroups
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes

Abstract

Mounting and populating a mirrored version of at least a portion of a file system of a recruiter machine at a volunteer machine performing one or more data processing tasks for the recruiter machine is disclosed. In some embodiments, an input stored in such a dedicated file system for the recruiter machine at the volunteer machine may be locally retrieved and reused, if still consistent with a corresponding input stored in a file system at the recruiter machine, when performing a data processing task for the recruiter machine at the volunteer machine. In some embodiments, if an input required by the volunteer machine to perform a data processing task for the recruiter machine is not cached in such a dedicated file system for the recruiter machine at the volunteer machine in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, the input is obtained from the recruiter machine.

Claims

1. A method for accomplishing a data processing task to obtain output data, comprising:

receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, wherein cached input is cached in a dedicated file system associated with the recruiter machine at the volunteer machine, and wherein the dedicated file system comprises at least partially mirrored view of the recruiter machine'"'"'s file system, and wherein the dedicated file system is mounted under a mount directory of a master file system at the volunteer machine, and wherein the recruiter machine distributed the data processing task to the volunteer machine, and wherein the data processing task is associated with uncompiled code; and

providing the input to the volunteer machine in a state that is consistent with the corresponding input at the recruiter machine.

2. The method as recited in claim 1, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine comprises receiving a request for the input from the volunteer machine.

3. The method as recited in claim 1, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that one or more parameters associated with a version of the input as cached in the dedicated file system associated with the recruiter machine at the volunteer machine are not consistent with one or more corresponding parameters of the corresponding input as stored in a file system at the recruiter machine.

4. The method as recited in claim 1, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that a hash of a version of the input as cached in the dedicated file system associated with the recruiter machine at the volunteer machine is not consistent with a hash of the corresponding input as stored in a file system at the recruiter machine.

5. The method as recited in claim 1, further comprising sending from the recruiter machine to the volunteer machine a request to perform the data processing task.

6. The method as recited in claim 1, wherein the data processing task is part of a set of tasks that comprise a data processing job that is performed at the recruiter machine.

7. The method of claim 1, wherein a state other than the consistent state comprises the input not existing in the dedicated file system associated with the recruiter machine at the volunteer machine.

8. The method of claim 1, wherein a state other than the consistent state comprises the version of the input cached in the dedicated system associated with the recruiter machine at the volunteer machine being marked invalid.

9. The method of claim 1, wherein one or more other dedicated file systems associated with other recruiter machines are mounted under the mount directory of the master file system at the volunteer machine.

11. The method as recited in claim 7, further comprising receiving from the recruiter machine a request to perform the data processing task.

12. The method as recited in claim 11, wherein receiving the request includes configuring at the volunteer machine in response to the request a process to intercept and direct file system calls associated with the data processing task to the dedicated file system.

10. A method for performing a data processing task, comprising:

determining at a volunteer machine that an input that is used by the volunteer machine to perform a data processing task on behalf of a recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, wherein cached input is cached in a dedicated file system associated with the recruiter machine at the volunteer machine, and wherein the dedicated file system comprises at least partially mirrored view of the recruiter machine'"'"'s file system, and wherein the dedicated file system is mounted under a mount directory of a master file system at the volunteer machine, and wherein the recruiter machine distributed the data processing task to the volunteer machine, and the data processing task is associated with uncompiled code; and

obtaining the input from the recruiter machine in a state that is consistent with the corresponding input at the recruiter machine.

13. The method as recited in claim 10, wherein determining at a volunteer machine that an input that is used by the volunteer machine to perform a data processing task on behalf of a recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that the input does not exist in the dedicated file system associated with the recruiter machine at the volunteer machine.

14. The method as recited in claim 10, wherein determining at a volunteer machine that an input that is used by the volunteer machine to perform a data processing task on behalf of a recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that a version of the input cached in the dedicated file system associated with the recruiter machine at the volunteer machine is marked invalid.

15. The method as recited in claim 10, further comprising receiving from the recruiter machine an invalid list that includes files that have been previously provided to the volunteer machine but have since been updated at the recruiter machine and are no longer valid at the volunteer machine.

16. A system for performing a data processing task, comprising:

a processor configured to;

determine that an input that is used to perform a data processing task on behalf of a recruiter machine is not locally cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, wherein cached input is cached in a dedicated file system associated with the recruiter machine at the volunteer machine, and wherein the dedicated file system comprises at least partially mirrored view of the recruiter machine'"'"'s file system, and wherein the dedicated file system is mounted under a mount directory of a master file system at the volunteer machine, and wherein the recruiter machine distributed the data processing task to the system, and wherein the data processing task is associated with uncompiled code; and

obtain the input from the recruiter machine in a state that is consistent with the corresponding input at the recruiter machine; and

a memory coupled to the processor and configured to provide instructions to the processor.

17. The system as recited in claim 16, wherein the processor is further configured to receive from the recruiter machine a request to perform the data processing task.

18. The system as recited in claim 16, wherein to determine that an input that is used to perform a data processing task on behalf of a recruiter machine is not locally cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that one or more parameters associated with a version of the input as locally cached in the dedicated file system associated with the recruiter machine are not consistent with one or more corresponding parameters of the corresponding input as stored in a file system at the recruiter machine.

19. The system as recited in claim 16, wherein to determine that an input that is used to perform a data processing task on behalf of a recruiter machine is not locally cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that a hash of a version of the input as locally cached in the dedicated file system associated with the recruiter machine is not consistent with a hash of the corresponding input as stored in a file system at the recruiter machine.

20. The system as recited in claim 16, wherein to determine that an input that is used to perform a data processing task on behalf of a recruiter machine is not locally cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that the input does not locally exist in the dedicated file system associated with the recruiter machine.

21. The system as recited in claim 16, wherein to determine that an input that is used to perform a data processing task on behalf of a recruiter machine is not locally cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that a version of the input locally cached in the dedicated file system associated with the recruiter machine is marked invalid.

22. A computer program product for performing a data processing task, the computer program product being embodied in a non-transitory computer readable storage medium and storing computer instructions for:

determining at a volunteer machine that an input that is required used by the volunteer machine to perform a data processing task on behalf of a recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine, wherein cached input is cached in a dedicated file system associated with the recruiter machine at the volunteer machine, and wherein the dedicated file system comprises at least partially mirrored view of the recruiter machine'"'"'s file system, and wherein the dedicated file system is mounted under a mount directory of a master file system at the volunteer machine, and wherein the recruiter machine distributed the data processing task to the volunteer machine, and the data processing task is associated with uncompiled code; and

obtaining the input from the recruiter machine in a state that is consistent with the corresponding input at the recruiter machine.

23. The computer program product as recited in claim 22, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine comprises receiving a request for the input from the volunteer machine.

24. The computer program product as recited in claim 22, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that one or more parameters associated with a version of the input as cached in the dedicated file system associated with the recruiter machine at the volunteer machine are not consistent with one or more corresponding parameters of the corresponding input as stored in a file system at the recruiter machine.

25. The computer program product as recited in claim 22, wherein receiving at a recruiter machine an indication that an input that is used by a volunteer machine to perform a data processing task on behalf of the recruiter machine is not cached in a state that is consistent with a corresponding input stored in a file system at the recruiter machine includes determining that a hash of a version of the input as cached in the dedicated file system associated with the recruiter machine at the volunteer machine is not consistent with a hash of the corresponding input as stored in a file system at the recruiter machine.