Adaptive Parallel Data Processing

• US 20150067003A1
• Filed: 09/03/2013
• Published: 03/05/2015
• Est. Priority Date: 09/03/2013
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: MacLeod, Peter S.
• Granted Patent: US 10,162,829 B2
• Time in Patent Office: Days
• US Class Current: 707/827
• CPC Class Codes: G06F 16/16 File or folder operations, e.g. details of user interfaces specifically adapted to file systems
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 3/124 Parallel printing or parallel ripping

Abstract

Adaptive parallel data processing techniques are described. In one or more embodiments, a request is received to process a data file. The data file is split into multiple portions and sent to multiple nodes, where each node is configured to process a respective portion of the data file. Responsive to an amount of processing of the data file being completed, at least one of the multiple portions of the data file is dynamically split into multiple sub-portions. The sub-portions are submitted to one or more of the multiple nodes for processing of the sub-portions.

Claims

1. A method comprising:

receiving a request to process a data file;

splitting the data file into multiple portions;

sending the multiple portions to multiple nodes, each node configured to process a respective portion of the data file;

responsive to an amount of processing of the data file being completed, dynamically splitting at least one of the multiple portions of the data file that have yet to complete processing into multiple sub-portions; and

submitting the multiple sub-portions to one or more of the multiple nodes for processing of the multiple sub-portions.

2. A method as described in claim 1, further comprising receiving the data file from a client device over a network.

3. A method as described in claim 1, further comprising:

monitoring progress of the processing of the data file; and

comparing the progress of the processing to a predefined threshold.

4. A method as described in claim 1, further comprising, responsive to receiving a response corresponding to the at least one of the multiple portions of the data file, using the response corresponding to the at least one of the multiple portions as part of a result of the processing of the data file.

5. A method as described in claim 1, further comprising, responsive to receiving from the one or more of the multiple nodes results corresponding to the multiple sub-portions, using the results corresponding to the multiple sub-portions as part of a result of the processing of the data file.

6. A method as described in claim 5, wherein the results corresponding to the multiple sub-portions are received prior to receiving a response that corresponds to the at least one of the multiple portions of the data file.

7. A method as described in claim 1, wherein the processing of the data file includes conversion from a first format to a second format.

8. A method as described in claim 1, wherein the processing of the data file includes conversion from a non-editable format to an editable format.

9. A method as described in claim 1, wherein the processing of the data file includes conversion from a non-searchable format to a searchable format.

10. A method as described in claim 1, wherein the multiple nodes include parallel computing engines in a cloud computing infrastructure.

11. A method as described in claim 1, wherein the data file comprises a digital document having a plurality of pages, wherein each of the multiple portions comprises one or more of the plurality of pages.

12. A system comprising:

a memory and a processor configured to execute instructions in the memory to implement a service manager module, the service manager module configured to;

monitor progress of conversion of a document, the document split into multiple portions and submitted to a pool of nodes for conversion of respective portions;

responsive to completion of conversion of a predefined amount of the document;

split at least one of the multiple portions into a plurality of sub-portions, the at least one of the multiple portions being non-completed; and

submit the plurality of sub-portions to the pool of nodes for conversion of each sub-portion.

13. A system as described in claim 12, wherein the pool of nodes comprises a plurality of parallel computing systems in a cloud computing infrastructure.

14. A system as described in claim 12, wherein the conversion of the document includes conversion from a non-editable format to an editable format.

15. A system as described in claim 12, wherein the service manager module is further configured to use, for the conversion of the document:

a first result corresponding to the at least one of the multiple portions that are non-completed;

or

a second result corresponding to the plurality of sub-portions.

16. A system as described in claim 15, wherein the service manager module is further configured to, in an event that the second result is received prior to the first result, use the second result to complete the conversion of the document.

17. One or more computer-readable storage media having instructions that, responsive to execution by a computer, cause the computer to perform operations comprising:

monitoring progress of processing of a file that is split into a plurality of portions to be processed by a plurality of nodes;

receiving results corresponding to one or more of the plurality of portions;

based on the monitoring, identifying that one or more non-completed portions of the plurality of portions of the file remain;

splitting the one or more non-completed portions into a plurality of sub-portions; and

submitting the plurality of sub-portions to the plurality of nodes for processing.

18. One or more computer-readable storage media as described in claim 17, wherein the processing of the file comprises conversion of the file from a non-searchable format to a searchable format.

19. One or more computer-readable storage media as described in claim 17, wherein the plurality of nodes include parallel computing instances operating in a cloud infrastructure.

20. One or more computer-readable storage media as described in claim 17, wherein the processing of the file comprises conversion of the file from a non-editable format to an editable format.