Systems and methods to process a request received at an application program interface

• US 9,043,401 B2
• Filed: 10/08/2009
• Issued: 05/26/2015
• Est. Priority Date: 10/08/2009
• Status: Active Grant

• Current Assignee: PayPal, Inc. (PayPal Holdings, Inc.)
• Original Assignee: eBay Inc.
• Inventors: Wong, Laura, Munamala, Srikala, Pereshyvaylo, Sergiy, Tamhankar, Hemant, Zou, Ping
• Primary Examiner(s): SISON, JUNE Y
• Application Number:
  US12/576,097
• Publication Number:
  US 20110087731A1
• Time in Patent Office: 2,056 Days
• Field of Search: 709/204
• US Class Current: 709/205
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/505 considering the load
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/1078 Resource delivery mechanisms
• CPC Class Codes: H04L 67/1091 Interfacing with client-server systems or between P2P systems

Abstract

Methods and systems to process a request received at an application program interface are described. The system receives a request from a client machine that includes a job that is associated with data. The request is received at an application program interface. Next, a peer-to-peer network of processing nodes generates a plurality of sub-jobs based on the job. The peer-to-peer network of processing nodes schedules the plurality of sub-jobs for parallel processing based on an availability of resources that are respectively utilized by the sub-jobs and parallel processes the plurality of sub-jobs before generating task results that are respectively associated with the plurality of sub-jobs.

Claims

1. A system to process a request, the system comprising:

at least one processor in communication with a non-transitory machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one processor to receive, over a network, the request from a client machine, the request comprising a job that is associated with data, the request received at an application program interface, the job indicating a set of tasks to be performed;

a peer-to-peer network of processing modules configured to generate a plurality of sub-jobs by dividing the job into equal quantities of work, the job being utilized to retrieve transaction data from a network-based marketplace, the plurality of sub-jobs respectively comprising at least one task of the set of tasks to be performed that corresponds to a portion of the data, the peer-to-peer network of processing modules configured to schedule the plurality of sub-jobs for parallel processing based on an availability of resources that are respectively utilized by the sub-jobs, the peer-to-peer network of processing modules configured to parallel process the plurality of sub-jobs and to generate task results that are respectively associated with the plurality of sub-jobs and to move one or more sub-jobs to a sub-job retry queue based on an availability of resources, moving the one or more sub-jobs to the sub-job retry queue making the one or more sub-jobs visible to one or more processing modules of the peer-to-peer network to which the one or more sub-jobs were not previously visible; and

a pool of database servers, the peer-to-peer network of processing modules limits a number of sub-jobs that are scheduled based on a peak load associated with the pool of database servers.

2. The system of claim 1, wherein the job is utilized to add listings on a network-based marketplace, and wherein the peer-to-peer network of processing modules is to communicate a first request for listing data to the client machine based on a first sub-job that is associated with listing data for a first plurality of listings, and wherein the peer-to-peer network of processing modules is to communicate a second request for listing data to the client machine based on a second sub-job that is associated with listing data for a second plurality of listings and wherein the peer-to-peer network of processing modules is further to automatically add the first plurality of listings and the second plurality of listings to the network-based marketplace.

3. The system of claim 1, wherein the system includes a network-based marketplace and wherein the job is utilized to communicate transaction data from the network-based marketplace to the client machine and wherein the peer-to-peer network of processing modules is to communicate the transaction data for a first plurality of transactions to the client machine and to communicate the transaction data for a second plurality of transactions to the client machine.

4. The system of claim 1, wherein the resources include local resources and the peer-to-peer network of processing modules is configured to schedule the plurality of sub-jobs based on an availability of local resources that are utilized by the sub-jobs.

5. The system of claim 1, wherein the resources include remote resources and the peer-to-peer network of processing modules is configured to schedule the plurality of sub-jobs based on an availability of remote resources that are utilized by the sub-jobs.

6. The system of claim 1, wherein the peer-to-peer network of processing modules is configured to compute a priority of the job.

7. The system of claim 1, wherein the peak load associated with the pool of database servers is caused by a utilization of the data base servers by an entity other than the peer-to-peer network.

8. The system of claim 1, wherein the peer-to-peer network of processing modules identifies a database server that is not accessible and retries to access the database server.

9. The system of claim 1, wherein the peer-to-peer network of processing modules gathers task results into a job result.

10. A method to process a request, the method comprising:

receiving, over a network, a request from a client machine, the request comprising a job that is associated with data, the request received at an application program interface, the job indicating a set of tasks to be performed;

generating a plurality of sub-jobs by dividing the job into equal quantities of work, the job being utilized for retrieving transaction data from a network-based marketplace, the plurality of sub-jobs respectively comprising at least one task of the set of tasks to be performed that corresponds to a portion of the data, the generating performed by a processing node in a peer-to-peer network of processing nodes;

scheduling the plurality of sub-jobs for parallel processing based on an availability of resources that are respectively utilized by the sub-jobs, the resources including a pool of database servers, the scheduling including limiting a number of sub-jobs that are scheduled responsive to an identifying of a peak load associated with the pool of database servers, the scheduling being performed by a processing node in a peer-to-peer network of processing nodes

moving one or more sub-jobs of the plurality of sub-jobs to a sub-job retry queue based on an availability of resources, moving the one or more sub-jobs to the sub-job retry queue making the one or more sub-jobs visible to one or more processing nodes of the peer-to-peer network to which the one or more sub-jobs were not previously visible; and

parallel processing the sub-jobs with a plurality of processors to generate task results that are respectively associated with the plurality of sub-jobs, the parallel processing performed by at least one processing node in the peer-to-peer network of processing nodes.

11. The method of claim 10, wherein the job is utilized to add listings on a network-based marketplace, the method of claim 10 further comprising:

communicating a first request for listing data from the network-based marketplace to the client machine based on a first sub-job that is associated with listing data for a first plurality of listings;

communicating a second request for listing data from the network-based marketplace to the client machine based on a second sub-job that is associated with listing data for a second plurality of listings; and

automatically adding the first plurality of listings and the second plurality of listings to the network-based marketplace.

12. The method of claim 10, wherein the receiving of the request includes receiving a request to retrieve transaction data, and wherein the sub-jobs associated with the job include a first sub-job to retrieve transaction data for a first plurality of transactions, and a second sub-job to retrieve transaction data for a second plurality of transactions, and wherein the parallel processing of the sub-jobs includes:

communicating the transaction data for the first plurality of transactions to the client machine; and

communicating the transaction data for the second plurality of transactions to the client machine.

13. The method of claim 10, further comprising adding a node to the peer-to-peer network of processing nodes to increase processing and storage capacity of the peer-to-peer network.

14. The method of claim 10, further comprising removing a node from the peer-to-peer network of processing nodes to decrease processing and storage capacity of the peer-to-peer network.

15. The method of claim 10, wherein the resources include local resources and wherein the scheduling of the plurality of sub-jobs is based on an availability of local resources that are utilized by the sub-jobs.

16. The method of claim 10, wherein the resources include remote resources wherein the scheduling of the plurality of sub-jobs is based on an availability of remote resources that are utilized by the sub-jobs.

17. The method of claim 10, wherein the peak load associated with the pool of database servers is caused by a utilization of the database servers by an entity that utilizes the pool of database servers other than the peer-to-peer network.

18. The method of claim 10, further comprising identifying a pool of database servers that is not accessible and retrying to access the pool of database servers responsive to the identifying.

19. The method of claim 10, further including gathering task results into a job result, the gathering performed by a processing node in the peer-to-peer network of processing nodes.

21. The method of claim 10 further comprising computing a priority of the job prior to the scheduling the priority of sub-jobs and wherein the scheduling of the plurality of sub-jobs is based on the priority of the job.

20. A non-transitory machine-readable medium storing instructions that, when executed by at least one processor, cause the at least one processor to perform the following actions:

receive, over a network, a request from a client machine, the request comprising a job that is associated with data, the request received at an application program interface, the job indicating a set of tasks to be performed;

generate a plurality of sub-jobs by dividing the job into equal quantities of work, the job is utilized to retrieve transaction data from a network-based marketplace, the plurality of sub-jobs respectively comprising at least one task of the set of tasks to be performed that corresponds to a portion of the data;

schedule the plurality of sub-jobs for parallel processing based on an availability of resources that are respectively utilized by the sub-job, the resources include a pool of database servers, the schedule of the plurality of jobs includes a limitation of a number of sub-jobs that are scheduled responsive to an identification of a peak load associated with the pool of database servers

move one or more sub-jobs of the plurality of sub-jobs to a sub-job retry queue based on an availability of resources, moving the one or more sub-jobs to the sub-job retry queue making the one or more sub-jobs visible to one or more processing nodes of the peer-to-peer network to which the one or more sub-jobs were not previously visible; and

parallel process the sub-jobs with a plurality of processors to generate task results that are respectively associated with the sub-jobs.