METHODS AND SYSTEMS FOR BULK UPLOADING OF DATA IN AN ON-DEMAND SERVICE ENVIRONMENT

• US 20110246434A1
• Filed: 12/20/2010
• Published: 10/06/2011
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Joergensen, Jesper, Cheenath, Manoj, Fell, Simon Z.
• Granted Patent: US 8,965,860 B2
• Time in Patent Office: Days
• US Class Current: 707/703
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]

Abstract

Methods and systems for bulk uploading of data in an on-demand service environment are described. In one embodiment, such a method includes retrieving a bulk transaction request from a transaction queue within a host organization, where the bulk transaction request specifies a plurality of database transactions to be processed against a multi-tenant database system of the host organization. Such a method further includes determining a current computational load of the multi-tenant database system, processing a subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below a threshold, and updating a bulk transaction results log based on the subset of the plurality of database transactions processed. Remaining database transactions specified by the bulk transaction request may then be re-queued for later processing.

Claims

1. A method in a host organization, wherein the method comprises:

retrieving a bulk transaction request from a transaction queue, the bulk transaction request specifying a plurality of database transactions to be processed against a multi-tenant database system of the host organization;

determining a current computational load of the multi-tenant database system;

processing a subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below a threshold; and

updating a bulk transaction results log based on the subset of the plurality of database transactions processed.

2. The method of claim 1, further comprising:

receiving the bulk transaction request at the host organization; and

inserting the bulk transaction request into the transaction queue among a plurality of bulk transaction requests and a plurality of real-time transaction requests in the transaction queue.

3. The method of claim 2:

wherein the plurality of real-time transaction requests are assigned a higher priority than the plurality of bulk transaction requests and are selected for processing against the multi-tenant database system based on the higher priority; and

wherein the plurality of bulk transaction requests are assigned a lower priority than the plurality of real-time transaction requests and are selected for processing against the multi-tenant database system based on the lower priority and based further on the determined current computational load of the multi-tenant database system;

4. The method of claim 3, further comprising:

sending a response to a client device having submitted the bulk transaction request, wherein the response specifies a session identifier assigned to the bulk transaction request, and further wherein the response acknowledges the bulk transaction request is queued for asynchronous processing.

5. The method of claim 3, further comprising:

receiving a real-time transaction request at the host organization subsequent to receiving the bulk transaction request;

processing the real-time transaction request against the multi-tenant database system before processing any part of the bulk transaction request based on the higher priority of the real-time transaction request; and

sending a response to a client device having submitted the real-time transaction request, wherein the response specifies a result for the real-time transaction request and further wherein the response indicates synchronous processing of the real-time transaction request is complete.

7. The method of claim 1, further comprising:

validating the bulk transaction request at the host organization against one or more specifications.

8. The method of claim 7, wherein:

validating the bulk transaction request comprises;

verifying the bulk transaction request complies with a maximum record count threshold, verifying the bulk transaction request complies with a maximum size threshold, verifying the bulk transaction request complies with an accepted request format; and

wherein the method further comprises;

submitting the bulk transaction request to the transaction queue when the bulk transaction request is successfully validated.

9. The method of claim 1:

wherein the bulk transaction request comprises a bulk record upload request; and

wherein each of the plurality of database transactions specified by the bulk transaction request comprises a record and/or row to be uploaded to a table within the multi-tenant database system.

10. The method of claim 1, wherein each of the plurality of database transactions specified by the bulk transaction request comprises:

a transaction type selected from add, update, or delete;

a named table within the multi-tenant database system against which to process the respective database transaction based on the transaction type; and

transaction data to be added, updated, or selected for deletion based on the transaction type during the respective database transaction.

11. The method of claim 1, wherein determining the current computational load of the multi-tenant database system comprises:

retrieving one or more operational statistics from the multi-tenant database system, each operating statistic describing Input/Output utilization of the multi-tenant database system;

CPU utilization of the multi-tenant database system;

number of free server nodes available for processing database transactions within a pool of server nodes;

number of active threads currently processing database transactions against the multi-tenant database system;

current queue time of real-time database transactions to be processed against the multi-tenant database system; and

number of queued real-time database transactions to be processed against the multi-tenant database system.

12. The method of claim 11, wherein processing the subset of the plurality of database transactions when the current computational load is below the threshold comprises:

determining computational resources are available for processing bulk transaction requests based on a comparison of the one or more operational statistics against a corresponding one or more operational thresholds;

processing the subset of the plurality of database transactions against the multi-tenant database system, wherein processing the subset comprises processing the plurality of database transactions for a pre-determined period of time or processing a pre-determined number of the plurality of database transactions; and

re-queuing the bulk transaction request in the transaction queue having the subset of the plurality of database transactions processed and having a remaining portion of the plurality of database transactions unprocessed.

13. The method of claim 1:

wherein the updated bulk transaction results log comprises a transaction result for each of the subset of database transactions processed; and

transmitting the contents of the updated bulk transaction results log to a customer organization responsive to a request.

14. The method of claim 1, further comprising:

issuing a job status for the bulk transaction request indicating one or more of;

completion status;

queue status, percentage complete, number of records transacted, number of records successfully transacted, number of transaction errors;

total aggregate queue time;

total aggregate processing time; and

estimated time to completion; and

returning the job status information to a customer organization responsive to a status request from the customer organization identifying the bulk transaction request.

15. The method of claim 1, further comprising:

re-retrieving a bulk transaction request from the transaction queue;

re-determining the current computational load of the multi-tenant database system;

processing a second subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below a threshold, wherein the second subset is non-overlapping with the first subset of the plurality of database transactions;

re-updating the bulk transaction results log based on the second subset of the plurality of database transactions processed; and

issuing a job status for the bulk transaction request.

16. The method of claim 1, wherein processing the subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below the threshold comprises:

selecting serial processing when the current computational load is below the threshold by a first margin, wherein serial processing comprises issuing the subset of database transactions to one server node within a pool of server nodes for processing; and

selecting parallel processing when the current computational load is below the threshold by a second margin, wherein parallel processing comprises a) segmenting the subset of database transactions into a plurality of parts, and b) issuing the plurality of parts constituting the subset of database transactions to a plurality of server nodes within the pool of server nodes for processing in parallel.

17. The method of claim 1, wherein the bulk transaction request is one of a plurality of requests received from a plurality of customer organizations, wherein each customer organization is an entity selected from the group consisting of:

a separate and distinct remote organization, an organizational group within the host organization, a business partner of the host organization, or a customer organization that subscribes to cloud computing services provided by the host organization.

18. The method of claim 1, wherein the multi-tenant database system comprises elements of hardware and software that are shared by a plurality of separate and distinct customer organizations, each of the separate and distinct customer organizations being remotely located from the host organization having the multi-tenant database system operating therein.

19. The method of claim 1, further comprising:

receiving a second one or more bulk transaction requests at a file repository of the host organization;

associating the bulk transaction request and the second one or more additional bulk transaction requests with one of a plurality customer organizations based on an Organization ID (OrgID) and/or a User ID (UserID) accompanying each bulk transaction request;

loading all received bulk transaction requests into the transaction queue; and

wherein retrieving the bulk transaction request from the transaction queue comprises selecting, on a round robin scheduling basis, one bulk transaction request from among all bulk transaction requests in the transaction queue.

20. The method of claim 1, further comprising:

marking each of the subset of the plurality of database transactions specified by the bulk transaction request as processed;

re-queuing the bulk transaction request in the transaction queue based on a determination that one or more database transactions specified by the bulk transaction request remain unprocessed; and

finalizing the bulk transaction results log based on a determination that all database transactions specified by the bulk transaction request are marked as processed.

21. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processor in a host organization, cause the host organization to perform a method comprising:

retrieving a bulk transaction request from a transaction queue, the bulk transaction request specifying a plurality of database transactions to be processed against a multi-tenant database system of the host organization;

determining a current computational load of the multi-tenant database system;

processing a subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below a threshold; and

updating a bulk transaction results log based on the subset of the plurality of database transactions processed.

22. The non-transitory computer readable storage medium of claim 21, wherein the method further comprises:

re-queuing the bulk transaction request in the transaction queue having the subset of the plurality of database transactions processed and having a remaining portion of the plurality of database transactions unprocessed; and

issuing a job status for the bulk transaction request indicating completion status of the bulk transaction request.

23. A host organization comprising:

a processor and a memory;

web-server to receive a bulk transaction request from a customer organization, the bulk transaction request specifying a plurality of database transactions to be processed against a multi-tenant database system of the host organization;

a transaction queue to queue the bulk transaction request for processing among a plurality of pending transactions;

a load analyzer to determine a current computational load of the multi-tenant database system;

a load balancer to retrieve the bulk transaction request from the transaction queue; and

a distribution module to distribute at least a subset of the plurality of database transactions specified by the bulk transaction to the multi-tenant database system for processing when the current computational load is below a threshold.

24. The host organization of claim 23, further comprising:

a results module to update a bulk transaction results log based on the subset of the plurality of database transactions processed and to further issue a job status for the bulk transaction request indicating completion status of the bulk transaction request.