Methods and systems for bulk uploading of data in an on-demand service environment

• US 8,965,860 B2
• Filed: 12/20/2010
• Issued: 02/24/2015
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Cheenath, Manoj, Fell, Simon Z., Joergensen, Jesper
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): GMAHL, NAVNEET K
• Application Number:
  US12/973,687
• Publication Number:
  US 20110246434A1
• Time in Patent Office: 1,527 Days
• Field of Search: 707702-705
• US Class Current: 707/703
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/2386 Bulk updating operations data conversion details G06F16/258
• CPC Class Codes: G06F 16/273 Asynchronous replication or reconciliation
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]

Abstract

Methods and systems for bulk uploading of data in an on-demand service environment are described. In one embodiment, such a method includes retrieving a bulk transaction request from a transaction queue within a host organization, where the bulk transaction request specifies a plurality of database transactions to be processed against a multi-tenant database system of the host organization. Such a method further includes determining a current computational load of the multi-tenant database system, processing a subset of the plurality of database transactions specified by the bulk transaction request against the multi-tenant database system when the current computational load is below a threshold, and updating a bulk transaction results log based on the subset of the plurality of database transactions processed. Remaining database transactions specified by the bulk transaction request may then be re-queued for later processing.

Claims

1. A method in a host organization, wherein the method comprises:

receiving, from a client device communicatively interfaced to the host organization, a bulk transaction request specifying a plurality of database transactions to be processed against a database system of the host organization;

sending a response to the client device having submitted the bulk transaction request, wherein the response provides at least;

(a) an indication of successful receipt of the bulk transaction request and (b) an acknowledgement that the bulk transaction request is queued for processing or queued for asynchronous processing;

retrieving the bulk transaction request from a transaction queue;

determining a current computational load of the database system;

processing a subset of the plurality of database transactions specified by the bulk transaction request against the database system when the current computational load is below a threshold;

updating a bulk transaction results log based on the subset of the plurality of database transactions processed;

wherein the plurality of real-time transaction requests are assigned a higher priority than the plurality of bulk transaction requests and are selected for processing against the database system based on the higher priority; and

wherein the plurality of bulk transaction requests are assigned a lower priority than the plurality of real-time transaction requests and are selected for processing against the database system based on the lower priority and based further on the determined current computational load of the database system.

2. The method of claim 1, further comprising:

inserting the bulk transaction request into the transaction queue among a plurality of bulk transaction requests and a plurality of real-time transaction requests in the transaction queue.

3. The method of claim 1, wherein the response specifies a session identifier assigned to the bulk transaction request, and further wherein the response specifies the bulk transaction request is queued for asynchronous processing.

4. The method of claim 1, further comprising:

receiving a real-time transaction request at the host organization subsequent to receiving the bulk transaction request;

processing the real-time transaction request against the database system before processing any part of the bulk transaction request based on the higher priority of the real-time transaction request; and

sending a new response to a client device having submitted the real-time transaction request, wherein the response specifies a result for the real-time transaction request and further wherein the response indicates synchronous processing of the real-time transaction request is complete.

5. The method of claim 1,

wherein the response does not indicate that the bulk transaction has completed but instead indicates successful receipt of the bulk transaction request;

and

wherein the response from the host organization further provides information to the client device to enable the client device to check for appropriate status at a later time or to retrieve the bulk transaction results log describing the results of the plurality of database transactions incorporated within the bulk transaction request submitted to the host organization.

6. The method of claim 1, further comprising:

validating the bulk transaction request at the host organization against one or more specifications.

7. The method of claim 6, wherein:

validating the bulk transaction request comprises;

verifying the bulk transaction request complies with a maximum record count threshold,

verifying the bulk transaction request complies with a maximum size threshold,

verifying the bulk transaction request complies with an accepted request format; and

wherein the method further comprises;

submitting the bulk transaction request to the transaction queue when the bulk transaction request is successfully validated.

8. The method of claim 1:

wherein the bulk transaction request comprises a bulk record upload request; and

wherein each of the plurality of database transactions specified by the bulk transaction request comprises a record and/or row to be uploaded to a table within the database system.

9. The method of claim 1, wherein each of the plurality of database transactions specified by the bulk transaction request comprises:

a transaction type selected from add, update, or delete;

a named table within the database system against which to process the respective database transaction based on the transaction type; and

transaction data to be added, updated, or selected for deletion based on the transaction type during the respective database transaction.

10. The method of claim 1, wherein determining the current computational load of the database system comprises:

retrieving one or more operational statistics from the database system, each operating statistic describing Input/Output utilization of the database system;

CPU utilization of the database system;

number of free server nodes available for processing database transactions within a pool of server nodes;

number of active threads currently processing database transactions against the database system;

current queue time of real-time database transactions to be processed against the database system; and

number of queued real-time database transactions to be processed against the database system.

11. The method of claim 10, wherein processing the subset of the plurality of database transactions when the current computational load is below the threshold comprises:

determining computational resources are available for processing bulk transaction requests based on a comparison of the one or more operational statistics against a corresponding one or more operational thresholds;

processing the subset of the plurality of database transactions against the database system, wherein processing the subset comprises processing the plurality of database transactions for a pre-determined period of time or processing a pre-determined number of the plurality of database transactions; and

re-queuing the bulk transaction request in the transaction queue having the subset of the plurality of database transactions processed and having a remaining portion of the plurality of database transactions unprocessed.

12. The method of claim 1:

wherein the updated bulk transaction results log comprises a transaction result for each of the subset of database transactions processed; and

transmitting the contents of the updated bulk transaction results log to a customer organization responsive to a request.

13. The method of claim 1, further comprising:

issuing a job status for the bulk transaction request indicating one or more of;

completion status;

queue status, percentage complete, number of records transacted, number of records successfully transacted, number of transaction errors;

total aggregate queue time;

total aggregate processing time; and

estimated time to completion; and

returning the job status information to a customer organization responsive to a status request from the customer organization identifying the bulk transaction request.

14. The method of claim 1, further comprising:

re-retrieving a bulk transaction request from the transaction queue;

re-determining the current computational load of the database system;

processing a second subset of the plurality of database transactions specified by the bulk transaction request against the database system when the current computational load is below a threshold, wherein the second subset is non-overlapping with the first subset of the plurality of database transactions;

re-updating the bulk transaction results log based on the second subset of the plurality of database transactions processed; and

issuing a job status for the bulk transaction request.

15. The method of claim 1, wherein processing the subset of the plurality of database transactions specified by the bulk transaction request against the database system when the current computational load is below the threshold comprises:

selecting serial processing when the current computational load is below the threshold by a first margin, wherein serial processing comprises issuing the subset of database transactions to one server node within a pool of server nodes for processing; and

selecting parallel processing when the current computational load is below the threshold by a second margin, wherein parallel processing comprises a) segmenting the subset of database transactions into a plurality of parts, and b) issuing the plurality of parts constituting the subset of database transactions to a plurality of server nodes within the pool of server nodes for processing in parallel.

16. The method of claim 1, wherein the bulk transaction request is one of a plurality of requests received from a plurality of customer organizations, wherein each customer organization is an entity selected from the group consisting of:

a separate and distinct remote organization, an organizational group within the host organization, a business partner of the host organization, or a customer organization that subscribes to cloud computing services provided by the host organization.

17. The method of claim 1, wherein the database system comprises elements of hardware and software that are shared by a plurality of separate and distinct customer organizations, each of the separate and distinct customer organizations being remotely located from the host organization having the multi tenant database system operating therein.

18. The method of claim 1, further comprising:

receiving a second one or more bulk transaction requests at a file repository of the host organization;

associating the bulk transaction request and the second one or more additional bulk transaction requests with one of a plurality customer organizations based on an Organization ID (OrgID) and/or a User ID (UserID) accompanying each bulk transaction request;

loading all received bulk transaction requests into the transaction queue; and

wherein retrieving the bulk transaction request from the transaction queue comprises selecting, on a round robin scheduling basis, one bulk transaction request from among all bulk transaction requests in the transaction queue.

19. The method of claim 1, further comprising:

marking each of the subset of the plurality of database transactions specified by the bulk transaction request as processed;

re-queuing the bulk transaction request in the transaction queue based on a determination that one or more database transactions specified by the bulk transaction request remain unprocessed; and

finalizing the bulk transaction results log based on a determination that all database transactions specified by the bulk transaction request are marked as processed.

20. Non-transitory computer readable storage medium having instructions stored thereon that, when executed by a processor in a host organization, the instructions cause the host organization to perform operations comprising:

receiving, from a client device communicatively interfaced to the host organization, a bulk transaction request specifying a plurality of database transactions to be processed against a database system of the host organization;

sending a response to the client device having submitted the bulk transaction request, wherein the response provides at least;

(a) an indication of successful receipt of the bulk transaction request and (b) an acknowledgement that the bulk transaction request is queued for processing or queued for asynchronous processing;

retrieving the bulk transaction request from a transaction queue;

determining a current computational load of the database system;

processing a subset of the plurality of database transactions specified by the bulk transaction request against the database system when the current computational load is below a threshold;

updating a bulk transaction results log based on the subset of the plurality of database transactions processed;

wherein the plurality of real-time transaction requests are assigned a higher priority than the plurality of bulk transaction requests and are selected for processing against the database system based on the higher priority; and

wherein the plurality of bulk transaction requests are assigned a lower priority than the plurality of real-time transaction requests and are selected for processing against the database system based on the lower priority and based further on the determined current computational load of the database system.

21. The non-transitory computer readable storage medium of claim 20, wherein the method further comprises:

re-queuing the bulk transaction request in the transaction queue having the subset of the plurality of database transactions processed and having a remaining portion of the plurality of database transactions unprocessed; and

issuing a job status for the bulk transaction request indicating completion status of the bulk transaction request.

22. A host organization comprising:

a processor and a memory;

receiving via a web-server, a bulk transaction request from a client, the bulk transaction request specifying a plurality of database transactions to be processed against a database system of the host organization;

queuing via a transaction queue, the bulk transaction request for processing among a plurality of pending transactions;

sending via the web-server, a response to the client device having submitted the bulk transaction request, wherein the response provides at least;

(a) an indication of successful receipt of the bulk transaction request and (b) an acknowledgement that the bulk transaction request is queued for processing or queued for asynchronous processing;

determining via a load analyzer, a current computational load of the database system;

receiving via a load balancer, the bulk transaction request from the transaction queue;

distributing via a distribution module, at least a subset of the plurality of database transactions specified by the bulk transaction to the database system for processing when the current computational load is below a threshold;

wherein the plurality of real-time transaction requests are assigned a higher priority than the plurality of bulk transaction requests and are selected for processing against the database system based on the higher priority; and

wherein the plurality of bulk transaction requests are assigned a lower priority than the plurality of real-time transaction requests and are selected for processing against the database system based on the lower priority and based further on the determined current computational load of the database system.

23. The host organization of claim 22, further comprising:

updating via a results module, a bulk transaction results log based on the subset of the plurality of database transactions processed and to further issue a job status for the bulk transaction request indicating completion status of the bulk transaction request.