Software performance profiling in a multi-tenant environment

• US 8,839,209 B2
• Filed: 02/23/2011
• Issued: 09/16/2014
• Est. Priority Date: 05/12/2010
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: Salesforce.com, Inc.
• Inventors: Gallagher, William, Zheng, Zuye, Hossain, Fiaz, Iry, James, Kulkarni, Nagraj
• Primary Examiner(s): Zhen, Li B
• Assistant Examiner(s): Paulino, Lenin
• Application Number:
  US13/033,566
• Publication Number:
  US 20110283266A1
• Time in Patent Office: 1,301 Days
• Field of Search: 717/130
• US Class Current: 717/130
• CPC Class Codes: G06F 11/3409 for performance assessment
• CPC Class Codes: G06F 11/3495 for systems
• CPC Class Codes: G06F 11/3612 by runtime analysis performance monitoring G06F11/3466
• CPC Class Codes: G06F 2201/815 Virtual middleware or OS functionality using virtual machines to implement generic software techniques for error detection or fault masking G06F11/1484
• CPC Class Codes: G06F 2201/865 Monitoring of software
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs

Abstract

A method for software performance analysis and debugging in a multi-tenant database network system is provided. In at least one embodiment, sampling is achieved using configuration files of each server cluster. Knowledge of the host names of each server in the cluster enables a profiler to target a single server for each sample, thereby facilitating a round-robin sample across a clustered server environment such that a CPU'"'"'s load and processing cost associated to sampling is fairly and uniformly distributed across all servers in the cluster. As a result, in at least one embodiment of the analysis and debugging tool each sample is a complete stack trace dump of each thread running on the application server at that moment in time.

Claims

1. A method for profiling a unit of code comprising:

on each of a plurality of servers of a cluster of servers running at least a unit of code that is the same as is running on others of the plurality of servers, the servers including a set of one or more processors;

sampling, by a host system having a set of one or more processors, performance information from the plurality of servers within the one or more server clusters, the sampling being performed uniformly across the plurality of servers in the cluster, the performance information being of the performance of the unit of code that is running on each of the servers of the plurality of severs, and each sample including at least a stack trace dump of each of a plurality of threads running on the server from the plurality of servers at a moment in time at which the sample was taken, the plurality of threads being a plurality of tasks running concurrently, and each stack trace dump of each thread including at least an ordered list of current methods being executed followed by a subsequent method that called the current method;

aggregating the stack trace dumps of each thread over the samples to generate both a call graph and a back trace graph;

generating, by the one or more processors of the host system, a performance profile characterizing performance of the unit of code running on the plurality of servers based on the sampling, the performance profile characterizes the performance of the unit of code, the performance profile includes at least a view having performance information from each server of the plurality of servers combined and formatted as one unified profile; and

storing, by the host system, the performance profile on one or more non-transitory computer readable medium for later analysis.

2. The method of claim 1, the storing of the performance profile including at least updating a profiling log for storing the performance information from a series of profiling runs for the plurality of servers, by at least entering the performance information of a current profiling run into the profiling log, and storing the updating.

3. The method of claim 1, further comprising:

creating a schedule for running profiles for the plurality of servers.

4. The method of claim 1, the sampling being performed at time intervals specified in a schedule that specifies how many samplings of the unit of code occur within a given time period.

5. The method of claim 1, further comprising:

accessing configuration files of each server cluster;

determining names of servers in a cluster, based on the configuration files accessed; and

determining a schedule for gathering performance information about each server of the plurality of servers.

6. The method of claim 1, the sampling of plurality of servers being performed according to a round-robin sampling schedule.

7. The method of claim 1, where the sampling across the plurality of servers is load balanced by sampling in equal time sample intervals at each of the plurality of servers in the server cluster, a profiler selecting a host name from a collection of host names that includes one host name for each of the plurality of servers in the server cluster, and the profiler targeting a single server from the plurality of servers for each sample.

8. The method of claim 1, further comprising:

selective profiling by isolating tasks of threads to a single entity;

where the isolating is carried out by at least

tagging each thread with an ID during sampling, and

creating filters to run profiles on threads based on IDs resulting in selective profiling.

9. The method of claim 1, further comprising:

providing a graphical user interface for analysis of the one or more server clusters.

10. The method of claim 9, where the graphical user interface further comprises a chart of a group of methods having a highest number of method calls listed according to percentage of method calls.

11. The method of claim 9, where the graphical user interface further comprises a user expandable call graph tree.

12. The method of claim 9, where the graphical user interface further comprises a user expandable back trace tree.

13. The method of claim 9, where the graphical user interface further comprises a pie chart of method calls to different methods, each slice of the pie representing how many method calls were made to a particular method as a percentage of a total of all method calls to all methods in the pie chart.

14. The method of claim 9, where the graphical user interface further comprises sampling information parameters.

15. The method of claim 1, the generating of the profile information being performed asynchronously with the storing of the performance information.

16. The method of claim 1, where the view is not divided according to server.

17. The method of claim 1, the unit of code including one or more machine instructions for implementing a multitenant, on demand, relational database system.

18. A non-transitory machine-readable medium carrying one or more instructions for profiling a system, which when executed cause a method to be carried out, the method comprising:

creating, by a host system, a profiling log for storing a series of profiling runs at the host system for one or more server clusters in the multi-tenant database network system;

creating, by the host system, a schedule for running profiles for the one or more server clusters;

determining, by the host system, from the schedule when a server cluster from the one or more server clusters requires a profile run;

wherein when the server cluster requires a profile run,

sampling, by the host system, a plurality of servers within the server cluster, wherein the sampling is performed uniformly across the plurality of servers in the cluster at a set interval for a set duration, each sample including at least a stack trace dump of each of a plurality of threads running on the server from the plurality of servers at a moment in time at which the sample was taken, the plurality of threads being a plurality of tasks running concurrently, and each stack trace dump of each thread including at least an ordered list of a current methods being executed followed by a subsequent method that called the current method;

aggregating the stack trace dumps of each thread over the samples to generate both a call graph and a back trace graph;

generating, by the host system, a performance profile based on sample sets formed from the sampling;

storing, by the host system, the performance profile; and

retrieving, by the host system, the performance profile for analysis.

19. A computer network configured for profiling a system, the computer network comprising:

a plurality of clusters of servers each having a processor system including at least one processor; and

a memory system including a machine readable medium having stored thereon one or more sequences of instructions which, when executed, cause a method to be carried out, the method comprising;

creating, by a host system, a profiling log for storing a series of profiling runs at the host system for one or more server clusters in the multi-tenant database network system;

creating, by the host system, a schedule for running profiles for the one or more server clusters;

determining, by the host system, from the schedule when a server cluster from the one or more server clusters requires a profile run;

wherein when the server cluster requires a profile run,

sampling, by the host system, a plurality of servers within the server cluster, wherein the sampling is performed uniformly across the plurality of servers in the cluster at a set interval for a set duration, each sample including at least a stack trace dump of each of a plurality of threads running on the server from the plurality of servers at a moment in time at which the sample was taken, the plurality of threads being a plurality of tasks running concurrently, and each stack trace dump of each thread including at least an ordered list of a current methods being executed followed by a subsequent method that called the current method;

aggregating the stack trace dumps of each thread over the samples to generate both a call graph and a back trace graph;

generating, by the host system, a performance profile based on sample sets formed from the sampling;

storing, by the host system, the performance profile; and

retrieving, by the host system, the performance profile for analysis.