Method, apparatus, and program for eliminating thread skew in multithreaded performance benchmarks

• US 20030055764A1
• Filed: 09/20/2001
• Published: 03/20/2003
• Est. Priority Date: 09/20/2001
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Arora, Rajiv, Gordon, Jesse Mannes
• Granted Patent: US 7,257,516 B2
• Time in Patent Office: Days
• US Class Current: 705/36
• CPC Class Codes: G06F 11/3428 Benchmarking
• CPC Class Codes: G06Q 20/40 Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
• CPC Class Codes: G06Q 40/06 Asset management; Financial planning or analysis

Abstract

A data processing system performs a multithreaded performance benchmark with a rampup interval and a rampdown interval. The master thread signals a start test event and begins the rampup interval. After the rampup interval, the master thread signals a start measurement event. In response to the start measurement event, the worker threads record the units of work they complete. After the measurement interval, the master signals the workers to stop measuring, but to continue running. This begins the rampdown interval. After the rampdown interval, the master thread signals a stop test event. The rampup and rampdown intervals are long enough to ensure that measurements are not recorded during skew intervals. Thus, thread skew does not impact the results.

Claims

1. A method for performing a benchmark, comprising:

signaling a start test event;

waiting during a rampup interval;

signaling a start measurement event at the end of the rampup interval;

collecting measurement results; and

signaling a stop test event.

2. The method of claim 1, wherein events are signaled by setting a shared memory flag.

3. The method of claim 1, wherein events are signaled using operating system signals.

4. The method of claim 1, further comprising starting a plurality of worker threads during the rampup interval.

5. The method of claim 4, wherein the length of the rampup interval is determined based on the number of worker threads in the plurality of worker threads.

6. The method of claim 4, wherein the step of collecting measurement results comprises:

signaling a stop measurement event; and

receiving the measurement results from the worker threads.

7. The method of claim 6, further comprising:

waiting during a rampdown interval after the stop measurement event; and

signaling the stop test event at the end of the rampdown interval.

8. The method of claim 7, wherein the length of the rampdown interval is determined based on the number of worker threads in the plurality of worker threads.

9. The method of claim 1, further comprising:

repeatedly increasing the rampup interval and performing the benchmark until the increase of the rampup interval has no measurable impact on the measurement results.

10. The method of claim 1, wherein the step of collecting measurement results comprises:

repeatedly running an iteration of a test until a plurality of metrics have converged.

11. The method of claim 10, further comprising, for each iteration of the test:

computing a plurality of metrics;

computing confidence intervals for each of the plurality of metrics; and

determining whether each confidence interval is less than a respective threshold, wherein the plurality of metrics have converged if all confidence intervals are less than their respective thresholds.

12. A method for performing a benchmark, comprising:

starting a plurality of worker threads;

waiting during a rampup interval;

signaling a start measurement event at the end of the rampup interval;

receiving measurement results from the worker threads;

signaling a stop measurement event;

waiting during a rampdown interval; and

signaling a stop test event at the end of the rampdown interval.

13. An apparatus for performing a benchmark, comprising:

first signal means for signaling a start test event;

rampup means for waiting during a rampup interval;

second signal means for signaling a start measurement event at the end of the rampup interval;

collection means for collecting measurement results; and

third signal means for signaling a stop test event.

14. The apparatus of claim 13, wherein events are signaled by setting a shared memory flag.

15. The apparatus of claim 13, wherein events are signaled using operating system signals.

16. The apparatus of claim 13, further comprising means for starting a plurality of worker threads during the rampup interval.

17. The apparatus of claim 16, wherein the collection means comprises:

means for signaling a stop measurement event; and

means for receiving the measurement results from the worker threads.

18. The apparatus of claim 17, further comprising:

means for waiting during a rampdown interval after the stop measurement event; and

means for signaling the stop test event at the end of the rampdown interval.

19. The apparatus of claim 13, further comprising:

means for repeatedly increasing the rampup interval and performing the benchmark until the increase of the rampup interval has no measurable impact on the measurement results.

20. The apparatus of claim 13, wherein the collection means comprises:

iteration means for repeatedly running an iteration of a test until a plurality of metrics have converged.

21. An apparatus for performing a benchmark, comprising:

means for starting a plurality of worker threads;

means for waiting during a rampup interval;

means for signaling a start measurement event at the end of the rampup interval;

means for receiving measurement results from the worker threads;

means for signaling a stop measurement event;

means for waiting during a rampdown interval; and

means for signaling a stop test event at the end of the rampdown interval.

22. A computer program product, in a computer readable medium for performing a benchmark, comprising:

instructions for signaling a start test event;

instructions for waiting during a rampup interval;

instructions for signaling a start measurement event at the end of the rampup interval;

instructions for collecting measurement results; and

instructions for signaling a stop test event.

23. A computer program product, in a computer readable medium, for performing a benchmark, comprising:

instructions for starting a plurality of worker threads;

instructions for waiting during a rampup interval;

instructions for signaling a start measurement event at the end of the rampup interval;

instructions for receiving measurement results from the worker threads;

instructions for signaling a stop measurement event;

instructions for waiting during a rampdown interval; and

instructions for signaling a stop test event at the end of the rampdown interval.