Distributed testing for computing features

• US 7,823,138 B2
• Filed: 11/14/2006
• Issued: 10/26/2010
• Est. Priority Date: 11/14/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gallegos, Gustavo, Arguelles, Carlos A.
• Primary Examiner(s): Zhen; Wei Y
• Assistant Examiner(s): Coyer; Ryan D
• Application Number:
  US11/599,022
• Publication Number:
  US 20080115107A1
• Time in Patent Office: 1,442 Days
• Field of Search: 717124-135, 714/38, 714/100
• US Class Current: 717/131
• CPC Class Codes: G06F 11/3668 Software testing software testing in telephone exchanges H04M3/242, testing of hardware G06F11/22
• CPC Class Codes: G06F 2209/5017 Task decomposition
• CPC Class Codes: G06F 9/5027 the resource being a machine, e.g. CPUs, Servers, Terminals

Abstract

A new method is provided for using distributed computing for computing process testing. The method includes distributing a feature testing tool, computing feature builds, and data sets to client computing systems. The feature testing tool is run on the client computing systems during times when the client computing system is not otherwise in active use. The feature testing tool prompts the computing feature build to process the data sets, and collects results from the processing of the data set, including any erroneous results. The results from processing the data sets on the client computing systems are provided to a managing computing system. The managing computing system may try to reproduce the erroneous results on other client computing systems with alternate hardware-software configurations, to send erroneous results to an investigation queue to isolate their cause, and to report useful information on the erroneous results to developers working on the computing feature.

Claims

1. A computer-implemented method comprising:

distributing a feature testing tool to one or more client computing systems;

distributing one or more computing feature builds from a managing computing system to the client computing systems;

distributing one or more data sets from the managing computing system to the client computing systems,

wherein the data sets are selected for distribution based on their priority in a cyclical queue of data sets,

wherein the managing computing system responds to erroneous results received from a first client computing system by assigning a mandate for a data set associated with the erroneous results to be sent to a second client computing system which has a different hardware-software configuration than that of the client computing system that originally reported the erroneous results,

wherein the different hardware-software configuration comprises at least one of a different type of central processing unit, a different clock speed, a different amount of random access memory (RAM), a different operating system, a different operating system version, a different set of concurrently loaded software, a different set of concurrently running software, and a different set of concurrently running processes,

wherein the managing computing system receives results from the second client computing system, and, if the erroneous results were not reproduced on the second client computing system, responds by sending the data set associated with the erroneous results back to the client computing system that originally reported the erroneous results, and

wherein the managing computing system responds to receiving a reproduction of the erroneous results by providing the data set associated with the erroneous results, along with information about how the erroneous results were produced and reproduced, including whether the erroneous results could be reproduced with the different hardware-software configuration, to an investigation queue;

running the feature testing tool on a given one of the client computing systems during times when the feature testing tool determines that the client computing system is not otherwise in active use, wherein the feature testing tool prompts the computing feature build to process one of the data sets, and collects results from the processing of the data set, including any erroneous results; and

providing to the managing computing system the results from the computing feature build processing the data sets on the client computing systems.

2. The method of claim 1, further comprising distributing a new one of the data sets from the managing computing system to one of the client computing systems after the client computing system provides the results from the computing feature build processing the previous data set to the managing computing system, wherein the feature testing tool on the client computing system prompts the computing feature build to process the new data set.

3. The method of claim 1, wherein the managing computer system derives the data sets as segments from a larger data collection, selected from among one or more data collections intended for processing by the computing feature build that are accessible to the managing computer system.

4. The method of claim 1, wherein each of the data sets in the queue has a priority based at least in part on a duration of time that has passed since a most recent occurrence of that data set being processed by a computing feature build.

5. The method of claim 4, further comprising adding data segments to the queue and removing data segments from the queue while the queue is in active use, wherein data sets newly added to the queue are prioritized above the data sets that have been processed at least once by a computing feature build.

6. The method of claim 4, wherein the managing computing system responds to erroneous results by assigning the data set associated with the erroneous results a higher priority in the queue than any of the data sets not associated with an unresolved erroneous result.

7. The method of claim 1, wherein the data sets provided to the investigation queue are iteratively divided into smaller segments, and the divided segments processed by the computing feature build, until the one or more smallest segments of the data set that reproduce the erroneous results are isolated, and information about the erroneous results and the isolated segments is provided.

8. The method of claim 7, wherein if the erroneous results are not reproduced by any of the divided segments individually, then out-of-order permutations of the divided segments are processed by the computing feature build until one or more smallest out-of-order permutations of the divided segments that reproduce the erroneous results are isolated.

9. The method of claim 1, further comprising identifying preferred new client computing systems that had not previously received the feature testing tool, and distributing the feature testing tool to one or more of the preferred new client computing systems.

10. The method of claim 9, wherein the preferred new client computing systems comprise client computing systems that have a hardware-software configuration that was either absent or underrepresented among the client computing systems that had previously received the feature testing tool and provided results to the managing computing system.

11. The method of claim 1, wherein the computing feature build comprises a natural language feature, comprising one or more of:

a spelling checker, a grammar checker, a style checker, a hyphenator, a thesaurus, a suggested correction list, a summarizer, a dictionary, a translation feature, an alternative font, an input method editor, a part-of-speech tagger, a tokenizer, a chunker, a word breaker, a parser, an inconsistent parse alerter, a script version converter, a pronunciation displayer, a stroke count displayer, a language detector, or a non-native language usage detector.

12. The method of claim 1, wherein the computing feature build comprises a natural language feature, in one or more languages selected from among Arabic, Basque, Bulgarian, Catalan, Simplified Chinese, Traditional Chinese, Croatian, Czech, Danish, Dutch, English, Estonian, Finnish, French, Galician, German, Greek, Gujarati, Hebrew, Hindi, Hungarian, Icelandic, Indonesian, Irish Gaelic, Italian, Japanese, Kannada, Korean, Latvian, Lithuanian, Marathi, Bokml Norwegian, Nynorsk Norwegian, Polish, Brazilian Portuguese, European Portuguese, Punjabi, Romanian, Russian, Serbian, Slovak, Slovenian, Spanish, Swedish, Tamil, Telugu, Thai, Turkish, Ukranian, Vietnamese, or Welsh.

13. The method of claim 1, wherein the feature testing tool determines that the client computing system is not otherwise in active use at least in part by consulting an anticipated usage schedule previously entered by a primary user of the client computing system.

14. The method of claim 1, wherein the feature testing tool determines that the client computing system is not otherwise in active use at least in part by consulting with calendar data from an information manager associated with a primary user of the client computing system.

15. The method of claim 1, wherein the feature testing tool determines that the client computing system is not otherwise in active use at least in part by consulting network-available indications that a primary user of the client computing system is in a remote location from the client computing system.

16. The method of claim 1, further comprising:

collecting on the managing computing system the results of the computing feature build processing the data sets, and thereby characterizing performance aspects of the computing feature build; and

distributing a revised computing feature build to the one or more client computing systems, wherein the revised computing feature build has been revised to improve one or more of the performance aspects that were identified in the prior computing feature build through the results provided by the client computing systems.

17. A computing system comprising a processor capable of executing instructions from a memory and configured to:

distribute a natural language testing utility to one or more remote computing devices having a variety of different hardware-software configurations;

iteratively distribute test versions of a natural language feature to the remote computing devices;

iteratively distribute corpus segments from a cyclical queue of corpus segments to the remote computing devices to test the natural language feature on

wherein the corpus segments are selected for distribution based on their priority in the cyclical queue of corpus segments,

wherein a managing computing system responds to erroneous results received from a first remote computing device by assigning a mandate for a corpus segment associated with the erroneous results to be sent to a second remote computing system which has a different hardware-software configuration than that of the client computing system that originally reported the erroneous results, and

wherein the different hardware-software configuration comprises at least one of a different type of central processing unit, a different clock speed, a different amount of random access memory (RAM), a different operating system, a different operating system version, a different set of concurrently loaded software, a different set of concurrently running software, and a different set of concurrently running processes;

iteratively receive test results from the remote computing devices;

record whether the test results include no bugs in a fresh corpus segment, an original bug in a fresh corpus segment, no bugs in a corpus segment sent to a different-configuration remote computing device to try to reproduce a bug, a reproduction of a bug in a corpus segment sent to a different-configuration remote computing device, or a reproduction of a bug sent back to the original remote computing device on which it occurred after failure to reproduce it on a different-configuration remote computing device;

send data segments with reproduced bugs to an investigation queue and run the natural language feature iteratively on increasingly smaller divisions or sequences of divisions of the data segments until the bug is isolated; and

provide information on the isolated bugs to support debugging the natural language feature.

18. A non-transitory computer storage medium comprising executable instructions capable of being read and executed by one or more components of a computing system to implement the computing system of claim 17.