Traffic replay to detect interface changes

• US 10,346,295 B2
• Filed: 04/14/2017
• Issued: 07/09/2019
• Est. Priority Date: 04/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Beyel, III, Harold E., Ciszewski, Piotr Aleksander, Zhanguzin, Daulet, Hamza, Syed Wali, Cheng, Dongyang
• Primary Examiner(s): Nguyen, Phillip H
• Application Number:
  US15/487,727
• Publication Number:
  US 20180300228A1
• Time in Patent Office: 816 Days
• Field of Search: 717126, 717127, 717124
• CPC Class Codes: G06F 11/3438 monitoring of user actions tracking the activity of the user H04L67/535
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3664 Environments for testing or debugging software
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3692 for test results analysis

Abstract

Systems, methods, apparatuses, and software for software testing systems in computing environments are provided herein. In one example, software modules are instantiated within an enveloping framework for replaying or recording of user activity with regards to the software modules. Based at least on the enveloping framework placed in a replay mode of operation, the enveloping framework provides replayed inputs to the software modules based on previously recorded user activity, the replayed inputs each associated with unique identifiers mapping into the previously recorded user activity. The enveloping framework monitors outputs from the software modules and correlates the outputs to the inputs using the unique identifiers. Changes are detected in operation of the software modules based in part on a comparison of previously recorded outputs from the software modules to the outputs that are determined by the software modules when using the replayed inputs.

Claims

1. A computing apparatus comprising:

one or more computer readable storage media;

a processing system comprising a processor operatively coupled with the one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media, that when executed by the processing system, direct the processing system to at least;

instantiate at least classes associated with one or more software modules within an enveloping framework that forms an encapsulation border for interception of inputs to the one or more software modules and outputs from the one or more software modules;

the enveloping framework configured to;

provide the inputs to the one or more software modules based on previously recorded user activity, the inputs associated with request identifiers mapping into the previously recorded user activity;

intercept the outputs comprising calls made by the one or more software modules to external dependencies by preventing the calls from reaching the external dependencies;

present results to the one or more software modules responsive to the calls to the external dependencies, wherein the results are selected based at least on correlations with the request identifiers associated with the inputs; and

detect changes in operation of the one or more software modules based in part on a comparison of previously recorded outputs from the one or more software modules to replayed outputs that are determined by the one or more software modules when using the inputs and the results.

2. The computing apparatus of claim 1, wherein the one or more software modules comprise application programming interface (API) software methods contained in one or more classes, and wherein instantiating at least the one or more classes associated with the one or more software modules within the enveloping framework comprises wrapping the one or more classes in the enveloping framework to selectively record or replay boundary activity for the API software methods comprising the inputs and the outputs.

3. The computing apparatus of claim 1, comprising further program instructions, when executed by the processing system, direct the processing system to at least:

based at least on the enveloping framework configured to record the user activity to the one or more software modules;

instantiate at least the classes associated with the one or more software modules within the enveloping framework for recording the user activity associated with the one or more software modules;

monitor user inputs among the user activity to the one or more software modules, and store input data related to the user inputs in association with corresponding request identifiers;

monitor outputs determined by the one or more software modules responsive to the user inputs, and store output data related to the outputs in association with ones of the request identifiers that correspond to the user inputs; and

monitor external activity generated responsive to the user inputs between the one or more software modules and the external dependencies, and store dependency data related to the external activity in association with the ones of the request identifiers that correspond to the user inputs.

4. The computing apparatus of claim 3, comprising further program instructions, when executed by the processing system, direct the processing system to at least:

during recording of the user activity, form mappings between the request identifiers and sets of user activity values comprising the input data, the output data, and the dependency data; and

store the mappings in one or more data structures for replay by the enveloping framework.

5. The computing apparatus of claim 4, comprising further program instructions, when executed by the processing system, direct the processing system to at least:

during replay of the user activity, initiate a series of requests comprising the input data to the one or more software modules, and track the replayed outputs using the request identifiers associated with the input data of the series of requests for comparison with the output data to detect the changes in the operation of the one or more software modules.

6. The computing apparatus of claim 3, wherein the enveloping framework determines whether to replay or to record the user activity with regards to the one or more software modules based at least on an instruction issued to the enveloping framework for instantiation of the one or more software modules.

7. The computing apparatus of claim 1, wherein the comparison of the previously recorded outputs to the replayed outputs comprises correlating, via the request identifiers, the previously recorded outputs to the replayed outputs.

8. The computing apparatus of claim 1, wherein a previous version of at least one of the software modules was employed to determine the previously recorded user activity;

and wherein to detect changes in the operation of the one or more software modules, a comparison of behavior of the one or more software modules is performed using the previously recorded user activity.

9. A method of detecting changes in operation of software elements, comprising:

instantiating at least classes associated with one or more software modules within an enveloping framework that forms an encapsulation border for interception of inputs to the one or more software modules and outputs from to the one or more software modules;

in the enveloping framework, providing the inputs to the one or more software modules based on previously recorded user activity, the inputs associated with request identifiers mapping into the previously recorded user activity;

in the enveloping framework, intercepting the outputs comprising calls made by the one or more software modules to external dependencies by preventing the calls from reaching the external dependencies;

in the enveloping framework, presenting results to the one or more software modules responsive to the calls to the external dependencies, wherein the results are selected based at least on correlations with the request identifiers associated with the inputs; and

detecting changes in operation of the one or more software modules based in part on a comparison of previously recorded outputs from the one or more software modules to replayed outputs that are determined by the one or more software modules when using the inputs and the results.

10. The method of claim 9, wherein the one or more software modules comprise application programming interface (API) software methods contained in one or more classes, and wherein instantiating at least the classes associated with the one or more software modules within the enveloping framework comprises wrapping the one or more classes in the enveloping framework to selectively record or replay boundary activity for the API software methods comprising the inputs and the outputs.

11. The method of claim 9, further comprising:

in the enveloping framework, based at least on the enveloping framework configured to record the user activity to the one or more software modules;

instantiating at least the classes associated with the one or more software modules within the enveloping framework for recording the user activity associated with the one or more software modules;

monitoring user inputs among the user activity to the one or more software modules, and storing input data related to the user inputs in association with corresponding request identifiers;

monitoring the outputs determined by the one or more software modules responsive to the user inputs, and storing output data related to the outputs in association with ones of the request identifiers that correspond to the user inputs; and

monitoring external activity generated responsive to the user inputs between the one or more software modules and the external dependencies, and storing dependency data related to the external activity in association with the ones of the request identifiers that correspond to the user inputs.

12. The method of claim 11, further comprising:

during recording of the user activity, forming mappings between the request identifiers and sets of user activity values comprising the input data, the output data, and the dependency data; and

storing the mappings in one or more data structures for replay by the enveloping framework.

13. The method of claim 12, further comprising:

during replay of the user activity, initiating a series of requests comprising the input data to the one or more software modules, and tracking the replayed outputs using the request identifiers associated with the input data of the series of requests for comparison with the output data to detect the changes in the operation of the one or more software modules.

14. The method of claim 11, wherein the enveloping framework determines whether to replay or to record the user activity with regards to the one or more software modules based at least on an instruction issued to the enveloping framework for instantiation of the one or more software modules.

15. The method of claim 9, wherein the comparison of the previously recorded outputs to the replayed outputs comprises correlating, via the request identifiers, the previously recorded outputs to the replayed outputs.

16. The method of claim 9, wherein a previous version of at least one of the software modules was employed to determine the previously recorded user activity;

and wherein to detect changes in the operation of the one or more software modules, a comparison of behavior of the one or more software modules is performed using the previously recorded user activity.

17. A computing apparatus comprising:

one or more computer readable storage media;

a processing system comprising a processor operatively coupled with the one or more computer readable storage media; and

program instructions stored on the one or more computer readable storage media, that when executed by the processing system, direct the processing system to at least;

instantiate at least classes associated with one or more software modules within an enveloping framework that forms a boundary for interception of inputs to the one or more software modules and interception of outputs from the one or more software modules based at least on an instruction issued to the enveloping framework for instantiation of the one or more software modules;

based at least on the enveloping framework placed in a replay mode of operation;

provide the inputs comprising replayed activity to the one or more software modules based on previously recorded user activity, the inputs each associated with unique identifiers mapping into the previously recorded user activity;

intercept the outputs by preventing calls made by the one or more software modules from reaching external services; and

monitor the outputs from the one or more software modules and correlate the outputs to the inputs using the unique identifiers; and

detect changes in operation of the one or more software modules based in part on a comparison of previously recorded outputs from the one or more software modules to the outputs that are determined by the one or more software modules when using the inputs.

18. The computing apparatus of claim 17, comprising further program instructions, when executed by the processing system, direct the processing system to at least:

based at least on the enveloping framework placed in a record mode of operation;

instantiate at least the classes associated with the one or more software modules within the enveloping framework for recording the user activity associated with the one or more software modules;

monitor user inputs among the user activity to the one or more software modules, and store input data related to the user inputs in association with corresponding unique identifiers;

monitor outputs determined by the one or more software modules responsive to the user inputs, and store output data related to the outputs in association with ones of the unique identifiers that correspond to the user inputs.

19. The computing apparatus of claim 17, comprising further program instructions, when executed by the processing system, direct the processing system to at least:

based at least on the enveloping framework placed in the replay mode of operation, intercept calls made by the one or more software modules to external services by withholding the calls to the external services and emulating behavior of the external services by at least presenting results to the one or more software modules responsive to the calls to the external services, wherein the results are selected based at least on correlations with the unique identifiers associated with the replayed inputs.

20. The computing apparatus of claim 17,

wherein the enveloping framework determines whether to replay or to record the user activity with regards to the one or more software modules based at least on an instruction issued to the enveloping framework for instantiation of the one or more software modules;

wherein the comparison of the previously recorded outputs to the replayed outputs comprises correlating, via the unique identifiers, the previously recorded outputs to the replayed outputs; and

wherein a previous version of at least one of the software modules was employed to determine the previously recorded user activity; and

wherein to detect changes in the operation of the one or more software modules, a comparison of behavior of the one or more software modules is performed using the previously recorded user activity.