Cloud architecture for automated testing

• US 10,635,476 B2
• Filed: 05/31/2017
• Issued: 04/28/2020
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Blum, William, Godefroid, Patrice, Molnar, David
• Primary Examiner(s): Duncan, Marc
• Assistant Examiner(s): Gibson, Jonathan D
• Application Number:
  US15/610,264
• Publication Number:
  US 20180329788A1
• Time in Patent Office: 1,063 Days
• CPC Class Codes: G06F 11/1484 involving virtual machines
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 11/3696 Methods or tools to render software testable
• CPC Class Codes: G06F 2009/45562 Creating, deleting, cloning virtual machine instances
• CPC Class Codes: G06F 2009/45591 Monitoring or debugging support
• CPC Class Codes: G06F 2201/815 Virtual
• CPC Class Codes: G06F 2201/84 Using snapshots, i.e. a logical point-in-time copy of the data
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects

Abstract

Apparatus and methods can be implemented to perform software testing or to perform emulated hardware testing using a cloud architecture that can utilize centralized testing technology and can enable scaling up to test for multiple tenants and scaling up to arbitrary numbers of programs tested for each tenant. A user can configure an initial test virtual machine on a cloud platform for a cloud service over a physical network such as the Internet. Components of the cloud architecture can create a set of clones of the initial test virtual machine and inject tools into each clone for testing. Testing of one or more clones of the set can be conducted in an environment isolated from the physical network and isolated from a backend of the cloud service. Additional apparatus, systems, and methods are disclosed.

Claims

1. A system comprising:

a set of processors;

a storage device comprising instructions, which when executed by the set of processors, cause the system to perform operations to;

receive a virtual machine, a virtual machine image, or a virtual hard disk by a cloud service, the cloud service operating on a processing platform, with the virtual machine, the virtual machine image, or the virtual hard disk having a test payload provided via a connection with an external site over a physical network;

create a set of clones of the virtual machine, the virtual machine image, or the virtual hard disk and inject tools into each clone;

create a reproduction of the received virtual machine, the received virtual machine image, or the received virtual hard disk;

test one or more clones of the set in an environment isolated from the physical network and isolated from a backend of the cloud service such that the test of the one or more clones includes use of the injected tools and is conducted within the isolated environment that is secured without interaction from the physical network and without interaction with the backend during the test once testing has begun, the backend including systems, other than the isolated environment, on the processing platform for the cloud service, the environment isolated from the backend by a security virtual machine that controls data flow from the isolated environment to the backend such that the backend is protected from untrusted features of the clones under test;

in response to a determination of a defect in a tested clone, provide the defect to the reproduction via the security virtual machine;

run the reproduction, in the isolated environment, with respect to the defect and provide results of running the reproduction to the security virtual machine; and

report results of the testing.

2. The system of claim 1, wherein the operation to receive the virtual machine by the cloud service includes operations to:

provide a pristine virtual machine provisioned by the cloud service; and

receive the virtual machine generated from installation and configuration of the test payload directly on the pristine virtual machine on the processing platform of the cloud service via the connection with the external site.

3. The system of claim 1, wherein the operation to test the one or more clones includes operations to provision a test lab for testing the one or more clones.

4. The system of claim 3, wherein the test lab is provisioned as a fuzzing test lab.

5. The system of claim 3, wherein the operations include operations to provision a validation machine along with testing machines of the test lab, and to start the testing of the one or more clones when a validation of the virtual machine or a validation of a clone of the virtual machine completes.

6. The system of claim 3, wherein operation to report results of the testing includes operations to communicate the results between the test lab and the backend of the cloud service through the security virtual machine.

7. The system of claim 3, wherein the system includes a representational state transfer-based communication interface to communicate between the test lab and the backend of the cloud service.

8. The system of claim 1, wherein the operations to report results of the testing includes operations to provide a user with a prioritized list of issues from the testing and an estimate as to the basis for the occurrence of each issue.

9. A method comprising:

receiving a virtual machine, a virtual machine image, or a virtual hard disk by a cloud service, the cloud service operating on a processing platform, with the virtual machine, the virtual machine image, or the virtual hard disk having a test payload provided via a connection with an external site over a physical network;

creating a set of clones of the virtual machine, the virtual machine image, or the virtual hard disk and inject tools into each clone;

creating a reproduction of the received virtual machine, the received virtual machine image, or the received virtual hard disk;

testing one or more clones of the set in an environment isolated from the physical network and isolated from a backend of the cloud service such that the testing of the one or more clones includes using the injected tools and conducting within the isolated environment that is secured without interaction from the physical network and without interaction with the backend during the test once testing has begun, the backend including systems, other than the isolated environment, on the processing platform for the cloud service, the environment isolated from the backend by a security virtual machine that controls data flow from the isolated environment to the backend such that the backend is protected from untrusted features of the clones under test;

providing, in response to a determination of a defect in a tested clone, the defect to the reproduction via the security virtual machine;

running the reproduction, in the isolated environment, with respect to the defect and providing results of running the reproduction to the security virtual machine; and

reporting results of the testing.

10. The method of claim 9, wherein receiving the virtual machine by the cloud service includes:

providing a pristine virtual machine provisioned by the cloud service; and

receiving the virtual machine generated from installation and configuration of the test payload directly on the pristine virtual machine on the processing platform of the cloud service via the connection with the external site.

11. The method of claim 9, wherein testing the one or more clones includes provisioning a test lab for testing the one or more clones.

12. The method of claim 11, wherein the test lab is a fuzzing test lab.

13. The method of claim 11, wherein the method includes provisioning a validation machine along with testing machines of the test lab, and starting the testing of the one or more clones when a validation of the virtual machine or a clone of the virtual machine completes.

14. The method of claim 11, wherein reporting results of the testing includes communicating the results between the test lab and the backend of the cloud service through the security virtual machine.

15. The method of claim 11, wherein the method includes communicating between the test lab and the backend of the cloud service via a representational state transfer-based communication interface.

16. The method of claim 9, wherein reporting results of the testing includes providing a user with a prioritized list of issues from the testing and an estimate as to the basis for the occurrence of each issue.

17. A non-transitory machine-readable medium comprising instructions, which when executed by a set of processors, cause a system to perform operations comprising:

receiving a virtual machine, a virtual machine image, or a virtual hard disk by a cloud service, the cloud service operating on a processing platform, with the virtual machine, the virtual machine image, or the virtual hard disk having a test payload provided via a connection with an external site over a physical network;

creating a set of clones of the virtual machine, the virtual machine image, or the virtual hard disk and inject tools into each clone;

creating a reproduction of the received virtual machine, the received virtual machine image, or the received virtual hard disk;

testing one or more clones of the set in an environment isolated from the physical network and isolated from a backend of the cloud service such that the testing of the one or more clones includes using the injected tools and conducting within the isolated environment that is secured without interaction from the physical network and without interaction with the backend during the test once testing has begun, the backend including systems, other than the isolated environment, on the processing platform for the cloud service, the environment isolated from the backend by a security virtual machine that controls data flow from the isolated environment to the backend such that the backend is protected from untrusted features of the clones under test;

providing, in response to a determination of a defect in a tested clone, the defect to the reproduction via the security virtual machine;

running the reproduction, in the isolated environment, with respect to the defect and providing results of running the reproduction to the security virtual machine; and

reporting results of the testing.

18. The non-transitory machine-readable medium of claim 17, wherein receiving the virtual machine by the cloud service includes:

providing a pristine virtual machine provisioned by the cloud service; and

receiving the virtual machine generated from installation and configuration of the test payload directly on the pristine virtual machine on the processing platform of the cloud service via the connection with the external site.

19. The non-transitory machine-readable medium of claim 17, wherein testing the one or more clones includes provisioning a test lab for testing the one or more clones.

20. The non-transitory machine-readable medium of claim 19, wherein the test lab is a fuzzing test lab.

21. The non-transitory machine-readable medium of claim 19, wherein the operations include provisioning a validation machine along with testing machines of the test lab, and starting the testing of the one or more clones when a validation of the virtual machine or a clone of the virtual machine completes.

22. The non-transitory machine-readable medium of claim 19, wherein reporting results of the testing includes communicating the results between the test lab and the backend of the cloud service through the security virtual machine.

23. The non-transitory machine-readable medium of claim 19, wherein the operations include communicating between the test lab and the backend of the cloud service via a representational state transfer-based communication interface.

24. The non-transitory machine-readable medium of claim 17, wherein reporting results of the testing includes providing a user with a prioritized list of issues from the testing and an estimate as to the basis for the occurrence of each issue.