VERIFYING ANALYTICS RESULTS

• US 20160098442A1
• Filed: 12/15/2015
• Published: 04/07/2016
• Est. Priority Date: 04/30/2014
• Status: Abandoned Application

• Current Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Original Assignee: Pivotal Software, Inc. (Broadcom, Inc.)
• Inventors: Zhu, Lei, Fu, Ning, Li, Xiong, Zhao, Yanhui, Wang, Xiaojian
• Application Number:
  US14/970,303
• Publication Number:
  US 20160098442A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/1487 using N-version programming
• CPC Class Codes: G06F 11/36 Preventing errors by testing or debugging software
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 9/44 Arrangements for executing specific programs

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for validating analytics results. One of the methods includes processing a subset of a dataset and polling an analytics system for a corresponding output subset and comparing the two subsets to validate the analytics system.

Claims

1. A method performed by one or more computers, the method comprising:

receiving a request for validation of an analytics process configured to execute on a distributed computing system comprising a plurality of physical computers;

receiving a raw subset of a dataset;

processing the raw subset of the dataset to generate a first output subset, including executing a test script specifying an expression of the analytics process and supplying the raw subset as an input to the test script;

receiving, from the distributed computing system that is executing the analytics process on the dataset, a second output subset that is a portion of an output of the analytics process executing on the distributed computing system, the second output subset resulting from the distributed computing system processing the raw subset;

comparing the first output subset to the second output subset; and

outputting, prior to the distributed computing system completing execution of the analytics process on the whole dataset, a validation result based on comparing the first output subset to the second output subset.

2. The method of claim 1, wherein the test script is written in the R programming language.

3. The method of claim 1, wherein receiving the second output subset comprises polling the analytics system using an asynchronous callback function implemented using a RESTful API, the method comprising polling the analytics system in parallel with processing the raw subset.

4. The method of claim 1, wherein the first output subset comprises a first plurality of data elements and the second output subset comprises a second plurality of data elements, and wherein comparing the first output subset to the second output subset comprises comparing each data element in the first plurality of elements to a corresponding data element in the second plurality of data elements.

5. The method of claim 4, comprising determining a number of elements in the first plurality of elements that are about equal to a corresponding element in the second plurality of elements, wherein outputting the validation result comprises outputting a positive validation result if the number of elements is above or equal to a threshold and outputting a negative validation result if the number of elements is below the threshold.

6. The method of claim 1, wherein:

receiving the raw subset comprises receiving the raw subset from a customer computer system, wherein the customer computer system provides the dataset to the analytics system and the analytics system provides the output to the customer computer system; and

outputting the validation result comprises outputting the validation result to the customer computer system.

7. The method of claim 1, wherein the test script specifies a process for requesting, from the analytics system, the portion of the output of the analytics system resulting from the analytics system processing the raw subset.

8. The method of claim 7, wherein the process for requesting the portion of the output of the analytics system resulting from the analytics system processing the raw subset comprises an API call to the analytics system for requesting the second output subset by specifying data characterizing the raw subset.

9. A system comprising one or more physical computers configured to perform operations comprising:

receiving a request for validation of an analytics process configured to execute on a distributed computing system comprising a plurality of physical computers;

receiving a raw subset of a dataset;

processing the raw subset of the dataset to generate a first output subset, including executing a test script specifying an expression of the analytics process and supplying the raw subset as an input to the test script;

receiving, from the distributed computing system that is executing the analytics process on the dataset, a second output subset that is a portion of an output of the analytics process executing on the distributed computing system, the second output subset resulting from the distributed computing system processing the raw subset;

comparing the first output subset to the second output subset; and

outputting, prior to the distributed computing system completing execution of the analytics process on the whole dataset, a validation result based on comparing the first output subset to the second output subset.

10. The system of claim 9, wherein the test script is written in the R programming language.

11. The system of claim 9, wherein receiving the second output subset comprises polling the analytics system using an asynchronous callback function implemented using a RESTful API, the method comprising polling the analytics system in parallel with processing the raw subset.

12. The system of claim 9, wherein the first output subset comprises a first plurality of data elements and the second output subset comprises a second plurality of data elements, and wherein comparing the first output subset to the second output subset comprises comparing each data element in the first plurality of elements to a corresponding data element in the second plurality of data elements.

13. The system of claim 12, the operations comprising determining a number of elements in the first plurality of elements that are about equal to a corresponding element in the second plurality of elements, wherein outputting the validation result comprises outputting a positive validation result if the number of elements is above or equal to a threshold and outputting a negative validation result if the number of elements is below the threshold.

14. The system of claim 9, wherein:

receiving the raw subset comprises receiving the raw subset from a customer computer system, wherein the customer computer system provides the dataset to the analytics system and the analytics system provides the output to the customer computer system; and

outputting the validation result comprises outputting the validation result to the customer computer system.

15. The system of claim 9, wherein the test script specifies a process for requesting, from the analytics system, the portion of the output of the analytics system resulting from the analytics system processing the raw subset.

16. The system of claim 15, wherein the process for requesting the portion of the output of the analytics system resulting from the analytics system processing the raw subset comprises an API call to the analytics system for requesting the second output subset by specifying data characterizing the raw subset.

17. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by a distributed computing system of a plurality of physical computers causes the distributed computing system to perform operations comprising:

receiving a request for validation of an analytics process configured to execute on a distributed computing system comprising a plurality of physical computers;

receiving a raw subset of a dataset;

processing the raw subset of the dataset to generate a first output subset, including executing a test script specifying an expression of the analytics process and supplying the raw subset as an input to the test script;

receiving, from the distributed computing system that is executing the analytics process on the dataset, a second output subset that is a portion of an output of the analytics process executing on the distributed computing system, the second output subset resulting from the distributed computing system processing the raw subset;

comparing the first output subset to the second output subset; and

outputting, prior to the distributed computing system completing execution of the analytics process on the whole dataset, a validation result based on comparing the first output subset to the second output subset.

18. The computer storage medium of claim 17, wherein the test script is written in the R programming language.

19. The computer storage medium of claim 17, wherein receiving the second output subset comprises polling the analytics system using an asynchronous callback function implemented using a RESTful API, the method comprising polling the analytics system in parallel with processing the raw subset.

20. The computer storage medium of claim 17, wherein the first output subset comprises a first plurality of data elements and the second output subset comprises a second plurality of data elements, and wherein comparing the first output subset to the second output subset comprises comparing each data element in the first plurality of elements to a corresponding data element in the second plurality of data elements.

21. The computer storage medium of claim 20, the operations comprising determining a number of elements in the first plurality of elements that are about equal to a corresponding element in the second plurality of elements, wherein outputting the validation result comprises outputting a positive validation result if the number of elements is above or equal to a threshold and outputting a negative validation result if the number of elements is below the threshold.

22. The computer storage medium of claim 17, wherein:

receiving the raw subset comprises receiving the raw subset from a customer computer system, wherein the customer computer system provides the dataset to the analytics system and the analytics system provides the output to the customer computer system; and

outputting the validation result comprises outputting the validation result to the customer computer system.

23. The computer storage medium of claim 17, wherein the test script specifies a process for requesting, from the analytics system, the portion of the output of the analytics system resulting from the analytics system processing the raw subset.

24. The computer storage medium of claim 23, wherein the process for requesting the portion of the output of the analytics system resulting from the analytics system processing the raw subset comprises an API call to the analytics system for requesting the second output subset by specifying data characterizing the raw subset.