SYSTEMS AND METHODS ASSOCIATED WITH SEQUENTIAL MULTIPLE HYPOTHESIS TESTING
First Claim
1. One or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors of a computing device, provide the computing device with a hypothesis testing system to iteratively:
- distribute base digital content to a first user, first alternative digital content to a second user, and second alternative digital content to a third user, the base digital content representing a base case of a multiple hypothesis test, the first alternative digital content representing a first alternative case of a first hypothesis test of the multiple hypothesis test and the second alternative digital content representing a second alternative case of a second hypothesis test of the multiple hypothesis test;
monitor interactions of the first user, second user and third user with the base digital content, first alternative digital content, and second alternative digital content, respectively, to collect feedback for the multiple hypothesis test;
based on the collected feedback, determine the first hypothesis test can be concluded utilizing a sequential rejection procedure designed to control a family-wise error rate or a false discovery rate associated with the multiple hypothesis test and the second hypothesis test has yet to conclude utilizing the sequential rejection procedure; and
generate a user interface that is indicative of the determination to inform a user of the hypothesis testing system of a state of the multiple hypothesis test.
2 Assignments
0 Petitions
Accused Products
Abstract
Embodiments of the present invention are directed at providing a sequential multiple hypothesis testing system. In one embodiment, feedback is collected for hypothesis tests of a multiple hypothesis tests. Based on the collected feedback, a sequential p-value is calculated for each of the hypothesis tests utilizing a sequential statistic procedure that is designed to compare an alternative case with a base case for a respective hypothesis test. A sequential rejection procedure can then be applied to determine whether any of the hypothesis tests have concluded based on the respective p-value. A result of the determination can then be output to apprise a user of a state of the multiple hypothesis test. This process can then be repeated until a maximum sample size is reached, termination criterion is met, or all tests are concluded. Other embodiments may be described and/or claimed.
-
Citations
20 Claims
-
1. One or more computer-readable storage media having instructions stored thereon, which, when executed by one or more processors of a computing device, provide the computing device with a hypothesis testing system to iteratively:
-
distribute base digital content to a first user, first alternative digital content to a second user, and second alternative digital content to a third user, the base digital content representing a base case of a multiple hypothesis test, the first alternative digital content representing a first alternative case of a first hypothesis test of the multiple hypothesis test and the second alternative digital content representing a second alternative case of a second hypothesis test of the multiple hypothesis test; monitor interactions of the first user, second user and third user with the base digital content, first alternative digital content, and second alternative digital content, respectively, to collect feedback for the multiple hypothesis test; based on the collected feedback, determine the first hypothesis test can be concluded utilizing a sequential rejection procedure designed to control a family-wise error rate or a false discovery rate associated with the multiple hypothesis test and the second hypothesis test has yet to conclude utilizing the sequential rejection procedure; and generate a user interface that is indicative of the determination to inform a user of the hypothesis testing system of a state of the multiple hypothesis test. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12)
-
-
13. A computer-implemented method for sequentially evaluating a multiple hypothesis test, the method comprising iteratively:
-
receiving feedback for a first hypothesis test of a plurality of hypothesis tests, each hypothesis test of the plurality of hypothesis tests including a respective base case and a respective alternative case, the respective base case for each of the plurality of hypothesis tests being the same across the plurality of hypothesis tests and the respective alternative case for each of the plurality of hypothesis tests being unique across the plurality of hypothesis tests; based on the received feedback, calculating a current sequential statistic value for the first hypothesis test utilizing a predefined sequential statistic procedure associated with the first hypothesis test, the predefined sequential statistic procedure designed to compare the respective base case with the respective alternative case of the first hypothesis test; determining whether the current sequential statistic value is larger than previously calculated sequential statistic values for the first hypothesis test; in response to determining the current sequential statistic value is larger than the previously calculated sequential statistic values, updating a sequential p-value utilizing the current sequential statistic value, the sequential p-value having an inverse relationship with the current sequential statistic; and determining whether the first hypothesis test has concluded, utilizing a sequential rejection procedure in conjunction with the updated sequential p-value. - View Dependent Claims (14, 15, 16, 17)
-
-
18. A computing system to sequentially evaluate a multiple hypothesis test, the computing system comprising:
-
one or more processors; and one or more computer readable storage media, coupled with the one or more processors, having instructions stored thereon, which, when executed by the one or more processors cause the computing system to iteratively; distribute to user computing devices, in response to requests from the user computing devices, digital content for each hypothesis test of an active set of hypothesis tests of the multiple hypothesis test; collect feedback for each hypothesis test of the active set of hypothesis tests, wherein the feedback is indicative of user interactions with the distributed digital content; based on the feedback, calculate a sequential p-value for each hypothesis test, of the active set of hypothesis tests; automatically determine a subset of hypothesis tests, of the set of active hypothesis tests, that have concluded utilizing a sequential rejection procedure in conjunction with the respective sequential p-value for each hypothesis test of the subset of hypothesis tests; remove the subset of hypothesis tests from the active set of hypothesis tests to terminate the collection of feedback for the subset of hypothesis tests to conserve resources of the computing system; and output a notification to a user of the computing system to indicate that the subset of hypothesis tests have concluded. - View Dependent Claims (19, 20)
-
Specification