EVALUATION SYSTEM, EVALUATION METHOD, AND EVALUATION PROGRAM

• US 20190311222A1
• Filed: 10/18/2017
• Published: 10/10/2019
• Est. Priority Date: 10/28/2016
• Status: Active Application

• Current Assignee: NEC Corporation
• Original Assignee: NEC Corporation
• Inventors: ITO, Shinji, YABE, Akihiro, FUJIMAKI, Ryohei
• Application Number:
  US16/345,496
• Publication Number:
  US 20190311222A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06N 99/00 Subject matter not provided for in other groups of this subclass
• CPC Class Codes: G06Q 30/0202 Market predictions or forecasting for commercial activities
• CPC Class Codes: G06Q 30/0206 Price or cost determination based on market factors

Abstract

An evaluation system 80 includes an evaluation unit 81 for evaluating, when there is a prediction model estimated using data generated from the true model, the optimal solution calculated from the prediction model in consideration of bias generated between evaluation based on the prediction model and evaluation based on the true model.

Claims

1. An evaluation system comprising:

a hardware including a processor; and

an evaluation unit, implemented by the processor, which evaluates, when there is a prediction model estimated using data generated from a true model, an optimal solution calculated from the prediction model in consideration of bias generated between evaluation based on the prediction model and evaluation based on the true model.

2. The evaluation system according to claim 1,

wherein the evaluation unit selects, by a bootstrap method, from the data used for generation of the prediction model, estimates a first model using the selected data, and calculates, as bias of evaluation values between the true model and the prediction model, an evaluation difference which is a difference between first evaluation obtained by evaluating an optimal solution, calculated from the first model, by the first model and second evaluation obtained by evaluating the optimal solution by the prediction model.

3. The evaluation system according to claim 2,

wherein the evaluation unit estimates a plurality of first models and calculate, as the bias of the evaluation values between the true model and the prediction model, an average of total sums of evaluation differences which are differences between first evaluation obtained by evaluating optimal solutions, calculated from the first models having been estimated, by the first models and second evaluation obtained by evaluating the optimal solutions by the prediction model.

4. The evaluation system according to claim 3,

wherein the evaluation unit rearranges the evaluation differences calculated each time one of the first models is estimated to determine an order of the evaluation differences and outputs, as an evaluation interval, a minimum value and a maximum value of the evaluation differences included in the order corresponding to a specified range.

5. The evaluation system according to claim 1,

wherein the evaluation unit generates a second model obtained by adding noise to the prediction model and performs evaluation based on the true model from a linear sum of maximum values of evaluation values calculated from the second model and maximum values of evaluation values calculated from the prediction model.

6. The evaluation system according to claim 1,

wherein the evaluation unit estimates a third model using at least a part of the data used for generation of the prediction model and performs evaluation based on the true model from a linear sum of maximum values of evaluation values calculated from the prediction model and maximum values of evaluation values calculated from the third model.

7. The evaluation system according to claim 6,

wherein the evaluation unit estimates a plurality of third models and performs evaluation based on the true model from a linear sum of the maximum values of the evaluation values calculated from the prediction model and an average of total sums of maximum values of evaluation values calculated from the third models.

8. An evaluation method comprising:

evaluating, when there is a prediction model estimated using data generated from a true model, an optimal solution calculated from the prediction model in consideration of bias generated between evaluation based on the prediction model and evaluation based on the true model.

9. The evaluation method according to claim 8, comprising

selecting, by a bootstrap method, from the data used for generation of the prediction model,

estimating a first model using the selected data, and

calculating, as bias of evaluation values between the true model and the prediction model, an evaluation difference which is a difference between first evaluation obtained by evaluating an optimal solution, calculated from the first model, by the first model and second evaluation obtained by evaluating the optimal solution by the prediction model.

10. A non-transitory computer readable information recording medium storing an evaluation program, when executed by a processor, that performs a method for:

evaluating, when there is a prediction model estimated using data generated from a true model, an optimal solution calculated from the prediction model in consideration of bias generated between evaluation based on the prediction model and evaluation based on the true model.

11. The non-transitory computer readable information recording medium according to claim 10, comprising

selecting, by a bootstrap method, from the data used for generation of the prediction model,

estimating a first model using the selected data, and

calculating, as bias of evaluation values between the true model and the prediction model, an evaluation difference which is a difference between first evaluation obtained by evaluating an optimal solution, calculated from the first model, by the first model and second evaluation obtained by evaluating the optimal solution by the prediction model.