Advanced analytic methods and systems utilizing trust-weighted machine learning models

• US 10,472,096 B2
• Filed: 05/30/2017
• Issued: 11/12/2019
• Est. Priority Date: 05/30/2017
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Sturlaugson, Liessman E., Ethington, James M.
• Primary Examiner(s): Figueroa, Jaime
• Application Number:
  US15/608,693
• Publication Number:
  US 20180346151A1
• Time in Patent Office: 896 Days
• Field of Search: 701 1, 701 329
• CPC Class Codes: B64D 2045/0085 Devices for aircraft health monitoring, e.g. monitoring flutter or vibration
• CPC Class Codes: B64F 5/60 Testing or inspecting aircraft components or systems
• CPC Class Codes: G05B 23/0254 based on a quantitative model, e.g. mathematical relationships between inputs and outputs; functions: observer, Kalman filter, residual calculation, Neural Networks
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning

Abstract

Systems and methods of the present disclosure include determining a performance status of a selected component in an aircraft. An ensemble of related machine learning models is applied to feature data extracted from flight data of the aircraft. Each model produces a positive score and a complementary negative score related to performance of the selected component. The positive scores are weighted based on the false positive rates of the models and the negative scores are weighted based on the false negative rates of the models. The weighted positive scores are combined, e.g., by averaging, and the weighted negative scores are combined, e.g., by averaging. The performance status of the selected component is determined as one of a positive category, a negative category, or an unclassified category based on the values of the combined weighted positive scores and the combined weighted negative scores.

Claims

1. A computer-implemented method of determining a performance status of a selected component in an aircraft, the method comprising:

extracting feature data from flight data collected by sensors during a flight of the aircraft, wherein the feature data relates to performance of one or more components of the aircraft;

applying an ensemble of related machine learning models to produce, for each model, a positive score and a complementary negative score related to performance of the selected component, wherein each model is characterized by a false positive rate and a false negative rate;

weighting the positive score of each model to produce a weighted positive score for each model based on the false positive rate of that model such that the weighted positive score of each model is anti-correlated with the false positive rate of that model;

weighting the negative score of each model to produce a weighted negative score for each model based on the false negative rate of that model such that the weighted negative score of each model is anti-correlated with the false negative rate of that model; and

determining the performance status of the selected component as one of;

a positive category when an average of the weighted positive scores of the models of the ensemble of related machine learning models is greater than a threshold value and an average of the weighted negative scores of the models of the ensemble of related machine learning models is less than or equal to the threshold value;

a negative category when the average of the weighted positive scores is less than or equal to the threshold value and the average of the weighted negative scores is greater than the threshold value;

or

an unclassified category when both of the average of the weighted positive scores and the average of the weighted negative scores are greater than the threshold value or both are less than or equal to the threshold value.

2. The method of claim 1, further comprising selecting the models of the ensemble of related machine learning models from a group of candidate models each having an ROC (receiver operating characteristic), wherein the models of the ensemble of related machine learning models are along a convex ROC hull of the group of candidate models.

3. The method of claim 2, wherein the models of the ensemble of related machine learning models are dispersed along the convex ROC hull of the group of candidate models.

4. The method of claim 1, wherein the ensemble of related machine learning models includes a model that has a positive class bias and a model that has a negative class bias.

5. The method of claim 1, wherein each model includes at least one of a naï

ve Bayes classifier, a support vector machine, a learned decision tree, an ensemble of learned decision trees, or a neural network.

6. The method of claim 1, wherein the weighted positive score for each model is a product of the positive score of the respective model and an inverse exponential with an exponent proportional to the false positive rate of the respective model.

7. The method of claim 1, wherein the weighted negative score for each model is a product of the negative score of the respective model and an inverse exponential with an exponent proportional to the false negative rate of the respective model.

8. The method of claim 1, further comprising determining a likelihood of non-performance of the selected component based on the average of the weighted positive scores and the average of the weighted negative scores.

9. The method of claim 1, further comprising collecting the flight data during a series of flights of the aircraft.

10. The method of claim 1, further comprising signaling by visual, audio, and/or tactile indicator that the selected component is likely to have a non-performance event.

11. A computerized system comprising:

a computer-readable memory;

a processing unit operatively coupled to the computer-readable memory; and

a computer-readable storage media assemblage, wherein the storage media assemblage is operatively coupled to the computer-readable memory and includes instructions, that when executed by the processing unit, cause the system to;

extract feature data from flight data collected by sensors during a flight of the aircraft, wherein the feature data relates to performance of one or more components of the aircraft;

apply an ensemble of related machine learning models to produce, for each model, a positive score and a complementary negative score related to performance of the selected component, wherein each model is characterized by a false positive rate and a false negative rate;

weight the positive score of each model to produce a weighted positive score for each model based on the false positive rate of that model such that the weighted positive score of each model is anti-correlated with the false positive rate of that model;

weight the negative score of each model to produce a weighted negative score for each model based on the false negative rate of that model such that the weighted negative score of each model is anti-correlated with the false negative rate of that model; and

determine a performance status of the selected component as one of;

a positive category when an average of the weighted positive scores of the models of the ensemble of related machine learning models is greater than a threshold value and an average of the weighted negative scores of the models of the ensemble of related machine learning models is less than or equal to the threshold value;

a negative category when the average of the weighted positive scores is less than or equal to the threshold value and the average of the weighted negative scores is greater than the threshold value;

or

an unclassified category when both of the average of the weighted positive scores and the average of the weighted negative scores are greater than the threshold value or both are less than or equal to the threshold value.

12. A system for determining a performance status of a selected component in an aircraft, the system comprising:

a feature extraction module configured to extract feature data from flight data collected by sensors during a flight of the aircraft, wherein the feature data relates to performance of one or more components of the aircraft;

a performance classification module configured to determine a performance status of the selected component of the aircraft based on the feature data, wherein the performance classification module is programmed to;

apply an ensemble of related machine learning models to produce, for each model, a positive score and a complementary negative score related to performance of the selected component, wherein each model has an ROC (receiver operating characteristic) characterized by a false positive rate and a false negative rate;

weight the positive score of each model to produce a weighted positive score for each model based on the false positive rate of that model such that the weighted positive score of each model is anti-correlated with the false positive rate of that model;

weight the negative score of each model to produce a weighted negative score for each model based on the false negative rate of that model such that the weighted negative score of each model is anti-correlated with the false negative rate of that model; and

determine the performance status of the selected component as one of;

a positive category when an average of the weighted positive scores of the models of the ensemble of related machine learning models is greater than a threshold value and an average of the weighted negative scores of the models of the ensemble of related machine learning models is less than or equal to the threshold value;

a negative category when the average of the weighted positive scores is less than or equal to the threshold value and the average of the weighted negative scores is greater than the threshold value;

or

an unclassified category when both of the average of the weighted positive scores and the average of the weighted negative scores are greater than the threshold value or both are less than or equal to the threshold value.

13. The system of claim 12, wherein the models of the ensemble of related machine learning models are selected from a group of candidate models each having an ROC, wherein the models of the ensemble of related machine learning models are along a convex ROC hull of the group of candidate models.

14. The system of claim 12, wherein at least one model of the ensemble of related machine learning models is class biased.

15. The system of claim 12, wherein the positive score of each model indicates a likelihood of non-performance of the selected component and wherein the complementary negative score of each model indicates a likelihood of continued performance of the selected component.

16. The system of claim 12, wherein the weighted positive score is a product of the positive score of the respective model and an inverse exponential with an exponent proportional to the false positive rate of the respective model.

17. The system of claim 12, wherein the weighted negative score is a product of the negative score of the respective model and an inverse exponential with an exponent proportional to the false negative rate of the respective model.

18. The system of claim 12, further comprising a data link configured to communicate with a flight data storage system.

19. The system of claim 12, further comprising a display, wherein the display is configured to indicate the performance status with at least one of a visual, audio, or tactile display.

20. The system of claim 12, further comprising a sensor on board the aircraft, wherein the sensor is configured to collect flight data during the flight of the aircraft.