Systems and methods to identify breaking application program interface changes

• US 10,599,550 B2
• Filed: 06/28/2019
• Issued: 03/24/2020
• Est. Priority Date: 07/06/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Walters, Austin, Goodsitt, Jeremy, Pham, Vincent, Key, Kate
• Primary Examiner(s): Jeon, Jae U
• Application Number:
  US16/457,548
• Publication Number:
  US 20200012583A1
• Time in Patent Office: 270 Days
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3628 of optimised code optimisation G06F8/443
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 : Presentation of query results
• CPC Class Codes: G06F 16/906 : Clustering; Classification
• CPC Class Codes: G06F 16/93 : Document management systems
• CPC Class Codes: G06F 17/15 : Correlation function computation including computation of convolution operations arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00
• CPC Class Codes: G06F 17/16 : Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 17/18 : for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 : by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/213 : Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 : Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2148 : characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2193 : based on specific statistical tests
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 : Clustering techniques
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/2411 : based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/2415 : based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/285 : Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 18/40 : Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06F 21/552 : involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/60 : Protecting data
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6254 : by anonymising data, e.g. decorrelating personal data from the owner's identification
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06F 40/117 : Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/166 : Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 : Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 8/71 : Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06F 9/541 : via adapters, e.g. between incompatible applications
• CPC Class Codes: G06F 9/547 : Remote procedure calls [RPC]; Web services
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 20/10 : using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 : Ensemble learning
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/047 : Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/06 : Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 : Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 5/00 : Computing arrangements using knowledge-based models
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 : Inference or reasoning models
• CPC Class Codes: G06N 7/00 : Computing arrangements based on specific mathematical models
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/04 : Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06T 11/001 : Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/254 : involving subtraction of images
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/993 : Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 30/1985 : Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1491 : using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04N 21/23412 : for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/8153 : comprising still images, e.g. texture, background image

Abstract

Systems and methods for managing Application Programming Interfaces (APIs) are disclosed. For example, the system may include one or more memory units storing instructions and one or more processors configured to execute the instructions to perform operations. The operations may include sending a first call to a first node-testing model associated with a first API and receiving a first model output comprising a first model result and a first model-result category. The operations may include identifying a second node-testing model associated with a second API and sending a second call to the second node testing model. The operations may include receiving a second model output comprising a second model result and a second model-result category. The operations may include performing at least one of sending a notification, generating an updated first node-testing model, generating an updated second node-testing model, generating an updated first call, or generating an updated second call.

Claims

1. A system for managing application programming interfaces (APIs), comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising;

training a first node-testing model corresponding to a first API node based on first API call data and a first indicator of a match between first node-testing model output and first API node output;

training a second node-testing model corresponding to a second API node based on second API call data and a second indicator of a match between second node-testing model output and second API node output;

training a translation model to generate a translated input based on a plurality of inputs, the first node-testing model output, and the second node-testing model output;

sending, using a routing module, a call to the first node-testing model;

generating, by the first node-testing model, a first model result and a first model-result category based on the call, the first model-result category comprising an estimate of a likelihood that the first model result matches a result of the first API node;

identifying, using the routing module, the second node-testing model based on the first model result;

generating, by the translation model, a translated model result based on the first node-testing model result;

sending, using the routing module, the translated model result to the second node-testing model based on the identification of the second node-testing model;

generating, by the second node-testing model, a second model result and a second model-result category based on the translated model result, the second model-result category comprising an estimate of a likelihood that the second model result matches a result of the second API node;

determining that at least one of;

the first model result comprises a first warning message,

the first model result comprises a first error message,

the second model result comprises a second warning message, or

the second model result comprises a second error message;

displaying, at an interface, a message based on at least one of the first model-result category or second model-result category; and

displaying, at the interface, at least one of the first warning message, first error message, second warning message, or second error message.

2. The system of claim 1, wherein at least one of the first or second indicator is a percent match between first node-testing model output and first API node output.

3. The system of claim 1, wherein at least one of the first or second indicator is a confidence level.

4. The system of claim 3, wherein:

the call comprises an API identifier; and

sending the call is based on the API identifier.

5. The system of claim 1, the operations further comprising:

receiving user inputs from the interface; and

based on the user inputs, performing at least one of generating an updated first node-testing model or generating an updated second node-testing model.

6. The system of claim 1, the operations further comprising:

training a translation model to generate a translated input based on a plurality of inputs, and first node-testing model output; and

generating, by the translation model, a translated call based on the call,

wherein generating first node-testing model output is based on the translated call.

7. The system of claim 1, wherein at least one of sending the call or sending the first model result is based on a routing table.

8. The system of claim 1, wherein:

the operations further comprise training a routing model to identify pathways in a network of APIs comprising the first API node and the second API node; and

at least one of sending the call or sending the first model result is based on output of the routing model.

9. The system of claim 1, wherein the first node-testing model comprises at least one of a sequence-to-sequence model (seq2seq), a recurrent neural network mode (RNN), a long-short term memory (LSTM) model, or a convolutional neural network (CNN).

10. The system of claim 1, the operations further comprising displaying, at the interface, a plurality of routing pathways between a plurality of node-testing models, the node-testing models comprising the first node-testing model and second node-testing model.

11. The system of claim 1, wherein at least one of training the first node-testing model or training the second node-testing model comprises hyperparameter tuning.

12. The system of claim 1, wherein:

the operations further comprise identifying, using the routing module, the first node-testing model of the node-testing models based on the call; and

sending the call is based on identifying the first node-testing model.

13. The system of claim 1, wherein:

the first model result comprises an API identifier; and

sending the first model result is based on the API identifier.

14. The system of claim 1, the operations further comprising:

training a third node-testing model corresponding to a third API node based on third API call data and a third indicator of a match between third node-testing model output and third API node output;

identifying, by the routing module, the third node-testing model based on the second model result;

sending, using the routing module, the second model result to the third node-testing model based on the identification of the third node-testing model;

generating, by the third node-testing model, a third model result and a third model-result category based on the second model result, the third model-result category comprising an estimate of a likelihood that the third model result matches a result of the third API node.

15. The system of claim 1, wherein:

the first model-result category comprises an unknown result; and

the operations further comprise training the first node-testing model based on the call, the API call data, and the first indicator.

16. The system of claim 1, wherein:

the second model-result category comprises an unknown result; and

the operations further comprise training the second node-testing model based on the call, the API call data, and the second indicator.

17. The system of claim 1, wherein the system comprises a cloud service.

18. A method for managing application programming interfaces (APIs), comprising:

training a first node-testing model corresponding to a first API node based on first API call data and a first indicator of a match between first node-testing model output and first API node output;

training a second node-testing model corresponding to a second API node based on second API call data and a second indicator of a match between second node-testing model output and second API node output;

training a translation model to generate a translated input based on a plurality of inputs, the first node-testing model output, and the second node-testing model output;

sending, using a routing module, a call to the first node-testing model;

generating, by the first node-testing model, a first model result and a first model-result category based on the call, the first model-result category comprising an estimate of a likelihood that the first model result matches a result of the first API node;

identifying, using the routing module, the second node-testing model based on the first model result;

generating, by the translation model, a translated model result based on the first node-testing model result;

sending, using the routing module, the translated model result to the second node-testing model based on the identification;

generating, by the second node-testing model, a second model result and a second model-result category based on the translated model result, the second model-result category comprising an estimate of a likelihood that the second model result matches a result of the second API node;

determining that at least one of;

the first model result comprises a first warning message,

the first model result comprises a first error message,

the second model result comprises a second warning message, or the second model result comprises a second error message;

displaying, at an interface, a message based on at least one of the first model-result category or second model-result category; and

displaying, at the interface, at least one of the first warning message, first error message, second warning message, or second error message.

19. A system for managing application programming interfaces (APIs), comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising;

training a plurality of node-testing models corresponding to a plurality of API nodes based on API call data and indicators of a match between output of the node-testing models and output of the corresponding API nodes;

training a translation model to generate a translated input based on a plurality of inputs, first node-testing model output, and second node-testing model output;

identifying, using a routing module, a first node-testing model of the node-testing models based on a call;

sending, using the routing module, the call to the first node-testing model;

generating, by the first node-testing model, a first model result and a first model-result category based on the call, the first model-result category comprising an estimate of a likelihood that the first model result matches a result of a first API node of the API nodes, the first API node corresponding to the first node-testing model;

identifying, using the routing module, a second node-testing model of the node-testing models based on the first model result;

generating, by the translation model, a translated model result based on the first node-testing model result;

sending, using the routing module, the translated model result to the second node-testing model based on the identification;

generating, by the second node-testing model, a second model result and a second model-result category based on the translated model result, the second model-result category comprising an estimate of a likelihood that the second model result matches a result of a second API node of the API nodes, the second API node corresponding to the second node-testing model;

determining that at least one of;

the first model result comprises a first warning message,

the first model result comprises a first error message,

the second model result comprises a second warning message, or

the second model result comprises a second error message;

displaying, at an interface, a message based on at least one of the first model-result category or second model-result category; and

displaying, at the interface, at least one of the first warning message, first error message, second warning message, or second error message.