Systems and methods to improve data clustering using a meta-clustering model

• US 10,671,884 B2
• Filed: 07/03/2019
• Issued: 06/02/2020
• Est. Priority Date: 07/06/2018
• Status: Active Grant

• Current Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Original Assignee: Capital One Services LLC (Capital One Financial Corporation)
• Inventors: Walters, Austin, Goodsitt, Jeremy, Truong, Anh, Farivar, Reza
• Primary Examiner(s): Osinski, Michael S
• Application Number:
  US16/503,428
• Publication Number:
  US 20200012886A1
• Time in Patent Office: 335 Days
• CPC Class Codes: G06F 11/3608 using formal methods, e.g. model checking, abstract interpretation theorem proving G06N5/013
• CPC Class Codes: G06F 11/3628 of optimised code optimisation G06F8/443
• CPC Class Codes: G06F 11/3636 by tracing the execution of the program
• CPC Class Codes: G06F 11/3684 for test design, e.g. generating new test cases
• CPC Class Codes: G06F 11/3688 for test execution, e.g. scheduling of test suites
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/2237 Vectors, bitmaps or matrices
• CPC Class Codes: G06F 16/2264 Multidimensional index structures
• CPC Class Codes: G06F 16/2423 Interactive query statement specification based on a database schema
• CPC Class Codes: G06F 16/24568 Data stream processing; Continuous queries
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/258 Data format conversion from or to a database
• CPC Class Codes: G06F 16/283 Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/288 Entity relationship models
• CPC Class Codes: G06F 16/335 Filtering based on additional data, e.g. user or group profiles filtering in web context G06F16/9535, G06F16/9536
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/90335 Query processing
• CPC Class Codes: G06F 16/9038 : Presentation of query results
• CPC Class Codes: G06F 16/906 : Clustering; Classification
• CPC Class Codes: G06F 16/93 : Document management systems
• CPC Class Codes: G06F 17/15 : Correlation function computation including computation of convolution operations arithmetic circuits for sum of products per se, e.g. multiply-accumulators G06F7/5443; digital filters, e.g. FIR, IIR, adaptive filters H03H17/00
• CPC Class Codes: G06F 17/16 : Matrix or vector computation , e.g. matrix-matrix or matrix-vector multiplication, matrix factorization matrix transposition G06F7/78
• CPC Class Codes: G06F 17/18 : for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2115 : by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/213 : Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/214 : Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2148 : characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2193 : based on specific statistical tests
• CPC Class Codes: G06F 18/22 : Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 : Clustering techniques
• CPC Class Codes: G06F 18/24 : Classification techniques
• CPC Class Codes: G06F 18/2411 : based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06F 18/2415 : based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/285 : Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 18/40 : Software arrangements specially adapted for pattern recognition, e.g. user interfaces or toolboxes therefor
• CPC Class Codes: G06F 21/552 : involving long-term monitoring or reporting
• CPC Class Codes: G06F 21/60 : Protecting data
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6254 : by anonymising data, e.g. decorrelating personal data from the owner's identification
• CPC Class Codes: G06F 30/20 : Design optimisation, verification or simulation optimisation, verification or simulation of circuit designs G06F30/30
• CPC Class Codes: G06F 40/117 : Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/166 : Editing, e.g. inserting or deleting
• CPC Class Codes: G06F 40/20 : Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 8/71 : Version control security arrangements therefor G06F21/57; Configuration management
• CPC Class Codes: G06F 9/54 : Interprogram communication
• CPC Class Codes: G06F 9/541 : via adapters, e.g. between incompatible applications
• CPC Class Codes: G06F 9/547 : Remote procedure calls [RPC]; Web services
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 20/10 : using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 20/20 : Ensemble learning
• CPC Class Codes: G06N 3/04 : Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 : Combinations of networks
• CPC Class Codes: G06N 3/047 : Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/06 : Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
• CPC Class Codes: G06N 3/08 : Learning methods
• CPC Class Codes: G06N 3/084 : Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 3/088 : Non-supervised learning, e.g. competitive learning
• CPC Class Codes: G06N 3/094 : Adversarial learning
• CPC Class Codes: G06N 5/00 : Computing arrangements using knowledge-based models
• CPC Class Codes: G06N 5/01 : Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 5/02 : Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/022 : Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 : Inference or reasoning models
• CPC Class Codes: G06N 7/00 : Computing arrangements based on specific mathematical models
• CPC Class Codes: G06N 7/01 : Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/04 : Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem" market predictions or forecasting for commercial activities G06Q30/0202
• CPC Class Codes: G06T 11/001 : Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20081 : Training; Learning
• CPC Class Codes: G06T 2207/20084 : Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/194 : involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 : using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/254 : involving subtraction of images
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 10/993 : Evaluation of the quality of the acquired pattern
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 30/1985 : Syntactic analysis, e.g. using a grammatical approach syntactic image representation G06V30/18171
• CPC Class Codes: H04L 63/1416 : Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1491 : using deception as countermeasure, e.g. honeypots, honeynets, decoys or entrapment
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04L 67/34 : involving the movement of software or configuration parameters network booting or remote initial program loading [RIPL] G06F9/4416
• CPC Class Codes: H04N 21/23412 : for generating or manipulating the scene composition of objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/8153 : comprising still images, e.g. texture, background image

Abstract

Systems and methods for clustering data are disclosed. For example, a system may include one or more memory units storing instructions and one or more processors configured to execute the instructions to perform operations. The operations may include receiving data from a client device and generating preliminary clustered data based on the received data, using a plurality of embedding network layers. The operations may include generating a data map based on the preliminary clustered data using a meta-clustering model. The operations may include determining a number of clusters based on the data map using the meta-clustering model and generating final clustered data based on the number of clusters using the meta-clustering model. The operations may include and transmitting the final clustered data to the client device.

Claims

1. A system for clustering data, comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform

operations comprising;

receiving data from a client device;

generating, using a plurality of embedding network layers, preliminary clustered-data based on the received data;

generating, using a meta-clustering model, a data map based on the preliminary clustered-data;

determining, using the meta-clustering model, a number of clusters based on the data map;

generating, using the meta-clustering model, final clustered-data based on the number of clusters; and

transmitting the final clustered-data to the client device.

2. The system of claim 1, wherein:

the operations further comprise;

generating an updated embedding-network by training an embedding network layer of the embedding network layers, the training being based on the number of data clusters; and

generating, using the updated embedding-network, updated clustered-data based on the received data; and

wherein generating final clustered-data is based on preliminary clustered-data.

3. The system of claim 1, wherein:

the operations further comprise;

transmitting clustered data samples to the client device, the clustered data samples being based on the clustered data; and

receiving tags associated with the clustered data samples; and

wherein determining the number of clusters is based on the tags.

4. The system of claim 3, wherein:

the operations further comprise generating tags associated with the clustered data samples using the meta-clustering model; and

determining the number of clusters is based on the tags.

5. The system of claim 1, wherein generating preliminary clustered-data comprises:

receiving training data;

generating an embedding network layer of the plurality of embedding network layers;

training the embedding network layer to classify training data; and

training the embedding network layer to cluster training data.

6. The system of claim 1, wherein generating preliminary clustered-data comprises repeating steps until a performance criterion is satisfied, the steps comprising:

adding a trained embedding network layer to the plurality of embedding network layers;

generating clustered data using the added embedding-network;

tagging the clustered data; and

determining whether a performance criterion of the plurality of embedding-networks is satisfied.

7. The system of claim 1, the operations further comprising:

generating the meta-clustering model;

generating, using the meta-clustering model, encoded data based on the clustered data by reducing the dimensionality of the clustered data;

generating a data map based on the encoded data; and

training the meta-clustering model to determine a number of clusters based on the data map and a performance criterion.

8. The system of claim 7, wherein training the meta-clustering model comprises iteratively repeating steps until the performance criterion is satisfied, the steps comprising:

determining, using the meta-clustering model, a number of clusters based on the data map;

generating one or more updated embedding-networks based on the number of clusters;

generating updated clustered-data using the one or more updated embedding-networks;

updating the meta-clustering model;

generating, using the updated meta-clustering model, updated encoded-data based on the updated clustered-data by reducing the dimensionality of the updated clustered-data;

generating an updated data-map based on the updated encoded-data; and

determining whether the performance criterion is satisfied based on the updated data-map.

9. The system of claim 1, wherein the data comprises image data and at least one of the embedding network layers comprises a convolutional neural network.

10. The system of claim 1, wherein the data comprises text data and at least one of the embedding network layers comprises a language representation model.

11. The system of claim 1, wherein the embedding network layers comprise at least one of a Bidirectional Encoder Representations from Transformers (BERT) model or an Embeddings from Language Models (ELMo) representation model.

12. The system of claim 1, wherein the meta-clustering model is a deep learning model.

13. The system of claim 1, the operations further comprising at least one of transmitting the data map to the client device or transmitting the meta-clustering model to the client device.

14. The system of claim 1, wherein the number of clusters is larger than a maximum count of clusters in the preliminary clustered-data.

15. The system of claim 1, wherein generating preliminary clustered-data based on the received data comprises performing a binary classification.

16. The system of claim 1, wherein generating preliminary clustered-data based on the received data comprises performing one-hot encoding.

17. The system of claim 1, wherein:

the operations further comprise generating, using the meta-clustering model, encoded data based on the preliminary clustered-data; and

generating the data map is based on the encoded data.

18. The system of claim 17, wherein generating encoded data comprises performing a method of principal component analysis.

19. A method for clustering data, comprising:

receiving data from a client device;

generating, using a plurality of embedding network layers, preliminary clustered-data based on the received data;

generating, using a meta-clustering model, a data map based on the preliminary clustered-data;

determining, using the meta-clustering model, a number of clusters based on the data map;

generating, using the meta-clustering model, final clustered-data based on the number of clusters; and

transmitting the final clustered-data to the client device.

20. A system for clustering data comprising:

one or more memory units storing instructions; and

one or more processors configured to execute the instructions to perform operations comprising;

receiving data from a client device;

generating preliminary clustered-data using a plurality of embedding network layers by repeating steps until a performance criterion is satisfied, the steps comprising;

adding a trained embedding network layer;

generating clustered data using the added embedding network layer;

tagging the clustered data; and

determining whether a performance criterion of the plurality of embedding network layers is satisfied;

generating a meta-clustering model comprising a neural network model;

generating, using the meta-clustering model, encoded data based on the clustered data by reducing the dimensionality of the clustered data;

generating a data map based on the encoded data;

training the meta-clustering model to determine a number of clusters based on the data map and a performance criterion;

transmitting clustered data samples to the client device, the clustered data samples being based on the clustered data; and

receiving tags associated with the clustered data samples;

determining, using the meta-clustering model, a number of clusters based on the data map and the tags;

generating, using the meta-clustering model, final clustered-data based on the number of clusters; and

transmitting the final clustered-data to the client device.