Hierarchical service oriented application topology generation for a network

• US 10,693,750 B2
• Filed: 01/29/2019
• Issued: 06/23/2020
• Est. Priority Date: 06/01/2015
• Status: Active Grant

• Current Assignee: Lightbend, Inc.
• Original Assignee: Lightbend, Inc.
• Inventors: Sasturkar, Amit, Surana, Vishal, Velipasaoglu, Omer Emre, Vora, Abhinav A., Ma, Aiyesha Lowe
• Primary Examiner(s): Dinh, Khanh Q
• Application Number:
  US16/261,134
• Publication Number:
  US 20190158369A1
• Time in Patent Office: 511 Days
• Field of Search: 709203, 709220, 709224, 709226, 709228, 709232
• CPC Class Codes: H04L 43/045 for graphical visualisation of monitoring data
• CPC Class Codes: H04L 43/062 related to network traffic
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]
• CPC Class Codes: H04L 9/40 Network security protocols
• CPC Class Codes: H04W 12/08 Access security

Abstract

The technology disclosed relates to understanding traffic patterns in a network with a multitude of processes running on numerous hosts. In particular, it relates to using at least one of rule based classifiers and machine learning based classifiers for clustering processes running on numerous hosts into local services and clustering the local services running on multiple hosts into service clusters, using the service clusters to aggregate communications among the processes running on the hosts and generating a graphic of communication patterns among the service clusters with available drill-down into details of communication links. It also relates to using predetermined command parameters to create service rules and machine learning based classifiers that identify host-specific services. In one implementation, user feedback is used to create new service rules or classifiers and/or modify existing service rules or classifiers so as to improve accuracy of the identification of the host-specific services.

Claims

1. A system for generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the system comprising:

a machine learning-based classifier trained to cluster the hosts into service profiles by;

evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and

based on the evaluation, classifying hosts that run similar processes as having a same service profile; and

a graphic generator that generates a graphic of the topology of the network based on the service profiles produced by the machine learning-based classifier.

2. The system of claim 1, wherein the service profiles cluster hosts that share common functionality.

3. The system of claim 1, wherein the machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.

4. The system of claim 1, wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the machine learning-based classifier.

5. The system of claim 1, wherein the machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.

6. The system of claim 1, wherein the machine learning-based classifier is trained using manually labelled training data.

7. The system of claim 1, wherein the machine learning-based classifier includes a neural network.

8. The system of claim 1, wherein the machine learning-based classifier applies a naï

ve bayes algorithm to command parameters.

9. A method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:

using a trained machine learning-based classifier to cluster the hosts into service profiles by;

evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and

based on the evaluation, classifying hosts that run similar processes as having a same service profile; and

generating a graphic of the topology of the network based on the service profiles produced by the trained machine learning-based classifier.

10. The method of claim 9, wherein the service profiles cluster hosts that share common functionality.

11. The method of claim 9, wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.

12. The method of claim 9, wherein the string vectors are chosen based on their respective term frequency-inverse document frequencies (TF-IDFs) prior to being evaluated by the trained machine learning-based classifier.

13. The method of claim 9, wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.

14. The method of claim 9, wherein the trained machine learning-based classifier is trained using manually labelled training data.

15. The method of claim 9, wherein the trained machine learning-based classifier includes a neural network.

16. The method of claim 9, wherein the trained machine learning-based classifier applies a naï

ve bayes algorithm to command parameters.

17. One or more non-transitory computer readable media having instructions stored thereon for performing a method of generating hierarchical service oriented application topology of a network with a multitude of processes running on numerous hosts, the method including:

using a trained machine learning-based classifier to cluster the hosts into service profiles by;

evaluating command parameters of respective processes running on the hosts by applying logistic regression to string vectors of the command parameters to calculate a probability of classifying a host into a particular service profile, and

based on the evaluation, classifying hosts that run similar processes as having a same service profile; and

generating a graphic of the topology of the network based on the service profiles produced by the trained machine learning-based classifier.

18. The one or more non-transitory computer readable media of claim 17, wherein the trained machine learning-based classifier includes a neural network.

19. The one or more non-transitory computer readable media of claim 17, wherein the trained machine learning-based classifier evaluates string vectors of the command parameters against process-specific rules stored in a rule database.

20. The one or more non-transitory computer readable media of claim 17, wherein the trained machine learning-based classifier applies a logistic regression algorithm that uses cross-validation to classify hosts into service profiles.