Risk prediction based on automated analysis of documents

• US 10,726,374 B1
• Filed: 02/19/2019
• Issued: 07/28/2020
• Est. Priority Date: 02/19/2019
• Status: Active Grant

• Current Assignee: Icertis, Inc.
• Original Assignee: Icertis, Inc.
• Inventors: Engineer, Sunu, Jain, Amitabh, Darda, Monish Mangalkumar
• Primary Examiner(s): Miller, Alan S
• Assistant Examiner(s): Wang, Jay-Ming
• Application Number:
  US16/279,974
• Publication Number:
  US 20200265355A1
• Time in Patent Office: 525 Days
• Field of Search: 705 711- 742
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/0635 Risk analysis of enterprise or organisation activities
• CPC Class Codes: G06Q 50/188 Electronic negotiation

Abstract

Embodiments are directed to managing documents over a network. A machine learning (ML) engine analyzes a plurality of documents associated with actions that were performed previously. The ML engine determines critical events associated with the performance of the actions based on the plurality documents. The ML engine generates ML models based on the critical events to compute risk values that may be associated with the critical events. In response to a request to compute risk values associated with pending actions, the ML engine determines documents that are associated with the pending actions based on the request. The ML engine determines the critical events associated with pending actions based on the documents. The ML engine employs the ML models to generate the risk values based on the documents and the critical events. The ML engine provides the risk values in response to the request.

Claims

1. A method for managing documents over a network using one or more processors that execute instructions to perform steps, comprising:

instantiating a machine learning (ML) engine to perform actions, including;

analyzing a plurality of documents that are associated with one or more actions that were performed previously;

determining one or more critical events that are associated with the performance of the one or more actions based on the plurality documents, wherein a portion of the one or more critical events are discovered through application of one or more of rules, heuristics, or ML modeling to the plurality of documents or the one or more actions associated with the plurality of documents;

generating one or more ML models based on the one or more critical events and the plurality of documents, wherein the ML models are trained to predict one or more risk values that are associated with the one or more critical events until an accuracy rate of the predicted one or more risk values meets one or more defined thresholds; and

in response to a request to provide one or more risk values that are associated with one or more pending actions, performing further steps, including;

determining one or more documents that are associated with the one or more pending actions based on the request;

determining the one or more critical events that are implicitly and explicitly associated with the one or more pending actions based on the request and the one or more documents, wherein the one or more implicit and explicit critical events are included in a data object that references the one or more documents to improve the performance of computing resources employed to execute the instructions;

employing the one or more ML models to generate the one or more risk values based on the one or more documents, the predicted one or more risk values and the one or more implicit and explicit critical events; and

providing the one or more risk values in response to the request.

2. The method claim 1, wherein the ML engine performs further steps, including:

receiving one or more external inputs of different types of information, including one or more of political information, weather information, commodity supply information, or cultural information; and

modifying the one or more ML models based on the external inputs.

3. The method of claim 1, wherein the ML engine performs further steps, including:

providing one or more heuristics that are configured to validate the one or more predicted risk values; and

modifying the one or more predicted risk values based on the one or more heuristics.

4. The method of claim 1, wherein the ML engine performs further steps, including:

monitoring one or more external input sources for updated external information; and

updating the one or more risk values based on the updated external information.

5. The method of claim 1, wherein the ML engine performs further steps, including:

analyzing another plurality of documents that are associated with one or more other actions that are previously performed; and

modifying the training of the one or more ML models to include training based on the other plurality of documents.

6. The method of claim 1, wherein the ML engine performs further steps, including:

automatically analyzing one or more other pending actions; and

in response to the one or more risk values exceeding a threshold value, automatically providing one or more notifications.

7. The method of claim 1, wherein the ML engine performs further steps, including:

determining a reputation score for each party to the one or more pending actions based on the one or more actions that were performed previously; and

modifying the one or more risk values based on the reputation score for each party.

8. A system for managing documents comprising:

a network computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform steps, including;

instantiating a machine learning (ML) engine to perform actions, including;

analyzing a plurality of documents that are associated with one or more actions that were performed previously;

determining one or more critical events that are associated with the performance of the one or more actions based on the plurality documents, wherein a portion of the one or more critical events are discovered through application of one or more of rules, heuristics, or ML modeling to the plurality of documents or the one or more actions associated with the plurality of documents;

generating one or more ML models based on the one or more critical events and the plurality of documents, wherein the ML models are trained to predict one or more risk values that are associated with the one or more critical events until an accuracy rate of the predicted one or more risk values meets one or more defined thresholds; and

in response to a request to provide one or more risk values that are associated with one or more pending actions, performing further steps, including;

determining one or more documents that are associated with the one or more pending actions based on the request;

determining the one or more critical events that are implicitly and explicitly associated with the one or more pending actions based on the request and the one or more documents, wherein the one or more implicit and explicit critical events are included in a data object that references the one or more documents to improve the performance of computing resources employed to execute the instructions;

employing the one or more ML models to generate the one or more risk values based on the one or more documents, the predicted one or more risk values, and the one or more implicit and explicit critical events; and

providing the one or more risk values in response to the request; and

a client computer, comprising;

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform actions, including;

providing the request to predict the one or more risk values.

9. The system of claim 8, wherein the ML engine performs further steps, including:

receiving one or more external inputs of different types of information, including one or more of political information, weather information, commodity supply information, or cultural information; and

modifying the one or more ML models based on the external inputs.

10. The system of claim 8, wherein the ML engine performs further steps, including:

providing one or more heuristics that are configured to validate the one or more predicted risk values; and

modifying the one or more predicted risk values based on the one or more heuristics.

11. The system of claim 8, wherein the ML engine performs further steps, including:

monitoring one or more external input sources for updated external information; and

updating the one or more risk values based on the updated external information.

12. The system of claim 8, wherein the ML engine performs further steps, including:

analyzing another plurality of documents that are associated with one or more other actions that are previously performed; and

modifying the training of the one or more ML models to include training based on the other plurality of documents.

13. The system of claim 8, wherein the ML engine performs further steps, including:

automatically analyzing one or more other pending actions; and

in response to the one or more risk values exceeding a threshold value, automatically providing one or more notifications.

14. The system of claim 8, wherein the ML engine performs further steps, including:

determining a reputation score for each party to the one or more pending actions based on the one or more actions that were performed previously; and

modifying the one or more risk values based on the reputation score for each party.

15. A processor readable non-transitory storage media that includes instructions for managing document over a network, wherein execution of the instructions by one or more processors on one or more network computers performs steps, comprising:

instantiating a machine learning (ML) engine to perform actions, including;

analyzing a plurality of documents that are associated with one or more actions that were performed previously;

determining one or more critical events that are associated with the performance of the one or more actions based on the plurality documents, wherein a portion of the one or more critical events are discovered through application of one or more of rules, heuristics, or ML modeling to the plurality of documents or the one or more actions associated with the plurality of documents;

generating one or more ML models based on the one or more critical events and the plurality of documents, wherein the ML models are trained to predict one or more risk values that are associated with the one or more critical events until an accuracy rate of the predicted one or more risk values meets one or more defined thresholds; and

in response to a request to provide one or more risk values that are associated with one or more pending actions, performing further steps, including;

determining one or more documents that are associated with the one or more pending actions based on the request;

determining the one or more critical events that are implicitly and explicitly associated with the one or more pending actions based on the request and the one or more documents, wherein the one or more implicit and explicit critical events are included in a data object that references the one or more documents to improve the performance of computing resources employed to execute the instructions;

employing the one or more ML models to generate the one or more risk values based on the one or more documents, the predicted one or more risk values and the one or more implicit and explicit critical events; and

providing the one or more risk values in response to the request.

16. The media of claim 15, wherein the ML engine performs further steps, including:

receiving one or more external inputs of different types of information, including one or more of political information, weather information, commodity supply information, or cultural information; and

modifying the one or more ML models based on the external inputs.

17. The media of claim 15, wherein the ML engine performs further steps, including:

providing one or more heuristics that are configured to validate the one or more predicted risk values; and

modifying the one or more predicted risk values based on the one or more heuristics.

18. The media of claim 15, wherein the ML engine performs further steps, including:

monitoring one or more external input sources for updated external information; and

updating the one or more risk values based on the updated external information.

19. The media of claim 15, wherein the ML engine performs further steps, including:

analyzing another plurality of documents that are associated with one or more other actions that are previously performed; and

modifying the training of the one or more ML models to include training based on the other plurality of documents.

20. The media of claim 15, wherein the ML engine performs further steps, including:

automatically analyzing one or more other pending actions; and

in response to the one or more risk values exceeding a threshold value, automatically providing one or more notifications.

21. The media of claim 15, wherein the ML engine performs further steps, including:

determining a reputation score for each party to the one or more pending actions based on the one or more actions that were performed previously; and

modifying the one or more risk values based on the reputation score for each party.

22. A network computer for managing documents, comprising:

a transceiver that communicates over the network;

a memory that stores at least instructions; and

one or more processors that execute instructions that perform steps, including;

instantiating a machine learning (ML) engine to perform actions, including;

analyzing a plurality of documents that are associated with one or more actions that were performed previously;

determining one or more critical events that are associated with the performance of the one or more actions based on the plurality documents, wherein a portion of the one or more critical events are discovered through application of one or more of rules, heuristics, or ML modeling to the plurality of documents or the one or more actions associated with the plurality of documents;

generating one or more ML models based on the one or more critical events and the plurality of documents, wherein the ML models are trained to predict one or more risk values that are associated with the one or more critical events until an accuracy rate of the predicted one or more risk values meets one or more defined thresholds; and

in response to a request to provide one or more risk values that are associated with one or more pending actions, performing further steps, including;

determining one or more documents that are associated with the one or more pending actions based on the request;

determining the one or more critical events that are implicitly and explicitly associated with the one or more pending actions based on the request and the one or more documents, wherein the one or more implicit and explicit critical events are included in a data object that references the one or more documents to improve the performance of computing resources employed to execute the instructions;

employing the one or more ML models to generate the one or more risk values based on the one or more documents, the predicted one or more risk values and the one or more implicit and explicit critical events; and

providing the one or more risk values in response to the request.

23. The network computer of claim 22, wherein the ML engine performs further steps, including:

receiving one or more external inputs of different types of information, including one or more of political information, weather information, commodity supply information, or cultural information; and

modifying the one or more ML models based on the external inputs.

24. The network computer of claim 22, wherein the ML engine performs further steps, including:

providing one or more heuristics that are configured to validate the one or more predicted risk values; and

modifying the one or more predicted risk values based on the one or more heuristics.

25. The network computer of claim 22, wherein the ML engine performs further steps, including:

monitoring one or more external input sources for updated external information; and

updating the one or more risk values based on the updated external information.

26. The network computer of claim 22, wherein the ML engine performs further steps, including:

analyzing another plurality of documents that are associated with one or more other actions that are previously performed; and

modifying the training of the one or more ML models to include training based on the other plurality of documents.

27. The network computer of claim 22, wherein the ML engine performs further steps, including:

automatically analyzing one or more other pending actions; and

in response to the one or more risk values exceeding a threshold value, automatically providing one or more notifications.

28. The network computer of claim 22, wherein the ML engine performs further steps, including:

determining a reputation score for each party to the one or more pending actions based on the one or more actions that were performed previously; and

modifying the one or more risk values based on the reputation score for each party.