Clinical content analytics engine
First Claim
1. A computer-implemented method of analyzing a clinical decision support (CDS) document and improving content analyzer system accuracy, the method comprising:
- improving content analyzer system accuracy in identifying CDS document deficiencies and consistencies with respect to reference content, the reference content comprising clinical guidelines, by repeatedly training a machine learning module, hosted by a content analyzer system, based on new incoming data,wherein the machine learning module is configured to automatically determine which content features are to be used to determine whether content is to be designated as relevant and matching to the reference content and which content is to be designated as non-relevant and to construct or modify an electronic model accordingly, wherein the features comprise one or more of text length, presence of a medication term, medical intervention language, use of a negation, or context,wherein improving content analyzer system accuracy by training the machine learning module comprises repeatedly;
collecting positive and negative cases from CDS documents,training the electronic model using the collected positive and negative cases from CDS documents,taking as input a text segment extracted from a CDS document and returning a likelihood that the text segment matches a reference checklist item, andwherein the new incoming data indicates whether the likelihood that the text segment matches a reference checklist item is correct or incorrect;
receiving at a computer system, including hardware and comprising an analytics engine, a clinical decision support document from a medical service provider system;
accessing over a network from a remote system, by the computer system, reference content corresponding at least in part to the clinical decision support document, the reference content comprising clinical guidelines;
using, by the computer system, the electronic model to identify and extract medical intervention content from the clinical decision support document;
using feedback with respect to the identification of the medical intervention content to refine the electronic model;
segmenting, by the computer system, at least a portion of the extracted medical intervention content into a first plurality of segments including;
at least a first segment, comprising a first set of text, anda second segment comprising a second set of text,wherein a given segment in the first plurality of segments is evaluated to identify a core concept, wherein identifying the core concept further comprises determining whether a given segment includes a plurality of medications, and determining which of the plurality of medications are part of the core concept and which of the plurality of medications are not part of the core concept, andif the core concept of the given segment comprises at least one a medical intervention,determining whether a negation is associated with the at least one medical intervention;
determining, by the trained machine learning engine, if the first segment corresponds to at least a first item included in the reference content, the first item comprising a third set of text comprising terminology not present in the first and second sets of text;
at least partly in response to determining that the first segment, comprising the first set of text, corresponds to the first item included in the reference content, the first item comprising the third set of text, causing a version of the clinical decision support document to be generated to include a visual indication that the first segment corresponds to the first item included in the reference content;
determining, by the trained machine learning engine, if a second item included in the reference content corresponds to at least one of the first plurality of segments;
at least partly in response to determining that the second item included in the reference content does not correspond to at least one of the first plurality of segments, causing the version of the clinical decision support document to be generated to include a visual indication that the first plurality of segments fails to include at least one segment that corresponds to the second item included in the reference content; and
at least partly in response to determining that the second item included in the reference content does correspond to at least one of the first plurality of segments, causing the version of the clinical decision support document to include a visual indication that the second item corresponds to at least one segment in the first plurality of segments.
1 Assignment
0 Petitions
Accused Products
Abstract
Clinical content analytics engines and associated processes are described. An engine receives a clinical decision support document, accesses corresponding reference content, identifies and extracts medical intervention content from the clinical decision support document, segments extracted medical intervention content into a first plurality of segments including at least a first segment comprising a first set of text, determines if the first segment corresponds to at least a first item included in the reference content, the first item comprising a second set of text comprising terminology different than that found in the first set of text, and in response to determining that the first segment corresponds to the first item included in the reference content, causing a report to include an indication that the first segment corresponds to the first item included in the reference content.
-
Citations
35 Claims
-
1. A computer-implemented method of analyzing a clinical decision support (CDS) document and improving content analyzer system accuracy, the method comprising:
-
improving content analyzer system accuracy in identifying CDS document deficiencies and consistencies with respect to reference content, the reference content comprising clinical guidelines, by repeatedly training a machine learning module, hosted by a content analyzer system, based on new incoming data, wherein the machine learning module is configured to automatically determine which content features are to be used to determine whether content is to be designated as relevant and matching to the reference content and which content is to be designated as non-relevant and to construct or modify an electronic model accordingly, wherein the features comprise one or more of text length, presence of a medication term, medical intervention language, use of a negation, or context, wherein improving content analyzer system accuracy by training the machine learning module comprises repeatedly; collecting positive and negative cases from CDS documents, training the electronic model using the collected positive and negative cases from CDS documents, taking as input a text segment extracted from a CDS document and returning a likelihood that the text segment matches a reference checklist item, and wherein the new incoming data indicates whether the likelihood that the text segment matches a reference checklist item is correct or incorrect; receiving at a computer system, including hardware and comprising an analytics engine, a clinical decision support document from a medical service provider system; accessing over a network from a remote system, by the computer system, reference content corresponding at least in part to the clinical decision support document, the reference content comprising clinical guidelines; using, by the computer system, the electronic model to identify and extract medical intervention content from the clinical decision support document; using feedback with respect to the identification of the medical intervention content to refine the electronic model; segmenting, by the computer system, at least a portion of the extracted medical intervention content into a first plurality of segments including; at least a first segment, comprising a first set of text, and a second segment comprising a second set of text, wherein a given segment in the first plurality of segments is evaluated to identify a core concept, wherein identifying the core concept further comprises determining whether a given segment includes a plurality of medications, and determining which of the plurality of medications are part of the core concept and which of the plurality of medications are not part of the core concept, and if the core concept of the given segment comprises at least one a medical intervention, determining whether a negation is associated with the at least one medical intervention; determining, by the trained machine learning engine, if the first segment corresponds to at least a first item included in the reference content, the first item comprising a third set of text comprising terminology not present in the first and second sets of text; at least partly in response to determining that the first segment, comprising the first set of text, corresponds to the first item included in the reference content, the first item comprising the third set of text, causing a version of the clinical decision support document to be generated to include a visual indication that the first segment corresponds to the first item included in the reference content; determining, by the trained machine learning engine, if a second item included in the reference content corresponds to at least one of the first plurality of segments; at least partly in response to determining that the second item included in the reference content does not correspond to at least one of the first plurality of segments, causing the version of the clinical decision support document to be generated to include a visual indication that the first plurality of segments fails to include at least one segment that corresponds to the second item included in the reference content; and at least partly in response to determining that the second item included in the reference content does correspond to at least one of the first plurality of segments, causing the version of the clinical decision support document to include a visual indication that the second item corresponds to at least one segment in the first plurality of segments. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
-
-
17. A computer-implemented method of analyzing a clinical decision support (CDS) document and improving content analyzer system accuracy, the method comprising:
-
improving content analyzer system accuracy in identifying CDS document deficiencies and consistencies with respect to reference content, the reference content comprising clinical guidelines, by repeatedly training a machine learning module, hosted by a content analyzer system, based on new incoming data, wherein the machine learning module is configured to automatically determine which content features are to be used to determine whether content is to be designated as relevant and matching to the reference content and which content is to be designated as non-relevant and to construct or modify an electronic model accordingly, wherein the features comprise one or more of text length, presence of a medication term, medical intervention language, use of a negation, or context, wherein improving content analyzer system accuracy by training the machine learning module comprises repeatedly; collecting positive and negative cases from CDS documents, training the electronic model using the collected positive and negative cases from CDS documents, taking as input a text segment extracted from a CDS document and returning a likelihood that the text segment matches a reference checklist item, and wherein the new incoming data indicates whether the likelihood that the text segment matches a reference checklist item is correct or incorrect; receiving at a computer system, including hardware and comprising an analytics engine, a clinical decision support document and/or a data extract from a medical service provider system; accessing over a network from a remote system, by the computer system, reference content, the reference content comprising clinical guidelines, corresponding at least in part to the clinical decision support document and/or the data extract from the medical service provider system; using, by the computer system, the electronic model to identify and extract medical intervention content from the clinical decision support document and/or the data extract from the medical service provider system; using feedback with respect to the identification of the medical intervention content to refine the electronic model; segmenting, by the computer system, at least a portion of the extracted medical intervention content into a first plurality of segments including at least a first segment, comprising a first set of text, and a second segment comprising a second set of text, wherein a given segment in the first plurality of segments is evaluated to identify a core concept, wherein identifying the core concept further comprises determining whether a given segment includes a plurality of medical interventions, and determining which of the plurality of medical interventions are part of the core concept and which of the plurality of medical interventions are not part of the core concept, and if the core concept of the given segment comprises at least one a medical intervention, determining whether a negation is associated with the medical intervention; determining, by the trained machine learning engine, if the first segment corresponds to at least a first item included in the reference content, the first item comprising a third set of text different than the first and second sets of text; at least partly in response to determining that the first segment, comprising the first set of text, corresponds to the first item included in the reference content, the first item comprising the third set of text, causing a report to be generated to include a visual indication that the first segment corresponds to the first item included in the reference content; determining, by the trained machine learning engine, if a second item included in the reference content corresponds to at least one of the first plurality of segments; at least partly in response to determining that the second item included in the reference content does not correspond to at least one of the first plurality of segments, causing the report to include a visual indication that the first plurality of segments fails to include at least one segment that corresponds to the second item included in the reference content; and at least partly in response to determining that the second item included in the reference content does correspond to at least one of the first plurality of segments, causing the report to include a visual indication that the first plurality of segments includes at least one segment that corresponds to the second item included in the reference content. - View Dependent Claims (18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30)
-
-
31. An analytics system configured to improve content analyzer system accuracy, comprising:
-
at least one processing device comprising hardware; non-transitory media comprising a text extraction module, an entity alignment module, and at least one or a pattern matching engine or a machine learning engine, the when executed by the at least one processing device, are configured to cause the analytics system to perform operations comprising; improving content analyzer system accuracy in identifying CDS document deficiencies and consistencies with respect to reference content, the reference content comprising clinical guidelines, by repeatedly training a machine learning module based on new incoming data, wherein the machine learning module is configured to automatically determine which content features are to be used to determine whether content is to be designated as relevant and matching to the reference content and which content is to be designated as non-relevant and to construct or modify an electronic model accordingly, wherein the features comprise one or more of text length, presence of a medication term, medical intervention language, use of a negation, or context, wherein improving content analyzer system accuracy by training the machine learning module comprises repeatedly; collecting positive and negative cases from CDS documents, training the electronic model using the collected positive and negative cases from CDS documents, taking as input a text segment extracted from a CDS document and returning a likelihood that the text segment matches a reference checklist item, and wherein the new incoming data indicates whether the likelihood that the text segment matches a reference checklist item is correct or incorrect; receiving a clinical decision support document; accessing reference content corresponding at least in part to the clinical decision support document, the reference content comprising clinical guidelines; identifying and extracting, using the electronic model, medical intervention content from the clinical decision support document; using feedback with respect to the identification of the medical intervention content to refine the electronic model; segmenting at least a portion of the extracted medical intervention content into a first plurality of segments including at least a first segment, comprising a first set of text, and a second segment comprising a second set of text, wherein a given segment in the first plurality of segments is evaluated to identify a core concept, wherein identifying the core concept further comprises determining whether a given segment includes a plurality of medical interventions, and determining which of the plurality of medical interventions are part of the core concept and which of the plurality of medical interventions are not part of the core concept, and if the core concept of the given segment comprises at least one a medical intervention, determining whether a negation is associated with the medical intervention; determining, using the trained machine learning engine, if the first segment corresponds to at least a first item included in the reference content, the first item comprising a third set of text different than the first and second sets of text; at least partly in response to determining that the first segment, comprising the first set of text, corresponds to the first item included in the reference content, the first item comprising the third set of text, causing a report be generated to include a visual indication that the first segment corresponds to the first item included in the reference content; determining, using the trained machine learning engine, if a second item included in the reference content corresponds to at least one of the first plurality of segments; at least partly in response to determining that the second item included in the reference content does not correspond to at least one of the first plurality of segments, causing the report to include a visual indication that the first plurality of segments fails to include at least one segment that corresponds to the second item included in the reference content; at least partly in response to determining that the second item included in the reference content does correspond to at least one of the first plurality of segments, causing the report to include a visual indication that the first plurality of segments includes at least one segment that corresponds to the second item included in the reference content. - View Dependent Claims (32, 33, 34, 35)
-
Specification