Evaluating vendor communications for accuracy and quality

• US 10,395,330 B2
• Filed: 02/17/2016
• Issued: 08/27/2019
• Est. Priority Date: 02/17/2016
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: International Business Machines Corporation
• Inventors: Dorris, Dustin M., Edwards, Alan C., Mudhiganti, Shilpa
• Primary Examiner(s): Holmes, Michael B
• Application Number:
  US15/045,820
• Publication Number:
  US 20170236063A1
• Time in Patent Office: 1,287 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 50/22 Social work or social welfare, e.g. community support activities or counselling services
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof
• CPC Class Codes: G16H 20/00 ICT specially adapted for therapies or health-improving plans, e.g. for handling prescriptions, for steering therapy or for monitoring patient compliance
• CPC Class Codes: G16H 20/10 relating to drugs or medications, e.g. for ensuring correct administration to patients
• CPC Class Codes: G16H 20/30 relating to physical therapies or activities, e.g. physiotherapy, acupressure or exercising
• CPC Class Codes: G16H 20/60 relating to nutrition control, e.g. diets

Abstract

Mechanisms are provided for monitoring quality and correctness of content in communications handled by a vendor. The mechanisms sample a set of communications handled by the vendor to generate a sample set of communications and extract content from the sample set of communications. The mechanisms compare the extracted content with expected content of communications handled by the vendor and analyze the extracted content and the expected content to thereby identify differences between the extracted content and the expected content based on results of the analysis. In addition, the mechanisms determine a level of significance of the differences and generate a notification of whether or not to modify the communications, or an operation of the vendor, based on the determined level of significance of differences.

Claims

1. A method, in a data processing system comprising a processor and a memory, the memory comprising instructions executed by the processor to cause the processor to implement an accuracy and quality evaluation engine for monitoring quality and correctness of content in communications handled by a vendor, wherein the accuracy and quality evaluation engine operates to:

sampling, by sampling logic within the accuracy and quality evaluation engine, a set of communications handled by the vendor to generate a sample set of communications;

extracting, by a quality characteristics extractor within the accuracy and quality evaluation engine, content from the sample set of communications;

comparing, by accuracy evaluator within the accuracy and quality evaluation engine, the extracted content with expected content of communications handled by the vendor;

analyzing, by quality evaluator within the accuracy and quality evaluation engine, the extracted content and the expected content thereby, identifying differences between the extracted content and the expected content based on results of the analysis;

determining, by the quality evaluator, a level of significance of the differences;

extracting, by the quality characteristics extractor, one or more quality characteristics of the sample set of communications, wherein the one or more quality characteristics of the sample set of communications comprises at least one of a speed of transmission of the communications, dropped data packets associated with the communications, audio output quality features of the communications, a number of confirmations of receipt received in response to the communications, or a number of responses received to the communications;

determining, by the quality evaluator, a level of quality of the sample set of communications based on the extracted one or more quality characteristics; and

modifying, by the accuracy and quality evaluation engine, the communications or an operation of the vendor, based on the determined level of significance of differences and the level of quality.

2. The method of claim 1, wherein deters determining a level of significance of the differences comprises:

identifying, by the accuracy evaluator, discrepancies between at least one of terms, phrases, tokens of text, or portions of graphics in communications of the sample set of communications in comparison to required terms, phrases, tokens of text, or portions of graphics specified in the expected content;

for each communication in the sample set of communications, generating, by the accuracy evaluator, an accuracy measure value based on a total number of discrepancies; and

determining, by the accuracy evaluator, a level of significance of the differences based on the accuracy measure values for the communications in the sample set of communications.

3. The method of claim 2, wherein generating, for each communication in the sample set of communications, an accuracy measure value based on the total number of discrepancies comprises weighting discrepancies in accordance with a type of a portion of content where the discrepancies are found such that a different weight is applied to discrepancies in at least two different types of portions of content.

4. The method of claim 1, further comprising:

analyzing, by a communication log analysis engine associated with the accuracy and quality evaluation engine, patient electronic medical records of a plurality of patients to identify, for each patient in the plurality of patients, whether or not the patient is non-compliant with an associated patient care plan;

determining, by a workflow selection engine associated with the accuracy and quality evaluation engine, a mode of communication to use to communicate with patients determined to be non-compliant with their associated patient care plan; and

sending, by a vendor system associated with the vendor, the set of communications to patients in the plurality of patients based on the determined mode of communication.

5. The method of claim 4, wherein the data processing system is a patient care plan creation and monitoring (PCPCM) system provided by a first provider, the vendor system is a sub-system associated with the PCPCM system, and the vendor is a part of an organization associated with the first provider.

6. The method of claim 4, wherein the data processing system is a patient care plan creation and monitoring (PCPCM) system provided by a first provider, the vendor is a third party vendor different from the first provider, and the vendor system is a computing system specifically configured to conduct communications with patients using the determined mode of communication.

7. The method of claim 1, wherein the communications are at least one of textual communications, graphical communications, or audible communications.

8. The method of claim 1, wherein the notification indicates the level of significance of the discrepancies and an identification of a corrective action to take to correct the discrepancies.

9. A computer program product comprising a non-transitory computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a recurring payment collection system for to implement an accuracy and quality evaluation engine for monitoring quality and correctness of content in communications handled by a vendor, and further causes the data processing system to:

sample, by sampling logic within the accuracy and quality evaluation engine, a set of communications handled by the vendor to generate a sample set of communications;

extract, by a quality characteristics extractor within the accuracy and quality evaluation engine, content from the sample set of communications;

compare, by accuracy evaluator within the accuracy and quality evaluation engine, the extracted content with expected content of communications handled by the vendor;

analyze, by quality evaluator within the accuracy and quality evaluation engine, the extracted content and the expected content thereby identifying differences between the extracted content and the expected content based on results of the analysis;

determine, by the quality evaluator, a level of significance of the differences;

extract, by the quality characteristics extractor, one or more quality characteristics of the sample set of communications, wherein the one or more quality characteristics of the sample set of communications comprises at least one of a speed of transmission of the communications, dropped data packets associated with the communications, audio output quality features of the communications, a number of confirmations of receipt received in response to the communications, or a number of responses received to the communications;

determine, by the quality evaluator, a level of quality of the sample set of communications based on the extracted one or more quality characteristics; and

modify, by the accuracy and quality evaluation engine, the communications or an operation of the vendor, based on the determined level of significance of differences, and the level of quality.

10. The computer program product of claim 9, the computer readable program further causes the data processing system to determine a level of significance of the differences at least by:

identifying, by the accuracy evaluator, discrepancies between at least one of terms, phrases, tokens of text, or portions of graphics in communications of the sample set of communications in comparison to required terms, phrases, tokens of text, or portions of graphics specified in the expected content;

for each communication in the sample set of communications, generating, by the accuracy evaluator, an accuracy measure value based on a total number of discrepancies; and

determining, by the accuracy evaluator, a level of significance of the differences based on the accuracy measure values for the communications in the sample set of communications.

11. The computer program product of claim 10, wherein the computer readable program further causes the data processing system to generate, for each communication in the sample set of communications, an accuracy measure value based on the total number of discrepancies at least by weighting discrepancies in accordance with a type of a portion of content where the discrepancies are found such that a different weight is applied to discrepancies in at least two different types of portions of content.

12. The computer program product of claim 9, wherein the computer readable program further causes the data processing system to:

analyze, by a communication log analysis engine associated with the accuracy and quality evaluation engine, patient electronic medical records of a plurality of patients to identify, for each patient in the plurality of patients, whether or not the patient is non-compliant with an associated patient care plan;

determine, by a workflow selection engine associated with the accuracy and quality evaluation engine, a mode of communication to use to communicate with patients determined to be non-compliant, with their associated patient care plan; and

send, by a vendor system associated with the vendor, the set of communications to patients in the plurality of patients based on the determined mode of communication.

13. The computer program product of claim 12, wherein the data processing system is a patient care plan creation and monitoring (PCPCM) system provided by a first provider, the vendor system is a sub-system associated with the PCPCM system, and the vendor is a part of an organization associated with the first provider.

14. The computer program product of claim 12, wherein the data processing system is a patient care plan creation and monitoring (PCPCM) system provided by a first provider, the vendor is a third party vendor different from the first provider, and the vendor system is a computing system specifically configured to conduct communications with patients using the determined mode of communication.

15. The computer program product of claim 9, wherein the communications are at least one of textual communications, graphical communications, or audible communications.

16. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to implement a recurring payment collection system for to implement an accuracy and quality evaluation engine for monitoring quality and correctness of content in communications handled by a vendor, and further cause the processor to;

sample, by sampling logic within the accuracy and quality evaluation engine, a set of communications handled by the vendor to generate a sample set of communications;

extract, by a quality characteristics extractor within the accuracy and quality evaluation engine, content from the sample set of communications;

compare, by accuracy evaluator within the accuracy and quality evaluation engine, the extracted content with expected content of communications handled by the vendor;

analyze, by quality evaluator within the accuracy and quality evaluation engine, the extracted content and the expected content thereby identifying differences between the extracted content and the expected content based on results of the analysis;

determine, by the quality evaluator, a level of significance of the differences;

extract, by the quality characteristics extractor, one or more quality characteristics of the sample set of communications, wherein the one or more quality characteristics of the sample set of communications comprises at least one of a speed of transmission of the communications, dropped data packets associated with the communications, audio output quality features of the communications, a number of confirmations of receipt received in response to the communications, or a number of responses received to the communications;

determine, by the quality evaluator, a level of quality of the sample set of communications based on the extracted one or more quality characteristics; and

modify, by the accuracy and quality evaluation engine, the communications or an operation of the vendor, based on the determined level of significance of differences, and the level of quality.