System for standardization of goal setting in performance appraisal process

• US 10,699,236 B2
• Filed: 10/17/2016
• Issued: 06/30/2020
• Est. Priority Date: 10/17/2015
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Apte, Manoj Madhav, Pawar, Sachin, Palshikar, Girish Keshav, Baskaran, Sriram, Aaeer, Amol Madhukar, Pandita, Deepak
• Primary Examiner(s): Diaz, Susanna M.
• Assistant Examiner(s): Brown, Sara Grace
• Application Number:
  US15/295,637
• Publication Number:
  US 20170109680A1
• Time in Patent Office: 1,352 Days
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G06Q 10/06398 Performance of employee with respect to a job function

Abstract

This disclosure relates generally to performance appraisal management, and more particularly to standardization of goals associated with performance appraisal. In one embodiment, a method for standardization of goals includes identifying labeled and unlabeled goals associated with a role. The goals includes template and manually created goals. Each of the template goals is associated with a class label, and includes corresponding goal description and self-comments. First and second classifiers are trained using goal description and self-comments. Candidate negative goals are identified and excluded from the goals to obtain a set of unlabeled goals. The set of unlabeled goals are classified by the first and second classifier, and a confidence score associated with the classification is determined. The unlabeled goals with high confidence score are added to labeled goals to obtain an updated set of labeled goals. The first and second classifiers are iteratively co-trained using the updated set of labeled goals.

Claims

1. A processor-implemented method for standardization of goals associated with a performance appraisal, the method comprising steps of:

(a) identifying, via one or more hardware processors, a plurality of labeled goals and a plurality of unlabeled goals from a plurality of goals associated with a role, the plurality of goals comprising a plurality of template goals and a plurality of manually created goals, each of the plurality of template goals associated with a class label, and wherein each of the plurality of goals associated with a corresponding goal description and corresponding self-comments;

(b) training a first classifier using the goal description of the plurality of labeled goals associated with the role, wherein training the first classifier obtains a first set of feature vectors such that a root-word of each word in the goal description becomes a feature, via the one or more hardware processors;

(c) training a second classifier using the self-comments of the plurality of labeled goals associated with the role, wherein training the second classifier is based on identifying root-words of all the words used in self-comments, wherein the root-words are derived from all the words of self-comments to form the feature vector of a second set of feature vectors, via the one or more hardware processors;

(d) identifying candidate negative goals from the plurality of goals, via the one or more hardware processors;

(e) excluding the candidate negative goals from the plurality of unlabeled goals to obtain a set of unlabeled goals, via the one or more hardware processors;

(f) classifying the set of unlabeled goals into a plurality of classes by the first classifier and the second classifier, by utilizing a co-training framework for semi-supervised learning based approach and matching the manually created goals with the plurality of template goals, via the one or more hardware processors, wherein the plurality of labeled goals are constructed automatically using the plurality of template goals for classification;

(g) determining a confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes, via the one or more hardware processors;

(h) adding from amongst the set of unlabeled goals, one or more unlabeled goals to the plurality of labeled goals to obtain an updated set of labeled goals, the one or more unlabeled goals associated with a class label for each of the first classifier and the second classifier, the one or more unlabeled goals having the confidence score greater than or equal to a threshold value, via the one or more hardware processors; and

(i) iteratively co-training the first classifier and the second classifier using the updated set of labeled goals for a threshold number of iterations by performing steps of training the first and second classifiers using the updated set of the labeled goals, identifying candidate negative goals, classifying the set of unlabeled goals into the plurality of classes by the first and second classifier, determining the confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes, and updating the set of labeled goals, via the one or more hardware processors.

2. The method of claim 1, wherein obtaining the feature vector of the first set of feature vectors, comprises:

identifying two or more words from the goal description of the labeled goals; and

combining the two or more words obtained from the goal descriptions of the labeled goals.

3. The method of claim 1, wherein identifying the root-words, comprises:

creating a bag-of-words for each goal description of the plurality of goal-descriptions by using the set of self-comments; and

computing Term Frequency-Inverse Document Frequency (TF-IDF) scores corresponding to each word of the bag-of-words using an Information Retrieval (IR) literature.

4. The method of claim 1, wherein identifying the candidate negative goals from the plurality of goals comprises:

associating a first word vector and a second word vector with each goal of the plurality of goals;

determining cosine similarity between the first word vector and the second word vector of the each goal; and

identifying one or more manually created goals from amongst the plurality of manually created goals having highest similarity with a template goal of the plurality of template goals and lower than a pre-defined threshold.

5. The method of claim 4, further comprising:

classifying each of the candidate negative goals using the first classifier to obtain a first class label and the second classifier to obtain a second class label for each of the candidate negative goal;

determining a confidence score associated with the candidate negative goals classified into each of the first class label and the second class label; and

assigning a null class label to the candidate negative goal on determination of the confidence score associated with the classification into the first class label being less than a first threshold score and the confidence score associated with the classification into the second class label being less than a second threshold score.

6. The method of claim 5, further comprising:

clustering a set of manually created goals belonging to the null class label into one or more clusters; and

assigning new class labels to the manually created goals belonging to each of the one or more clusters based on identification of the words in the goal descriptions of the one or more clusters.

7. A system for standardization of goals associated with a performance appraisal, the system comprising

one or more memories;

and

one or more hardware processors, the one or more memories coupled to the one or more hardware processors, wherein the one or more hardware processors are programmed to execute instructions stored in the one or more memories to;

(a) identify a plurality of labeled goals and a plurality of unlabeled goals from a plurality of goals associated with a role, the plurality of goals comprising a plurality of template goals and a plurality of manually created goals, each of the plurality of template goals associated with a class label, and wherein each of the plurality of goals associated with a corresponding goal description and corresponding self-comments;

(b) train a first classifier using the goal description of the plurality of labeled goals associated with the role, wherein training the first classifier obtains a first set of feature vectors such that root-word of each word in the goal description becomes a feature;

(c) train a second classifier using the self-comments of the plurality of labeled goals associated with the role, wherein training the second classifier is based on identifying root-words of all the words used in self-comments, wherein the root-words are derived from all the words of self-comments to form the feature vector of a second set of feature vectors;

(d) identify candidate negative goals from the plurality of goals;

(e) exclude the candidate negative goals from the plurality of unlabeled goals to obtain a set of unlabeled goals;

(f) classify the set of unlabeled goals into a plurality of classes by the first classifier and the second classifier, by utilizing a co-training framework for semi-supervised learning based approach and matching the manually created goals with the plurality of template goals, wherein the plurality of labeled goals are constructed automatically using the plurality of template goals for classification;

(g) determine a confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes;

(h) add from amongst the set of unlabeled goals, one or more unlabeled goals to the plurality of labeled goals to obtain an updated set of labeled goals, associated with a class label for each of the first classifier and the second classifier, the one or more unlabeled goals having the confidence score greater than or equal to a threshold value; and

(i) iteratively co-train the first classifier and the second classifier using the updated set of labeled goals for a threshold number of iterations by performing steps of training the first and second classifiers using the updated set of the labeled goals, identifying candidate negative goals, classifying the set of unlabeled goals into the plurality of classes by the first and second classifier, determining the confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes, and updating the set of labeled goals.

8. The system of claim 7, wherein to obtain the feature vector of the first set of feature vectors, the one or more hardware processors are programmed to execute instructions to:

identify two or more words from the goal description of the labeled goals; and

combine the two or more words obtained from the goal descriptions of the labeled goals.

9. The system of claim 8, wherein to identify the candidate negative goals from the plurality of goals, the one or more hardware processors are programmed to execute instructions to:

associate a first word vector and a second word vector with each goal of the plurality of goals;

determine cosine similarity between the first word vector and the second word vector of the each goal; and

identify one or more manually created goals from amongst the plurality of manually created goals having highest similarity with a template goal of the plurality of template goals and lower than a pre-defined threshold.

10. The system of claim 9, wherein the one or more hardware processors are programmed to execute instructions to:

classify each of the candidate negative goals using the first classifier to obtain a first class label and the second classifier to obtain a second class label for the candidate negative goal;

determine a confidence score associated with the candidate negative goals classified into each of the first class label and the second class label; and

assign a null class label to the candidate negative goal on determination of the confidence score associated with the classification into the first class label being less than a first threshold score and the confidence score associated with the classification into the second class label being less than a second threshold score.

11. The system of claim 10, wherein the one or more hardware processors are programmed to execute instructions to:

cluster a set of manually created goals belonging to the null class label into one or more clusters; and

assign new class labels to the manually created goals belonging to each of the one or more clusters based on identification of the words in the goal descriptions of the one or more clusters.

12. The system of claim 7, wherein to identify the root-words, the one or more hardware processors are programmed to execute instructions to

create a bag-of-words for each goal description of the plurality of goal-descriptions by using the set of self-comments;

and

compute Term Frequency-Inverse Document Frequency (TF-IDF) scores corresponding to each word of the bag-of-words using an Information Retrieval (IR) literature.

13. A non-transitory computer readable medium embodying a program executable in computing device for standardization of goals associated with a performance appraisal, the method comprising:

(a) identifying a plurality of labeled goals and a plurality of unlabeled goals from a plurality of goals associated with a role, the plurality of goals comprising a plurality of template goals and a plurality of manually created goals, each of the plurality of template goals associated with a class label, and wherein each of the plurality of goals associated with a corresponding goal description and corresponding self-comments;

(b) training a first classifier using the goal description of the plurality of labeled goals associated with the role, wherein training the first classifier obtains a first set of feature vectors such that root-word of each word in the goal description becomes a feature;

(c) training a second classifier using the self-comments of the plurality of labeled goals associated with the role, wherein training the second classifier is based on identifying root-words of all the words used in self-comments, wherein the root-words are derived from all the words of self-comments to form the feature vector of a second set of feature vectors;

(d) identifying candidate negative goals from the plurality of goals;

(e) excluding the candidate negative goals from the plurality of unlabeled goals to obtain a set of unlabeled goals;

(f) classifying the set of unlabeled goals into a plurality of classes by the first classifier and the second classifier, by utilizing a co-training framework for semi-supervised learning based approach and matching the manually created goals with the plurality of template goals, wherein the plurality of labeled goals are constructed automatically using the plurality of template goals for classification;

(g) determining a confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes;

(h) adding from amongst the set of unlabeled goals, one or more unlabeled goals to the plurality of labeled goals to obtain an updated set of labeled goals, the one or more unlabeled goals associated with a class label for each of the first classifier and the second classifier, the one or more unlabeled goals having the confidence score greater than or equal to a threshold value; and

(i) iteratively co-training the first classifier and the second classifier using the updated set of labeled goals for a threshold number of iterations by performing steps of training the first and second classifiers using the updated set of the labeled goals, identifying candidate negative goals, classifying the set of unlabeled goals into the plurality of classes by the first and second classifier, determining the confidence score associated with the classification of each of the set of unlabeled goals into the plurality of classes, and updating the set of labeled goals.