Model-driven candidate sorting

• US 9,305,286 B2
• Filed: 03/25/2015
• Issued: 04/05/2016
• Est. Priority Date: 12/09/2013
• Status: Active Grant

• Current Assignee: Hirevue Incorporated
• Original Assignee: Hirevue Incorporated
• Inventors: Larsen, Loren, Taylor, Benjamin
• Primary Examiner(s): VO, HUYEN X
• Application Number:
  US14/668,864
• Publication Number:
  US 20150206103A1
• Time in Patent Office: 377 Days
• Field of Search: 704/235, 704/251, 704/255, 704/257, 704/272, 704/270, 704/277, 704/243, 704/250
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06Q 10/1053 Employment or hiring
• CPC Class Codes: G10L 2015/088 Word spotting
• CPC Class Codes: G10L 25/48 specially adapted for particular use
• CPC Class Codes: G10L 25/54 for retrieval

Abstract

Methods and systems for model-driven candidate sorting for evaluating digital evaluations are described. In one embodiment, a sorting tool selects a data set of digital evaluation data for sorting. The data set includes candidate for evaluation candidates. The sorting tool analyzes the candidate data for the respective evaluation candidate to identify digital evaluation cues and applies the digital evaluation cues to a prediction model to predict an achievement index for the respective evaluation candidate. The list of evaluation candidates is sorted according the predicted achievement indices and the sorted list is presented to the reviewer in a user interface.

Claims

1. A method comprising:

selecting a data set of digital evaluation data for model-driven candidate sorting on a digital evaluation platform comprising a server device, wherein the data set comprises candidate data recorded for a plurality of evaluation candidates;

determining, by the processing device, a predictive model by solving a first linear equation using a system identification algorithm with historical digital evaluation cues as a first input to the first linear equation and historical achievement indices as a second input to the first linear equation;

for the plurality of evaluation candidates,

analyzing, by the server device, the digital candidate data for the respective evaluation candidate to identify a plurality of digital evaluation cues; and

calculating, by the server device, an achievement index using a second linear equation with the plurality of digital evaluation cues as a first input to the second linear equation and the prediction model as a second input to the second linear equation;

sorting a list of the plurality of evaluation candidates based on the predicted achievement indices; and

presenting the sorted list of the plurality of evaluation candidates in a user interface to a reviewer.

2. The method of claim 1, wherein the plurality of digital evaluation cues comprise audio cues, and wherein the analyzing the digital candidate data comprises inspecting an audio signal of the digital candidate data to identify the audio cues.

3. The method of claim 1, wherein the plurality of digital evaluation cues comprises video cues, and wherein the analyzing the digital candidate data comprises inspecting a video signal of the candidates data to identify the video cues.

4. The method of claim 1, wherein the plurality of digital evaluation cues comprise at user interaction cues, and wherein the analyzing the digital candidate data comprises inspecting a log of user interactions to identify the user interaction cues.

5. The method of claim 1, wherein the plurality of digital evaluation cues comprise at least one of audio cues, video cues or user interaction cues, and wherein the analyzing the digital candidate data further comprises identifying additional digital evaluation cues comprising personal digital candidate data.

6. The method of claim 1, further comprising calculating an additional achievement index using the second linear equation with the digital evaluation cues for the respective evaluation candidate as the first input to the second linear equation and the prediction model as the second input to the second linear equation, and wherein the sorting comprises sorting the list of the plurality of evaluation candidates based on at least one of the achievement indices or the additional achievement indices.

7. The method of claim 1, further comprising collecting the digital evaluation data of the data set, wherein the collecting the digital evaluation data comprises collecting timing data, wherein the collecting the timing data comprises at least one of:

determining a time metric representative of the respective evaluation candidate'"'"'s timeliness on starting and completing an evaluation;

or

determining whether the respective evaluation candidate missed a deadline or requested additional time to complete the evaluation.

8. The method of claim 1, further comprising collecting the digital evaluation data of the data set, wherein the collecting the digital evaluation data comprises collecting audio data, wherein the sorting tool comprises an audio cue generator, wherein the collecting the audio data comprises:

identifying, by the audio cue generator, utterances in the audio data of a digital evaluation by the respective candidate, wherein the utterances each comprise a group of one or more words spoken by a candidate in the digital evaluation; and

generating, by the audio cue generator, the audio cues of the digital evaluation based on the identified utterances.

9. The method of claim 1, further comprising collecting the digital evaluation data of the data set, wherein the collecting the digital evaluation data comprises collecting video data, wherein the sorting tool comprises a video cue generator, wherein the collecting the video data comprises at least one of:

determining, by the video cue generator, video metrics in video data of a digital evaluation by the respective candidate; and

generating, by the video cue generator, the video cues of the digital evaluation based on the video metrics, wherein the video metrics comprise at least one of the following;

a heart rate detection;

a candidate facial expression;

eye movement data;

environment data;

or

candidate movement data.

10. The method of claim 1, wherein the presenting the sorted list of the plurality of evaluation candidates in the user interface comprises presenting the sorted list in a view of a digital evaluation platform in response to activation of a user interface element for achievement index sorting.

11. The method of claim 1, further comprising developing the prediction model, wherein developing the prediction model comprises:

gathering the historical digital evaluation cues; and

linking the historical digital evaluation cues to the historical achievement indices.

12. The method of claim 1, further comprising:

obtaining the historical digital evaluation cues;

storing the historical digital evaluation cues in a cue matrix, wherein rows of the cue matrix each represent one of a plurality of past candidates and columns of the cue matrix represent each of the historical digital evaluation cues;

storing historical achievement indices of the plurality of candidates in an achievement score vector;

building the prediction model using the cue matrix and the achievement score vector, wherein the cue matrix represents an input matrix of the system identification algorithm and the achievement score vector represents an output matrix of the system identification algorithm; and

training the prediction model, wherein the system identification algorithm is at least one of a support vector machine, regressions, neural networks, tree-structure classifiers, or symbolic regression using genetic programming.

13. The method of claim 1, wherein the achievement index is at least one of an expected candidate evaluation score, an expected candidate decision, a work performance metric, a metric indicative of a likelihood of termination after a defined period, an academic performance metric, or a future performance metric.

14. A non-transitory computer readable storage medium including instructions that, when executed by a processing device of a computing system, cause the computing system to perform operations comprising:

selecting a data set of digital evaluation data for model-driven candidate sorting on a digital evaluation platform comprising a server device, wherein the data set comprises digital candidate data recorded for a plurality of evaluation candidates;

determining, by the server device, a predictive model by solving a first linear equation using a system identification algorithm with historical digital evaluation cues as a first input to the first linear equation and historical achievement indices as a second input to the first linear equation;

for the plurality of evaluation candidates,

analyzing, by the server device, the digital candidate data for the respective evaluation candidate to identify a plurality of digital evaluation cues; and

calculating, by the server device, an achievement index using a second linear equation with the plurality of digital evaluation cues as a first input to the second linear equation and the prediction model as a second input to the second linear equation;

sorting, by the server device, a list of the plurality of evaluation candidates based on the predicted achievement indices; and

presenting, by the server device, the sorted list of the plurality of evaluation candidates in a user interface to a reviewer.

15. The non-transitory computer readable storage medium of claim 14, wherein the plurality of digital evaluation cues comprise at least one of audio cues, video cues or user interaction cues, and wherein the analyzing the digital candidate data further comprises identifying additional digital evaluation cues comprising personal digital candidate data.

16. The non-transitory computer readable storage medium of claim 14, wherein the digital evaluation data comprises audio data, wherein the operations further comprise:

identifying, by the processing device, utterances in the audio data of a digital evaluation by the respective candidate, wherein the utterances each comprise a group of one or more words spoken by a candidate in the digital evaluation; and

generating, by the processing device, audio cues of the digital evaluation based on the identified utterances.

17. The non-transitory computer readable storage medium of claim 16, wherein the operations further comprise:

obtaining the historical digital evaluation cues;

storing the historical digital evaluation cues in a cue matrix, wherein rows of the cue matrix each represent one of a plurality of past candidates and columns of the cue matrix represent each of the historical digital evaluation cues;

storing historical achievement indices of the plurality of candidates in an achievement score vector;

building the prediction model using the cue matrix and the achievement score vector, wherein the cue matrix represents an input matrix of the system identification algorithm and the achievement score vector represents an output matrix of the system identification algorithm; and

training the prediction model.

18. A computing system comprising:

a data storage device; and

a server device of a digital evaluation platform, the server device being coupled to the data storage device, wherein the server device is operable to execute a sorting tool to perform the following;

select a data set of digital evaluation data for model-driven candidate sorting on the digital evaluation platform, wherein the data set comprises digital candidate data recorded for a plurality of evaluation candidates;

determine a predictive model by solving a first linear equation using a system identification algorithm with historical digital evaluation cues as a first input to the first linear equation and historical achievement indices as a second input to the first linear equation;

for the plurality of evaluation candidates,

analyze the digital candidate data for the respective evaluation candidate to identify a plurality of digital evaluation cues; and

calculate an achievement index using a second linear equation with the plurality of digital evaluation cues as a first input to the second linear equation and the prediction model as a second input to the second linear equation;

sort a list of the plurality of evaluation candidates based on the predicted achievement indices; and

present the sorted list of the plurality of evaluation candidates to a reviewer in a user interface of the digital evaluation platform.

19. The computing system of claim 18, wherein the digital evaluation platform is a web-based application, and wherein the data storage device stores a digital evaluation data repository comprising the data set of digital evaluation data, the prediction model, and the digital evaluation cues.

20. The computing system of claim 18, wherein the digital evaluation data comprises audio data, wherein the sorting tool is further to:

identify utterances in the audio data of a digital evaluation by the respective candidate, wherein the utterances each comprise a group of one or more words spoken by a candidate in the digital evaluation; and

generate audio cues of the digital evaluation based on the identified utterances.