E-learning utilizing remote proctoring and analytical metrics captured during training and testing

• US 9,875,348 B2
• Filed: 07/21/2014
• Issued: 01/23/2018
• Est. Priority Date: 07/21/2014
• Status: Active Grant

• Current Assignee: Green Grade Solutions Limited
• Original Assignee: Green Grade Solutions Limited
• Inventors: Sadeh, Sharon
• Primary Examiner(s): Yip, Jack
• Application Number:
  US14/336,521
• Publication Number:
  US 20160019379A1
• Time in Patent Office: 1,282 Days
• Field of Search: 434350, 434362, 340 583, 382118
• CPC Class Codes: G06F 21/316 by observing the pattern of computer usage, e.g. typical user behaviour
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/50 Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 50/20 Education
• CPC Class Codes: G09B 5/00 Electrically-operated educational appliances working with questions and answers G09B7/00; simulators G09B9/00; advertising or displaying in general G09F
• CPC Class Codes: G09B 7/00 Electrically-operated teaching apparatus or devices working with questions and answers mechanically operated G09B3/00; computing arrangements G06F
• CPC Class Codes: G09B 7/02 of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student
• CPC Class Codes: H04L 63/0861 using biometrical features, e.g. fingerprint, retina-scan cryptographic mechanisms or cryptographic arrangements for entity authentication using biological data H04L9/3231

Abstract

A device configured to utilize training techniques and to gather information while training and administering certification examinations. Certification examinations may require security to ensure users taking the certification examination are actually the user designated on any issued certificate. When certification examinations are administered remotely from a proctor, different techniques to identify the user and attempt to guarantee that the user is not obtaining outside aid during the examination may be performed. Tests may be administered off-line while not connected to any network and locally cached data may be collected on the testing device for later transmittal to a back-end server for further analysis. In addition to remote proctoring/invigilating capabilities, the disclosed techniques and device may collect metrics that have a number of uses. For example, skill level of different geographical regions may be calculated to assist companies in determining a location for a new factory (e.g., assembly plant).

Claims

1. A device, comprising:

a processor;

a memory communicatively coupled to the processor;

one or more video capture devices communicatively coupled to the processor; and

one or more sensors, wherein the memory stores instructions, that when executed by the processor cause the processor to;

verify that a user is authenticated to the device, using only local information stored on the device;

provide an examination or training session to the user using only local information stored on the device, the examination or training session each comprising a plurality of questions selected from one or more question banks;

capture and locally store a plurality of metrics during the examination or training session, the plurality of metrics captured periodically and intermittently at a frequency unknown to the user, wherein the plurality of metrics comprise at least two of security metrics, performance metrics, and analytical metrics;

determine an expectation of performance of the user based on previously stored performance metrics, the previously stored performance metrics associated with the user and collected during one or more previous training or mock examination sessions of the user;

monitor actual performance of the user during a formal examination;

compare the expectation of performance with the actual performance; and

increase capture of security metrics for the formal examination or for another formal examination based on the actual performance exceeding the expectation of performance by a pre-determined threshold,

wherein the security metrics comprise information, obtained from the one or more video capture devices or the one or more sensors, related to determining an identity of a person interacting with the device during the examination or training session,

wherein the performance metrics comprise information related to the user'"'"'s performance in answering the questions of the examination or training session, and

wherein the analytic metrics comprise information, other than the security metrics, pertaining to a defined group of users, the information having commercial value or being useful for improving or evaluating training, the defined group of users belonging to a given category or entity, the category being a demographic, geographic, industry, market, or business category, and the entity being a manufacturing, production, supplier or marketing entity.

2. The device of claim 1, wherein the instructions further comprise instructions, that when executed by the processor cause the processor to:

disable access to one or more applications on the device; and

disable access to one or more communication interfaces, wherein both the one or more applications and one or more communication interfaces are disabled prior to initiating the examination or training session.

3. The device of claim 1, wherein the instructions further comprise instructions, that when executed by the processor cause the processor to:

monitor performance of the user during one or more training sessions; and

prompt the user to initiate an examination session based, at least in part, on the monitored performance.

4. The device of claim 1, wherein the one or more sensors comprise a global positioning system (GPS) and the analytical metrics comprise location of the device.

5. The device of claim 1, wherein after completion of a formal examination session the device is connected to a network and information regarding the plurality of metrics captured and locally stored during the one or more training sessions and the formal examination session is uploaded to a server for further analysis.

6. The device of claim 1, wherein the examination or training session is a mock or formal examination session and wherein the instructions further comprise instructions, that when executed by the processor cause the processor to:

capture and locally store the security metrics during the mock or the formal examination session.

7. The device of claim 6, wherein the security metrics comprise:

image capture from a front facing camera of the device;

image capture from a rear facing camera of the device; and

audio recording from a microphone.

8. The device of claim 1, wherein verification of the user comprises:

capturing a picture or video of the user by using the one or more video capture devices;

performing face recognition on the captured picture or video;

storing the captured picture or video and results of face recognition locally on the device as stored authentication information; and

associating the stored authentication information with a training or examination session and the authenticated user.

9. The device of claim 1, wherein verification of the user comprises:

capturing a voice sample of the user by using a microphone;

performing voice recognition on the captured voice sample;

storing the captured voice sample and results of voice recognition locally on the device as stored authentication information; and

associating the stored information with a training or examination session and the authenticated user.

10. The device of claim 1, wherein the examination or training session comprises a formal examination sesion and wherein the plurality of questions selected from one or more question banks comprise questions selected from a question bank only available for formal examination sessions.

11. The device of claim 1, wherein the one or more question banks comprise questions overlapping in subject matter content with no repeated questions in any different question bank.

12. The device of claim 1, wherein the plurality of metrics comprises security metrics and performance metrics.

13. The device of claim 1, wherein the plurality of metrics comprises security metrics and analytical metrics.

14. A method to provide one or more training or examination sessions, the method comprising:

verifying, using a processor, that a user is authenticated to a training device, using only local information stored on the training device;

providing an examination or training session to the user using only local information stored on the training device, the examination or training session each comprising a plurality of questions selected from one or more question banks;

capturing and locally storing a plurality of metrics during the examination or training session, the plurality of metrics captured periodically and intermittently at a frequency unknown to the user, wherein the plurality of metrics comprise at least two of security metrics, performance metrics, and analytical metrics;

determining an expectation of performance of the user based on previously stored performance metrics, the previously stored performance metrics associated with the user and collected during one or more previous training or mock examination sessions of the user;

monitoring actual performance of the user during a formal examination;

comparing the expectation of performance with the actual performance; and

increasing capture of security metrics for the formal examination or for another formal examination based on the actual performance exceeding the expectation of performance by a pre-determined threshold,

wherein the security metrics comprise information, obtained from one or more video capture devices or one or more sensors communicatively coupled to the training device, related to determining an identity of a person interacting with the device during the examination or training session,

wherein the performance metrics comprise information related to the user'"'"'s performance in answering the questions of the examination or training session, and

wherein the analytic metrics comprise information, other than the security metrics, pertaining to a defined group of users, the information having commercial value or being useful for improving or evaluating training, the defined group of users belonging to a given category or entity, the category being a demographic, geographic, industry, market, or business category, and the entity being a manufacturing, production, supplier or marketing entity.

15. The method of claim 14, further comprising:

disabling access to one or more applications on the training device and disabling access to one or more communication interfaces of the training device prior to initiating the examination or training session.

16. The method of claim 14, further comprising:

monitoring performance of the user during one or more training sessions; and

prompting the user to initiate an examination session based, at least in part, on the monitored performance.

17. The method of claim 14, further comprising:

capturing a geographical position of the training device using an integrated global positioning system (GPS) sensor during the training or examination session.

18. The method of claim 14, further comprising:

connecting the training device to a network after completion of a formal examination session; and

uploading locally stored information regarding the plurality of metrics captured during one or more training sessions and the formal examination session to a server for further analysis.

19. The method of claim 14, further comprising:

capturing security metrics periodically and intermittently throughout a mock or formal examination session.

20. The method of claim 19, wherein the security metrics comprise:

image capture from a front facing camera of the training device;

image capture from a rear facing camera of the training device; and

audio recording from a microphone configured to provide audio information to the training device.

21. A non-transitory computer readable medium comprising instructions stored thereon that when executed by a processor of a training device cause the processor to:

verify that a user is authenticated to the training device using only local information stored on the training device;

provide an examination or training session to the user using only local information stored on the training device, the examination or training session each comprising a plurality of questions selected from one or more question banks;

capture and locally store a plurality of metrics during the examination or training session, the plurality of metrics captured periodically and intermittently at a frequency unknown to the user, wherein the plurality of metrics comprise at least two of security metrics, performance metrics, and analytical metrics;

determine an expectation of performance of the user based on previously stored performance metrics, the previously stored performance metrics associated with the user and collected during one or more previous training or mock examination sessions of the user;

monitor actual performance of the user during a formal examination;

compare the expectation of performance with the actual performance; and

increase capture of security metrics for the formal examination or for another formal examination based on the actual performance exceeding the expectation of performance by a pre-determined threshold,

wherein the security metrics comprise information, obtained from the one or more video capture devices or the one or more sensors, related to determining an identity of a person interacting with the device during the examination or training session,

wherein the performance metrics comprise information related to the user'"'"'s performance in answering the questions of the examination or training session, and

wherein the analytic metrics comprise information, other than the security metrics, pertaining to a defined group of users, the information having commercial value or being useful for improving or evaluating training, the defined group of users belonging to a given category or entity, the category being a demographic, geographic, industry, market, or business category, and the entity being a manufacturing, production, supplier or marketing entity.

22. A device, comprising:

a processor;

a memory communicatively coupled to the processor;

one or more video capture devices communicatively coupled to the processor; and

one or more sensors, wherein the memory stores instructions, that when executed by the processor cause the processor to;

verify that a user is authenticated to the device, using only local information stored on the device;

provide an examination or training session to the user using only local information stored on the device, the examination or training session each comprising a plurality of questions selected from one or more question banks;

capture and locally store a plurality of metrics during the examination or training session, the plurality of metrics captured periodically and intermittently at a frequency unknown to the user, wherein the plurality of metrics comprise security metrics, performance metrics, and analytical metric;

determine an expectation of performance of the user based on previously stored performance metrics, the previously stored performance metrics associated with the user and collected during one or more previous training or mock examination sessions of the user;

monitor actual performance of the user during a formal examination;

compare the expectation of performance with the actual performance; and

increase capture of security metrics for the formal examination or for another formal examination based on the actual performance exceeding the expectation of performance by a pre-determined threshold,

wherein the security metrics comprise information, obtained from the one or more video capture devices or the one or more sensors, related to determining an identity of a person interacting with the device during the examination or training session,

wherein the performance metrics comprise information related to the user'"'"'s performance in answering the questions of the examination or training session, and

wherein the analytic metrics comprise information, other than the security metrics, pertaining to a defined group of users, the information having commercial value or being useful for improving or evaluating training, the defined group of users belonging to a given category or entity, the category being a demographic, geographic, industry, market, or business category, and the entity being a manufacturing, production, supplier or marketing entity.