Detection of unauthorized user assistance of an electronic device based on the detection or tracking of eyes

• US 10,354,057 B2
• Filed: 12/28/2016
• Issued: 07/16/2019
• Est. Priority Date: 12/28/2016
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Motorola Solutions, Inc.
• Inventors: Lim, Bing Qin, Hooi, Boon Kheng, Lee, Wai Mun, Tham, Mun Yew
• Primary Examiner(s): Ho, Dao Q
• Application Number:
  US15/393,094
• Publication Number:
  US 20180181776A1
• Time in Patent Office: 930 Days
• CPC Class Codes: G06F 21/32 using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6218 to a system of files or objects, e.g. local or distributed file system or database
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/84 output devices, e.g. displays or monitors
• CPC Class Codes: G06F 2221/2149 Restricted operating environment
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction

Abstract

Methods and systems are provided in this disclosure for operating an electronic device. The electronic device includes a user interface, a camera, and an electronic processor. The electronic processor determines whether a user assistance restriction is established for a current functional operation of the electronic device. The camera captures at least one image, the electronic processor analyzes the at least one image to detect one or more eyes in the field of view of the camera, and the electronic processor detects a violation of the user assistance restriction for the current functional operation based at least in part on the analysis of the at least one image. When a violation of the user assistance restriction is detected, the electronic processor adjusts one or more operations of the electronic device.

Claims

1. A method of operating an electronic device by detecting unauthorized user assistance, the electronic device including a user interface, a camera, and an electronic processor, the method comprising:

determining, by the electronic processor, whether a user assistance restriction is established for a current functional operation of the electronic device, wherein the user assistance restriction for the current functional operation includes a user viewing another device or object while a user input is received from the user through the user interface;

capturing, by the camera, a series of images, wherein the camera is positioned to capture a series of images of a user of the electronic device;

analyzing, by the electronic processor, the series of images to detect changes in a gaze direction of one or more eyes in the field of view of the camera;

detecting, by the electronic processor, a violation of the user assistance restriction by determining, based at least in part on the analysis of the series of images, that the gaze direction of the one or more eyes detected in the series of images repeatedly moves between the user interface and the other device or object while the user input is received through the user interface; and

adjusting, by the electronic processor, one or more operations of the electronic device in response to detecting the violation of the user assistance restriction.

2. The method of claim 1, further comprising detecting a violation of another user assistance restriction by detecting two or more pairs of eyes from the plurality of eyes in the field of view of the camera.

3. The method of claim 1, further comprising:

analyzing at least one image captured by the camera by detecting whether a gaze direction of each eye from a plurality of eyes in the at least one image is directed towards the user interface; and

detecting a violation of another user assistance restriction by detecting that the gaze direction of more than two eyes from the plurality of eyes detected in the at least one image is directed at the user interface of the electronic device.

4. The method of claim 1, wherein adjusting the one or more operations of the electronic device includes transmitting an alert signal to a remote computer system in response to detecting the violation of the user assistance restriction.

5. The method of claim 1, wherein adjusting the one or more operations of the electronic device includes storing a data record logging the violation in response to detecting the violation of the user assistance restriction.

6. The method of claim 1, wherein adjusting the one or more operations of the electronic device includes adjusting a displayed output to display a warning message or to black out the displayed output in response to detecting two or more users viewing the user interface.

7. The method of claim 1, further comprising detecting when the violation of the user assistance restriction occurs during password entry,

wherein adjusting the one or more operations of the electronic device includes providing access to some operations of the electronic device in response to receiving a password entry while limiting access to the one or more operations of the electronic device in response to detecting the violation of the user assistance restriction during the password entry.

8. The method of claim 1, further comprising calculating a value indicative of a probability of unauthorized user assistance based at least in part on the analysis of the series of images captured by the camera, and

wherein detecting the violation of the user assistance restriction includes detecting the violation when the value is greater than a predefined threshold.

17. The method of claim 1, wherein adjusting one or more operations of the electronic device in response to detecting the violation of the user assistance restriction includes automatically adjusting the one or more operations of the electronic device without providing any indication of the detected violation through the user interface.

18. The method of claim 1, wherein determining whether a user assistance restriction is established for the current functional operation of the electronic device includes

identifying the current functional operation of the electronic device based at least in part on an identification of a software application with which the user is currently interacting through the user interface;

accessing a look-up table stored on a non-transitory computer-readable memory; and

determining, by the electronic processor, whether a user assistance restriction is defined in the look-up table for the current functional operation.

9. An electronic device comprising:

a user interface;

a camera positioned to capture images of a user of the electronic device; and

an electronic processor configured to

determine whether a user assistance restriction is established for a current functional operation of the electronic device, wherein the user assistance restriction for the current functional operation include a user viewing another device or object while a user input is received from the user through the user interface,

analyze a series of images captured by the camera to detect changes in a gaze direction of one or more eyes in the series of images captured by the camera,

detect a violation of the user assistance restriction by determining, based at least in part on the analysis of the series of images, that the gaze direction of the one or more eyes detected in the series of images repeatedly moves between the user interface and the other device or object while the user input is received through the user interface, and

adjust one or more operations of the electronic device in response to detecting the violation of the user assistance restriction.

10. The electronic device of claim 9, wherein the electronic processor is further configured to detect a violation of another user assistance restriction by detecting two or more pairs of eyes from a plurality of eyes in at least one image captured by the camera.

11. The electronic device of claim 9, wherein the electronic processor is further configured to

analyze at least one image captured by the camera to detect whether a gaze direction of each eye from a plurality of eyes in the at least one image is directed towards the user interface, and

detect a violation of another user assistance restriction by detecting that the gaze direction of more than two eyes from the plurality of eyes detected in the at least one image is directed at the user interface of the electronic device.

12. The electronic device of claim 9, wherein the electronic processor is configured to adjust the one or more operations of the electronic device by transmitting an alert signal to a remote computer system in response to detecting the violation of the user assistance restriction.

13. The electronic device of claim 9, wherein the electronic processor is configured to adjust the one or more operations of the electronic device by storing a data record logging the violation in response to detecting the violation of the user assistance restriction.

14. The electronic device of claim 9, wherein the electronic processor is configured to adjust the one or more operations of the electronic device by adjusting a displayed output to display a warning message or to black out the displayed output in response to detecting two or more users viewing the user interface.

15. The electronic device of claim 9, wherein the electronic processor is further configured to detect when the violation of the user assistance restriction occurs during password entry, and

wherein the electronic processor is configured to adjust the one or more operations of the electronic device by providing access to some operations of the electronic device in response to receiving a password entry while limiting access to the one or more operations of the electronic device in response to detecting the violation of the user assistance restriction during the password entry.

16. The electronic device of claim 9, wherein the electronic processor is further configured to calculate a value indicative of a probability of unauthorized user assistance based at least in part on the analysis of the series of images captured by the camera, and

wherein the electronic processor is configured to detect the violation of the user assistance restriction by detecting the violation when the value is greater than a predefined threshold.

19. The electronic device of claim 9, wherein the electronic processor is configured to adjust the one or more operations of the electronic device in response to detecting the violation of the user assistance restriction by automatically adjusting the one or more operations of the electronic device without providing any indication of the detected violation through the user interface.

20. The electronic device of claim 9, wherein the electronic processor is configured to determine whether a user assistance restriction is established for the current functional operation of the electronic device by

identifying the currently functional operation of the electronic device based at least in part on an identification of a software application with which the user is currently interacting through the user interface,

accessing a look-up table stored on a non-transitory computer-readable memory;

and

determining whether a user assistance restriction is defined in the look-up table for the current functional operation.