MULTIPURPOSE CONTROLLER FOR ELECTRONIC DEVICES, FACIAL EXPRESSIONS MANAGEMENT AND DROWSINESS DETECTION

• US 20120229248A1
• Filed: 03/12/2012
• Published: 09/13/2012
• Est. Priority Date: 03/12/2011
• Status: Active Grant

• Current Assignee: Perceptive Devices, LLC
• Original Assignee: Perceptive Devices, LLC
• Inventors: Parshionikar, Uday, Parshionikar, Mihir
• Granted Patent: US 9,013,264 B2
• Time in Patent Office: Days
• US Class Current: 340/3.1
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

A hands-free controller, a facial expression management system, a drowsiness detection system and methods for using them are disclosed. The controller monitors facial expressions of the user, monitors motions of the user'"'"'s body, generates commands for an electronic device based on the monitored facial expressions and body motions, and communicates the commands to the electronic device. Monitoring facial expressions can include sensing facial muscle motions using facial expression sensors. Monitoring user body motions can include sensing user head motions. Facial expression management can includes monitoring user facial expressions, storing monitored expressions, and communicating monitored expressions to an electronic device. Drowsiness detection can include monitoring eye opening of the user, generating an alert when drowsiness is detected, monitoring proper usage of the device, and generating a warning when improper usage is detected.

Claims

1. A hands-free method of controlling an electronic device by a user, the method comprising:

monitoring facial expressions of the user;

monitoring motions of the user'"'"'s body;

generating commands for the electronic device based on the monitored facial expressions of the user and the monitored motions of user'"'"'s body; and

communicating the commands to the electronic device.

2. The hands-free method of claim 1, wherein monitoring facial expressions of the user comprises sensing motions of facial muscles of the user using a facial expression sensor.

3. The hands-free method of claim 2, wherein the facial expression sensor is a proximity or touch sensor.

4. The hands-free method of claim 2, wherein the facial expression sensor is a mechanical sensor.

5. The hands-free method of claim 4, wherein the mechanical sensor is a mechanical switch, flex sensor, piezoelectric membrane or strain gauge.

6. The hands-free method of claim 2, wherein the facial expression sensor is a biometric sensor.

7. The hands-free method of claim 6, wherein the biometric sensor is an EMG or EOG sensor.

8. The hands-free method of claim 2, wherein the facial expression sensor is an image processing system.

9. The hands-free method of claim 1, wherein monitoring facial expressions of the user comprises sensing touch of facial sensors by facial muscles of the user.

10. The hands-free method of claim 9, wherein the facial sensors are proximity, touch or mechanical sensors.

11. The hands-free method of claim 1, wherein generating commands for the electronic device comprises:

receiving sensor readings from a facial expression sensor monitoring facial expressions of the user;

determining an expression baseline value for the facial expression sensor;

determining an expression threshold value for the facial expression sensor;

ignoring readings from the facial expression sensor below the expression baseline value;

detecting an active facial expression when readings from the facial expression sensor cross the expression threshold value.

12. The hands-free method of claim 11, wherein generating commands for the electronic device further comprises:

receiving sensor readings from a motion sensor monitoring motions of the user'"'"'s body;

determining a motion baseline value for the motion sensor;

determining a motion threshold value for the motion sensor;

ignoring readings from the motion sensor below the motion baseline value;

detecting motion when readings from the motion sensor exceed the motion threshold value.

13. The hands-free method of claim 1, wherein generating commands for the electronic device further comprises:

receiving sensor readings from a motion sensor monitoring motions of the user'"'"'s body;

determining a motion baseline value for the motion sensor;

determining a motion threshold value for the motion sensor;

ignoring readings from the motion sensor below the motion baseline value;

detecting motion when readings from the motion sensor exceed the motion threshold value.

14. The hands-free method of claim 1, wherein monitoring motions of the user'"'"'s body comprises sensing motions of the user'"'"'s head.

15. The hands-free method of claim 14, wherein monitoring motions of the user'"'"'s body comprises sensing motions of the user'"'"'s head using inertial sensors.

16. The hands-free method of claim 14, wherein monitoring motions of the user'"'"'s body comprises sensing motions of the user'"'"'s head using an image processing system.

17. The hands-free method of claim 1, wherein generating commands for the electronic device comprises:

generating selection commands based on a combination of monitored facial expressions and monitored motions of the user'"'"'s body during the monitored facial expressions.

18. The hands-free method of claim 1, wherein generating commands for the electronic device comprises:

receiving sensor readings from a facial expression sensor monitoring facial expressions of the user;

receiving sensor readings from a motion sensor monitoring motions of the user'"'"'s body;

determining an expression threshold value for the facial expression sensor;

detecting an active facial expression when readings from the facial expression sensor cross the expression threshold value;

determining a motion baseline value for the motion sensor;

determining a motion threshold value for the motion sensor;

ignoring readings from the motion sensor below the motion baseline value;

generating a selection command for an object on the electronic device when the active facial expression is detected for more than a minimum selection hold time and less than a maximum selection hold time, and the motion sensor readings are below the motion threshold value for the minimum selection hold time.

19. The hands-free method of claim 18, wherein generating commands for the electronic device further comprises:

generating a click and drag command for the object on the electronic device when the active facial expression is detected for more than the maximum selection hold time and the motion sensor readings are above the motion baseline value;

dragging the object based on the motion sensor readings while the active facial expression is detected.

20. The hands-free method of claim 18, wherein generating commands for the electronic device further comprises:

generating a click and drag command for the object on the electronic device when the active facial expression is detected for more than the maximum selection hold time and the motion sensor readings are above the motion baseline value;

dragging the object based on the motion sensor readings while the facial expression sensor readings are above an expression maintain threshold, the expression maintain threshold being less than the expression threshold value.

21. A method of facial expressions management, the method comprising:

monitoring facial expressions of the user.

22. The facial expressions management method of claim 21, wherein monitoring facial expressions of the user comprises sensing motions of facial muscles of the user using a facial expression sensor.

23. The facial expressions management method of claim 22, wherein the facial expression sensor is a proximity or touch sensor.

24. The facial expressions management method of claim 22, wherein the facial expression sensor is a mechanical sensor.

25. The facial expressions management method of claim 24, wherein the mechanical sensor is a mechanical switch, flex sensor, piezoelectric membrane or strain gauge.

26. The facial expressions management method of claim 22, wherein the facial expression sensor is a biometric sensor.

27. The facial expressions management method of claim 26, wherein the biometric sensor is an EMG or EOG sensor.

28. The facial expressions management method of claim 21, wherein monitoring facial expressions of the user comprises watching the user using an image processing system.

29. The facial expressions management method of claim 21, wherein monitoring facial expressions of the user comprises sensing touch of facial sensors by facial muscles of the user.

30. The facial expressions management method of claim 29, wherein the facial sensors are proximity, touch or mechanical sensors.

31. The facial expressions management method of claim 22, wherein monitoring facial expressions of the user further comprises:

determining a baseline value for the facial expression sensor;

ignoring readings from the facial expression sensor below the baseline value.

32. The facial expressions management method of claim 31, wherein monitoring facial expressions of the user further comprises:

determining a threshold value for the facial expression sensor;

detecting an active facial expression when readings from the facial expression sensor cross the threshold value.

33. The facial expressions management method of claim 22, wherein monitoring facial expressions of the user further comprises:

determining a threshold value for the facial expression sensor;

detecting an active facial expression when readings from the facial expression sensor cross the threshold value.

34. The facial expressions management method of claim 21, further comprising:

monitoring body motions of the user.

35. The facial expressions management method of claim 34, wherein monitoring body motions of the user comprises.

monitoring head motions of the user.

36. The facial expressions management method of claim 34, wherein monitoring body motions of the user comprises.

monitoring head motions of the user using an inertial sensor.

37. The facial expressions management method of claim 21, further comprising. storing monitored facial expressions of the user.

38. The facial expressions management method of claim 37, further comprising.

communicating monitored facial expressions of the user to an electronic device.

39. The facial expressions management method of claim 21, wherein monitoring facial expressions of the user comprises using a device worn on the user'"'"'s head to monitor facial expressions of the user.

40. The facial expressions management method of claim 39, wherein the device worn on the user'"'"'s head has an eyewear structure.

41. The facial expressions management method of claim 39, wherein the device worn on the user'"'"'s head has a headphone structure.

42. A drowsiness detection method for detecting drowsiness of a user, the method comprising:

monitoring eye opening of the user using a monitoring device;

generating an alert when drowsiness is detected based on the monitored eye opening;

monitoring proper usage of the monitoring device; and

generating a warning when improper usage of the monitoring device is detected.

43. The drowsiness detection method of claim 42, wherein the monitoring device senses reflected light to monitor eye opening of the user.

44. The drowsiness detection method of claim 42, wherein monitoring eye opening of the user comprises:

transmitting a beam of light from a source to a sensor;

monitoring obstruction of the transmitted beam.

45. The drowsiness detection method of claim 42, wherein the monitoring device senses change in electric fields to monitor eye opening of the user.

46. The drowsiness detection method of claim 45, wherein the change in electric fields is sensed using electric field sensors.

47. The drowsiness detection method of claim 45, wherein the change in electric fields is sensed using capacitive sensors.

48. The drowsiness detection method of claim 42, wherein proper usage of the monitoring device is monitored using a proximity sensor.

49. The drowsiness detection method of claim 42, wherein proper usage of the monitoring device is monitored using a touch sensor.