MULTIPURPOSE CONTROLLERS AND METHODS

• US 20140078049A1
• Filed: 10/15/2013
• Published: 03/20/2014
• Est. Priority Date: 03/12/2011
• Status: Active Grant

• Current Assignee: Perceptive Devices, LLC
• Original Assignee: Uday Parshionikar
• Inventors: Parshionikar, Uday
• Granted Patent: US 9,785,242 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G08B 21/06 indicating a condition of sleep, e.g. anti-dozing alarms

Abstract

Method and apparatus is disclosed for a user to communicate with an electronic device. A processor receives user intention actions comprising facial expression (FE) information indicative of facial expressions and body information indicative of motion or position of one or more body parts of the user. When the FE or body information crosses a first level, the processor starts generating first signals based on the FE or body information to communicate with the electronic device. When the FE or body information crosses a second level, the processor can end generation of the first signals or modify the first signals. An image processing or eye gaze tracking system can provide some FE information or body information. The signals can modify attributes of an object of interest.

Claims

1. An apparatus for a user to use to communicate with an electronic device, the apparatus comprising:

memory;

a processor for receiving facial expression (FE) information indicative of one or more facial expressions performed by the user, for receiving body information indicative of motion or position of one or more body parts of the user, said body information being distinct from said FE information, for comparing said FE information to a first FE level, and for determining whether said FE information crosses said first FE level;

wherein when said FE information crosses said first FE level, said processor starts generation of first signals to communicate with the electronic device, said first signals being based on said body information.

2. The apparatus of claim 1, wherein said processor compares said FE information to a second FE level and determines whether said FE information crosses said second FE level;

and

wherein when said FE information crosses said second FE level, said processor ends generation of said first signals.

3. The apparatus of claim 2, wherein said apparatus further comprises a FE sensor providing at least a portion of said FE information.

4. The apparatus of claim 3, wherein said FE sensor comprises a proximity sensor.

5. The apparatus of claim 3, wherein said FE sensor comprises a touch sensor.

6. The apparatus of claim 3, wherein said FE sensor comprises a sliding motion sensor.

7. The apparatus of claim 3, wherein said FE sensor comprises a body orifice sensor.

8. The apparatus of claim 3, wherein said FE sensor comprises an image sensor.

9. The apparatus of claim 3, wherein said FE information is information indicative of touch between said FE sensor and said one or more facial muscles of the user.

10. The apparatus of claim 2, wherein said apparatus further comprises a motion sensor providing at least a portion of said body information.

11. The apparatus of claim 10, wherein said motion sensor further comprises an inertial sensor.

12. The apparatus of claim 10, wherein said motion sensor further comprises an image sensor.

13. The apparatus of claim 2, wherein said apparatus further comprises an image processing system providing a portion of at least one of said FE information and said body information.

14. The apparatus of claim 2, wherein said apparatus further comprises an eye gaze tracking system generating a portion of at least one of said FE information and said body information.

15. The apparatus of claim 2, wherein said first signals modify attributes of an object of interest (OOI) on the electronic device.

16. The apparatus of claim 15, wherein said first signals are modified based on said FE information.

17. The apparatus of claim 15, wherein said attributes of an OOI comprise location of said OOI.

18. The apparatus of claim 15, wherein said attributes of an OOI comprise volume of said OOI.

19. The apparatus of claim 15, wherein said attributes of an OOI comprise identifier of said OOI.

20. The apparatus of claim 15, wherein said OOI is invisible to human eyes.

21. The apparatus of claim 15, wherein said processor generates second signals based on said FE information.

22. The apparatus of claim 15, wherein said processor receives process information of the electronic device indicative of attributes of one or more processes running on the electronic device.

23. The apparatus of claim 22, wherein said processor modifies said first signals based on said process information of the electronic device.

24. The apparatus of claim 2, wherein said processor saves a portion of Facial Expression Management (FEM) information in said memory or on the electronic device.

25. The apparatus of claim 24, wherein said FEM information comprises at least one of said FE information, said body information and information indicative of said first signals.

26. The apparatus of claim 24, wherein said processor retrieves at least a portion of said FEM information from said memory.

27. The apparatus of claim 1, wherein said FE information is based on position of one or more facial muscles of the user.

28. The apparatus of claim 27, wherein said one or more facial muscles of the user comprises at least one of the muscles of the user'"'"'s cheek.

29. The apparatus of claim 27, wherein said one or more facial muscles of the user comprises at least one of the muscles of the user'"'"'s mouth.

30. The apparatus of claim 27, wherein said one or more facial muscles of the user comprises at least one muscle of the user'"'"'s eye brow.

31. The apparatus of claim 27, wherein one or more facial muscles of the user comprises at least one muscle of the user'"'"'s eye lid.

32. The apparatus of claim 1, wherein said apparatus further comprises a touch sensor.

33. The apparatus of claim 1, wherein said one or more body parts of the user comprises the user'"'"'s head.

34. The apparatus of claim 1, wherein said one or more body parts of the user comprises one or more of the user'"'"'s eye balls.

35. The apparatus of claim 1, wherein said apparatus is included in the electronic device.

36. The apparatus of claim 1, wherein when said processor is generating said first signals, said processor suspends generation of a first portion of said first signals for a first time duration after said FE information crosses said first FE level.

37. The apparatus of claim 36, wherein said processor suspends or resumes generation of a second portion of said first signals based on a rate of change of said FE information, said second portion of said first signals being the same as or different from said first portion of said first signals.

38. A method for a user to use to communicate with an electronic device having a processor and memory, the method comprising:

receiving facial expression (FE) information indicative of one or more facial expressions performed by the user;

comparing said FE information to a first FE level using said processor;

determining whether said FE information crosses said first FE level;

receiving body information indicative of motion or position of one or more body parts of the user, said body information being distinct from said FE information;

measuring said body information using said processor when said FE information crosses said first FE level; and

starting generation of first signals to communicate with the electronic device when said FE information crosses said first FE level, said first signals being based on said body information.

39. A computer readable medium comprising one or more programs configured to be executed by one or more processors to enable a user to communicate with an electronic device, the one or more programs comprising:

instructions for receiving facial expression (FE) information indicative of one or more facial expressions performed by the user;

instructions for comparing said FE information to a first FE level;

instructions for receiving body information indicative of motion or position of one or more body parts of the user, said body information being distinct from said FE information;

instructions for measuring said body information when said FE information crosses said first FE level; and

instructions for generating first signals to communicate with the electronic device when said FE information crosses said first FE level, said first signals being based on said body information.

40. An apparatus for a user to communicate with an electronic device, said apparatus comprising:

one or more processors;

memory; and

one or more programs stored in said memory and configured to be executed by said one or more processors, said one or more programs comprising;

instructions for receiving first body information indicative of motion or position of a first set of one or more body parts of the user, said first body information resulting from action by the user;

instructions for comparing said first body information to a first body level;

instructions for receiving second body information indicative of motion or position of a second set of one or more body parts of the user, said second body information resulting from action by the user, said second body information being distinct from said first body information;

instructions for receiving said second body information when said first body information crosses said first body level; and

instructions for generating first signals to communicate with the electronic device when said first body information crosses said first body level, said first signals being based on said second body information.

41. The apparatus of claim 40, wherein said first set of one or more body parts of the user comprises one or more parts of the first arm of the user.

42. The apparatus of claim 41, wherein said second set of one or more body parts of the user comprises one or more parts of the second arm of the user.

43. An apparatus for a user to communicate with an electronic device, said apparatus comprising:

one or more processors;

memory; and

one or more programs stored in said memory and configured to be executed by said one or more processors, said one or more programs comprising;

instructions for receiving first facial expression (FE) information indicative of a first set of one or more facial expressions of the user, said first FE information resulting from action by the user;

instructions for comparing said first FE information to a first FE level;

instructions for receiving second FE information indicative of a second set of one or more facial expressions of the user, said second FE information resulting from action by the user, said second FE information being distinct from said first FE information;

instructions for receiving said second FE information when said first FE information crosses said first FE level; and

instructions for generating first signals to communicate with the electronic device when said first FE information crosses said first FE level, said first signals being based on said first FE information.

44. The apparatus of claim 43, wherein said one or more programs further comprises:

instructions for comparing said second FE information to a second FE level;

instructions for generating second signals when said second FE information crosses said second FE level, said second signals being based on said second FE information, said second signals being distinct from said first signals.

45. The apparatus of claim 43, wherein said first set of one or more facial expressions comprise facial expression involving the mouth of the user.

46. The apparatus of claim 43, wherein said second set of one or more facial expressions comprise facial expression involving the eye brows of the user.