Apparatus and method for inputting keys using biological signals in head mounted display information terminal

• US 20060061544A1
• Filed: 03/09/2005
• Published: 03/23/2006
• Est. Priority Date: 09/20/2004
• Status: Abandoned Application

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Min, Kyung-Tae, Kim, Youn-Ho
• Application Number:
  US11/076,547
• Publication Number:
  US 20060061544A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A61B 5/389 Electromyography [EMG]
• CPC Class Codes: A61B 5/398 Electrooculography [EOG], e.g. detecting nystagmus; Electroretinography [ERG]
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection

Abstract

Disclosed is an apparatus and method for inputting keys using biological signals in an HMD (Head Mounted Display) mobile information terminal. The apparatus provides a virtual screen that includes a key map and a preview window to a user through a display unit having a micro-display, recognizes and inputs a key selected according to the user'"'"'s biological signals sensed through a biological signal sensing unit having an EOG (Electrooculogram) input unit and an EMG (Electromyogram) input unit for sensing and receiving the biological signals as key inputs. The apparatus recognizes through a recognition unit the key selected according to the user'"'"'s biological signals sensed through a biological signal sensing unit. The user can freely use the HMD mobile communication terminal without using his/her hands because the user can input his/her desired key to the HMD information terminal only by the movement of the user'"'"'s eyes and the biting of his/her right and left back teeth.

Claims

1. An apparatus for inputting keys using biological signals in an HMD (Head Mounted Display) mobile information terminal having an HMD, the apparatus comprising:

a micro-display for displaying a virtual screen;

a memory unit having a key information storage unit for storing key-map information of the virtual screen displayed by the micro-display;

a biological signal sensing unit for sensing biological signals that include voltages produced from a face of a user;

a recognition unit for recognizing the sensed biological signals and key information according to the recognized biological signals; and

a control unit for recognizing the key information according to the biological signals as an input of a specified key.

2. The apparatus as claimed in claim 1, wherein the biological signal includes an electrooculogram (EOG).

3. The apparatus as claimed in claim 1, wherein the biological signal includes an electromyogram (EMG) that is produced by the clenching of left or right back teeth.

4. The apparatus as claimed in claim 1, wherein the biological signal includes an electroencephalogram (EEG).

5. The apparatus as claimed in claim 2, wherein the biological signal sensing unit includes an EOG input unit for inputting a specified key selected by the user according to a potential difference of the EOG to the control unit.

6. The apparatus as claimed in claim 3, wherein the biological signal sensing unit includes an EMG input unit for inputting a specified key selected by the user according to a potential difference of the EMG to the control unit.

7. The apparatus as claimed in claim 5, wherein the biological signal sensing unit includes both the EOG input unit and the EMG input unit.

8. The apparatus as claimed in claim 6, wherein the biological signal sensing unit includes both the EOG input unit and the EMG input unit.

9. The apparatus as claimed in claim 7, wherein the biological signal sensing unit further comprises an EEG sensing unit for receiving the EEG of the user and analyzing a mental state of the user that includes at least one of a mental concentrating state and a resting state of the user.

10. The apparatus as claimed in claim 8, wherein the biological signal sensing unit further comprises an EEG sensing unit for receiving the EEG of the user and analyzing a mental state of the user that includes at least one of a mental concentrating state and a resting state of the user.

11. The apparatus as claimed in claim 1, wherein the control unit changes a background color of the virtual screen according to the EEG sensing unit.

12. The apparatus as claimed in claim 8, wherein the control unit changes a background color of the virtual screen according to the EEG sensing unit.

13. The apparatus as claimed in claim 1, wherein the biological signal sensing unit comprises:

a front sensing unit including sensors capable of sensing voltages produced from upper left and right parts of a nose of the user, and sensors capable of sensing voltages produced from left and right parts of a forehead of the user;

a left side sensing unit capable of sensing a voltage produced from a left temple of the user; and

a right side sensing unit capable of sensing a voltage produced from a right temple of the user.

14. The apparatus as claimed in claim 1, wherein the HMD mobile information terminal has a shape of goggles the frame of which is in close contact with a forehead of the user.

15. The apparatus as claimed in claim 7, wherein the sensors of the front sensing unit for sensing the voltages produced from the upper left and right parts of the nose of the user are positioned on a nose pad part of the goggles type HMD information terminal, and the sensors of the front sensing unit for sensing the voltages produced from the left and right parts of the forehead of the user are positioned on the frame of the goggles type HMD information terminal.

16. The apparatus as claimed in claim 14, wherein the sensors of the front sensing unit for sensing the voltages produced from the upper left and right parts of the nose of the user are positioned on a nose pad part of the goggles type HMD information terminal, and the sensors of the front sensing unit for sensing the voltages produced from the left and right parts of the forehead of the user are positioned on the frame of the goggles type HMD information terminal.

17. The apparatus as claimed in claim 7, wherein the left side sensing unit is positioned on a left temple part of the goggles type HMD information terminal.

18. The apparatus as claimed in claim 14, wherein the left side sensing unit is positioned on a left temple part of the goggles type HMD information terminal.

19. The apparatus as claimed in claim 7, wherein the right side sensing unit is positioned on a right temple part of the goggles type HMD information terminal.

20. The apparatus as claimed in claim 14, wherein the right side sensing unit is positioned on a right temple part of the goggles type HMD information terminal.

21. The apparatus as claimed in claim 17, wherein the EMG input unit comprises:

a left EMG sensing unit for sensing a voltage produced from a left temple muscle of the user and input from the left side sensing unit;

a left EMG potential difference detection unit for receiving the voltage produced from the left temple muscle of the user and detecting a left EMG signal;

a right EMG sensing unit for sensing a voltage produced from a right temple muscle of the user and input from the right side sensing unit;

a right EMG potential difference detection unit for receiving the voltage produced from the right temple muscle of the user and detecting a right EMG signal; and

an EMG signal detection unit for outputting EMG detection signals according to the input left and right EMG signals to the recognition unit.

22. The apparatus as claimed in claim 19, wherein the EMG input unit comprises:

a left EMG sensing unit for sensing a voltage produced from a left temple muscle of the user and input from the left side sensing unit;

a left EMG potential difference detection unit for receiving the voltage produced from the left temple muscle of the user and detecting a left EMG signal;

a right EMG sensing unit for sensing a voltage produced from a right temple muscle of the user and input from the right side sensing unit;

a right EMG potential difference detection unit for receiving the voltage produced from the right temple muscle of the user and detecting a right EMG signal; and

an EMG signal detection unit for outputting EMG detection signals according to the input left and right EMG signals to the recognition unit.

23. The apparatus as claimed in claim 21, wherein the EMG detection signals output when one of the left EMG signal is input, the right EMG signal is input, and both the left and right EMG signal are input.

24. The apparatus as claimed in claim 5, wherein the EOG input unit comprises:

an EOG detection unit for receiving the voltages sensed by the front sensing unit, the left side sensing unit and the right side sensing unit, and detecting EOG signals; and

an EOG recognition unit for recognizing position information to which eyes of the user are directed according to the detected EOG signals.

25. The apparatus as claimed in claim 1, wherein key information storage unit stores information related to at least one key map corresponding to different key input methods for respective mobile communication terminal manufacturers.

26. The apparatus as claimed in claim 1, wherein the key information storage unit stores key map information in which keys are arrange in a circle.

27. The apparatus as claimed in claim 1, further comprising an external interface unit that can be connected to any one of an extended memory and an extended battery.

28. The apparatus as claimed in claim 27, wherein the external interface unit is connected to a notebook PC (Personal Computer) or a post PC, and performs a key input according to the biological signals input from the user through the biological signal sensing unit and the recognition unit.

36. The method as claimed in claim 27, wherein the biological signal includes both the EOG and the EMG.

37. The method as claimed in claim 36, wherein the EOG includes potential difference values between a specified reference voltage and voltages sensed by sensors capable of sensing voltages produced from upper left and right parts of a nose of the user, sensors capable of sensing voltages produced from left and right parts of a forehead of the user, and sensors capable of sensing voltages produced from left and right temples of the user.

38. The method as claimed in claim 37, wherein a position of a cursor, which is recognized according to the input EOG, is determined in accordance with a horizontal coordinate value and a vertical coordinate value by

Horizontal Coordinate Value=(V1+V4)−

(V3+V6)

Vertical Coordinate Value=(V2+V5)−

(V3+V4)

wherein V1 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the right temple of the user, V2 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the right forehead part of the user, V3 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the upper right part of the user'"'"'s nose, V4 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the sensor for sensing the voltage produced from the left temple of the user, V5 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the left forehead part of the user, and V6 denotes a potential difference between the reference voltage and the voltage input from the sensor for sensing the voltage produced from the upper left part of the user'"'"'s nose.

40. The method as claimed in claim 36, wherein step (e) further comprises the steps of:

(g) receiving an input of the EOG from the user;

(h) moving a cursor to a position recognized according to the input EOG;

(i) receiving an input of the EMG from the user; and

(j) recognizing that a key corresponding to the present cursor position is selected according to the input EMG.

42. The method as claimed in claim 40, wherein step (h) further comprises:

(k) determining if the cursor is positioned on a menu selection key for displaying a screen for selecting a menu;

(l) receiving an input of the EMG from the user if the cursor is positioned on the menu selection key; and

(m) recognizing the menu selected by the user according to the input EMG.

44. The method as claimed in claim 42, wherein in step (m) the cursor in a cursor movement direction that corresponds to the user'"'"'s back teeth bitten by the user and sets the cursor to another menu if the user bites any one of the left and right back teeth.

46. The method as claimed in claim 42, wherein in step (m) it is determined if the menu currently set by the cursor is selected by the user if the EMG input by the user is the EMG produced when the user simultaneously bites the left and right back teeth.

29. A method for inputting keys using biological signals in an HMD (Head Mounted Display) mobile information terminal having an HMD, the method comprising:

(a) loading virtual screen information;

(b) displaying a virtual screen according to the loaded virtual screen information;

(c) determining a state of electrodes that receive biological signals produced from a face of a user;

(d) sensing the biological signals;

(e) recognizing keys according to the sensed biological signals; and

(f) receiving a key value according to the key if the key is recognized.

30. The method as claimed in claim 29, wherein the virtual screen information includes information about a kind and a type of key maps set according to a user'"'"'s selection.

31. The method as claimed in claim 30, wherein step (c) further includes the step of determining if the biological signals are in contact with a body of the user.

32. The method as claimed in claim 30, wherein step (c) includes the step of reporting a biological sensor error to the user by means of a message or a warning sound if the electrodes for receiving the input of the biological signals do not operate normally.

33. The method as claimed in claim 29, wherein the biological signal includes an electrooculogram (EOG).

34. The method as claimed in claim 29, wherein the biological signal includes an electromyogram (EMG) that is produced by the clenching of left or right back teeth.

35. The method as claimed in claim 33, wherein the biological signal includes both the EOG and the EMG.

39. The method as claimed in claim 35, wherein step (e) further comprises the steps of:

(g) receiving an input of the EOG from the user;

(h) moving a cursor to a position recognized according to the input EOG;

(i) receiving an input of the EMG from the user; and

(j) recognizing that a key corresponding to the present cursor position is selected according to the input EMG.

41. The method as claimed in claim 39, wherein step (h) further comprises:

(k) determining if the cursor is positioned on a menu selection key for displaying a screen for selecting a menu;

(l) receiving an input of the EMG from the user if the cursor is positioned on the menu selection key; and

(m) recognizing the menu selected by the user according to the input EMG.

43. The method as claimed in claim 41, wherein in step (m) the cursor in a cursor movement direction that corresponds to the user'"'"'s back teeth bitten by the user and sets the cursor to another menu if the user bites any one of the left and right back teeth.

45. The method as claimed in claim 41, wherein in step (m) it is determined if the menu currently set by the cursor is selected by the user if the EMG input by the user is the EMG produced when the user simultaneously bites the left and right back teeth.

47. The method as claimed in claim 35, wherein step (f) further comprises the steps of:

(n) loading at least one key value corresponding to the key selected at the key recognition step;

(o) determining if the EMG for selecting any one of the key values is input from the user; and

(p) receiving an input of the key value according to the input EMG as a key input selected by the user.

48. The method as claimed in claim 47, wherein step (p) further comprises the steps of:

(q) setting a character selection cursor set to any one of the key values according to the EMG; and

(r) receiving an input of the key set by the character selection cursor as the key input selected by the user.

49. The method as claimed in claim 48, wherein in step (q) the character selection cursor moves to the left if the EMG produced when the user bites the left back teeth is input, and moves the character selection cursor to the right if the EMG produced when the user bites the right back teeth is input.

50. The method as claimed in claim 48, wherein in step (r) an input of the key to which the character selection cursor is currently set is received as the key input selected by the user if the EMG produced when the user simultaneously bites the left and right back teeth is input.