User interface apparatus and method

• US 20080030463A1
• Filed: 08/06/2007
• Published: 02/07/2008
• Est. Priority Date: 03/27/1995
• Status: Abandoned Application

• Current Assignee: Donald Forest
• Original Assignee: Donald Forest
• Inventors: Forest, Donald
• Application Number:
  US11/890,355
• Publication Number:
  US 20080030463A1
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: A61F 4/00 Methods or devices enabling patients or disabled persons to operate an apparatus or a device not forming part of the body operating or control means for prostheses A61F2/48, A61F2/68
• CPC Class Codes: G06F 2203/04803 Split screen, i.e. subdividing the display area or the window area into separate subareas
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0219 Special purpose keyboards
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: Y10S 345/902 Menu display

Abstract

The apparatus and method of the invention relate to data entry and menu selection. Applications include: (a) data entry for ideographic languages, including Chinese, Japanese and Korean; (b) fast food ordering; (c) correction of documents generated by optical character recognition; and (d) computer access and speech synthesis by persons temporarily or permanently lacking normal motor capabilities. In a preferred embodiment, each option of a menu is associated respectively with a selectable region displayed adjacent an edge of a display, forming a perimeter menu and leaving a region in the center of the perimeter menu for the output of an application program. Selectable regions may be on the display, outside the display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold period or by a cursor path toward the selectable region, or by a combination thereof. Remaining dwell time required to select a selectable region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored indicating regions and selection of an ideograph may be made by speaking the name of the associated color.

Claims

1. For use with a display area, an apparatus for selecting a selectable region from a plurality of selectable regions, the apparatus comprising:

(a) a receiver for receiving a movement related signal including a first image of a body member of a user, the first image indicating a first location on the display area; and

(b) a processor, coupled to the display area and the receiver, for;

(1) simultaneously displaying a plurality of first selectable regions on the display area, the plurality of first selectable regions together at least partially circumscribing a first region on the display area; and

(2) selecting a particular one of the first selectable regions in response to an intersection of the first location and the particular first selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the first location and the particular first selectable region.

2. The apparatus of claim 1 wherein the processor is further operative to indicate to the user that the particular first selectable region has been selected.

3. The apparatus of claim 1 wherein the particular first selectable region does not border another of the first selectable regions.

4. The apparatus of claim 3 wherein none of the first selectable regions borders another of the first selectable regions.

5. The apparatus of claim 1 wherein the at least partially circumscribed first region is bordered by first selectable regions on at least two sides.

6. The apparatus of claim 1 wherein the movement related signal indicates a first relative position of the body member of the user.

7. The apparatus of claim 6 wherein the movement related signal further indicates a second relative position of the body member of the user;

and wherein the processor is further operative to calculate a difference between the first and the second relative positions of the body member of the user.

8. The apparatus of claim 7 wherein the processor is further operative to derive a movement of the body member of the user from the difference.

9. The apparatus of claim 1 wherein the movement related signal indicates a first absolute position of the body member of the user.

10. The apparatus of claim 9 wherein the movement related signal further indicates a second absolute position of the body member of the user;

and wherein the processor is further operative to calculate a difference between the first and the second absolute positions of the body member of the user.

11. The apparatus of claim 10 wherein the processor is further operative to derive a movement of the body member of the user from the difference.

12. The apparatus of claim 1 wherein the movement related signal indicates a path of movement of the body member of the user;

and wherein the processor is further responsive to the path of movement for selecting the particular first selectable region.

13. The apparatus of claim 1 wherein the movement related signal receiver includes a detector for detecting a movement of the body member of the user.

14. The apparatus of claim 1 wherein the movement related signal receiver includes a detector for detecting the first image of the body member of the user.

15. The apparatus of claim 14 wherein the processor is further operative to process the first image of the body member of the user to determine the first location on the display area indicated by the first image.

16. The apparatus of claim 1 wherein the body member is selected from the group consisting of each of the user'"'"'s limbs, each of the user'"'"'s hands, each of the user'"'"'s feet, each of the user'"'"'s digits, and the user'"'"'s head.

17. The apparatus of claim 1 wherein the processor is further operative, responsive to the intersection of the first location and the particular first selectable region, to simultaneously display a plurality of second selectable regions on the display area, the plurality of second selectable regions together at least partially circumscribing a second region on the display area;

wherein the receiver is operative to receive the movement related signal further including a second image of the body member of the user, the second image indicating a second location on the display area; and

wherein the processor is further operative to select a particular one of the second selectable regions in response to an intersection of the second location and the particular second selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the second location and the particular second selectable region.

18. The apparatus of claim 17 wherein the area occupied by the particular first selectable region on the display area borders the area occupied by the particular second selectable region on the display area.

19. The apparatus of claim 17 wherein the area occupied by the particular first selectable region on the display area at least partially overlaps the area occupied by the particular second selectable region on the display area.

20. The apparatus of claim 17 wherein the processor is further operative to display a plurality of indicating regions on or in close proximity to the particular first selectable region, each of the indicating regions associated respectively with one of the second selectable regions and indicating to the user the location of the associated second selectable region.

21. The apparatus of claim 20 wherein at least part of each indicating region is displayed in one manner and at least part of the second selectable region associated with the indicating region is displayed in a manner similar to the one manner.

22. The apparatus of claim 21 wherein the one manner of display includes any one of:

(a) a size;

(b) a shape;

(c) a hue;

(d) a brightness;

(e) a contrast;

(f) a dithering;

(g) a fill;

(h) a blinking;

(i) a hatching; and

(j) a pattern.

23. For use with a display area, an apparatus for selecting a menu option from a plurality of menu options, the apparatus comprising:

(a) a receiver for receiving a movement related signal including an image of a body member of a user, the image indicating a location on the display area; and

(b) a processor, coupled to the display area and the receiver, for;

(1) simultaneously displaying a plurality of selectable regions on the display area, the plurality of selectable regions together at least partially circumscribing a region on the display area, each of the selectable regions associated respectively with one of the menu options; and

(2) selecting a particular one of the menu options in response to an intersection of the location and the particular selectable region associated with the particular menu option and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region.

24. The apparatus of claim 23 wherein the processor is further operative, responsive to the intersection of the location and the particular selectable region, to display the particular menu option in the at least partially circumscribed region on the display area.

25. The apparatus of claim 23 wherein each of the menu options is associated with a sequence of one or more graphic symbols, one or more of the sequences including one or more ideographs, each of the sequences of one or more graphic symbols having a common characteristic.

26. The apparatus of claim 25 wherein each of the sequences of one or more graphic symbols belongs to any one of the Chinese language, Japanese language, and Korean language.

27. The apparatus of claim 25 wherein each of the sequences of one or more graphic symbols belongs to any one of the Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.

28. The apparatus of claim 23 wherein the particular menu option is associated with a particular sequence of one or more characters, a particular sequence of one or more words, or a particular sequence of one or more symbols representing the sequence of one or more words;

and further comprising a voice output device, coupled to the processor, for speaking the particular sequence responsive to the intersection of the location and the particular selectable region.

29. The apparatus of claim 23 wherein the particular menu option is associated with a particular sequence of one or more characters, a particular sequence of one or more words, or a particular sequence of one or more symbols representing the sequence of one or more words;

and further comprising a voice output device, coupled to the processor, for speaking the particular sequence responsive to the intersection of the location and the particular selectable region.

30. The apparatus of claim 23 wherein each of the menu options has in common a part of speech.

31. The apparatus of claim 23 wherein each of the menu options has in common a meaning.

32. The apparatus of claim 23 wherein each of the menu options has in common a meaning class.

33. The apparatus of claim 23 wherein the particular menu option includes or represents a particular sequence of one or more symbols, at least one symbol of the particular sequence representing a letter.

34. The apparatus of claim 23 wherein the particular menu option includes or represents a particular sequence of one or more symbols, at least one symbol of the particular sequence representing a word.

35. The apparatus of claim 23 wherein the particular menu option includes or represents a particular sequence of one or more symbols, at least one symbol of the particular sequence representing a sentence.

36. The apparatus of claim 23 wherein the particular menu option includes or represents a particular sequence of one or more symbols, at least one symbol of the particular sequence representing a topic of conversation.

37. The apparatus of claim 23 further comprising a computer system operative to execute an application program displaying at least part of its output in the at least partially circumscribed region on the display area;

wherein the particular menu option represents an input to the application program; and

wherein the processor is further operative, responsive to the intersection of the location and the particular selectable region, to provide the input to the application program.

38. The apparatus of claim 23 further comprising a signal generator, coupled to a device, for generating a device control signal corresponding to a device control function for controlling the device;

wherein the particular menu option represents the device control function; and

wherein the signal generator, in response to the selection of the particular menu option, generates the device control signal.

39. The apparatus of claim 38 wherein the device includes any one of:

(a) a computer peripheral;

(b) a device capable of playing previously recorded sound;

(c) a device capable of playing previously recorded video;

(d) a household appliance;

(e) a lamp;

(f) a wheelchair;

(g) a radio;

(h) a robot;

(i) a television;

(j) a thermostat;

(k) a voice output device;

(l) a workstation;

(m) an alarm; and

(n) an office appliance.

40. The apparatus of claim 23 wherein the processor is further operative to display each of the menu options on, or in close proximity to, its associated selectable region.

41. The apparatus of claim 23 wherein the processor is further operative to display each of the menu options on the display area;

and wherein the relative location of the displayed particular menu option with respect to the other displayed menu options indicates to the user the relative location of the particular selectable region with respect to the other selectable regions.

42. The apparatus of claim 23 wherein the processor is further operative to display on the display area each of the menu options and to display on the display area a plurality of indicating regions, each indicating region associated respectively with one of the menu options, each indicating region displayed in a location intersecting or in close proximity to the associated displayed menu option, each of the indicating regions indicating to the user the location of the selectable region associated with the menu option associated with the indicating region;

and wherein the indicating region associated with the particular menu option does not intersect the particular selectable region.

43. The apparatus of claim 42 wherein at least part of each indicating region is displayed in one manner and at least part of the selectable region associated with the menu option associated with the indicating region is displayed in a manner similar to the one manner.

44. The apparatus of claim 43 wherein the one manner of display includes any one of:

(a) a size;

(b) a shape;

(c) a hue;

(d) a brightness;

(e) a contrast;

(f) a dithering;

(g) a fill;

(h) a blinking;

(i) a hatching; and

(j) a pattern.

45. For use with a display area, an apparatus for selecting a desired submenu option from a menu hierarchy, the menu hierarchy including a menu including a plurality of menu options, the menu options including a desired menu option associated with a submenu including a plurality of submenu options, the submenu options including the desired submenu option, the apparatus comprising:

(a) a receiver for receiving a movement related signal including each of a first and a second image of the body member of the user, the first image indicating a first location on the display area, the second image indicating a second location on the display area; and

(b) a processor, coupled to the display area and the receiver, for;

(1) simultaneously displaying a plurality of first selectable regions on the display area, the plurality of first selectable regions together at least partially circumscribing a first region on the display area, each of the first selectable regions associated respectively with one of the menu options;

(2) selecting the desired menu option in response to an intersection of the first location and the particular one of the first selectable regions associated with the desired menu option, and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the first location and the particular first selectable region;

(3) simultaneously displaying a plurality of second selectable regions on the display area, the plurality of second selectable regions together at least partially circumscribing a second region on the display area, each of the second selectable regions associated respectively with one of the submenu options; and

(4) selecting the desired submenu option in response to an intersection of the second location and the particular one of the second selectable regions associated with the desired submenu option, and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the second location and the particular second selectable region.

46. The apparatus of claim 45 wherein the processor is further operative:

(a) to display the submenu options on or in close proximity to the particular first selectable region; and

(b) responsive to the intersection of the first location and the particular first selectable region, to display each of the submenu options on or in close proximity to the second selectable region associated with the submenu option.

47. The apparatus of claim 46 the wherein the ordering on the display area of at least two of the submenu options displayed responsive to the intersection of the first location and the particular first selectable region is responsive to a frequency of use of each of the at least two submenu options.

48. The apparatus of claim 46 wherein the ordering on the display area of at least two of the submenu options displayed responsive to the intersection of the first location and the particular first selectable region is responsive to a recency of use of the each of the at least two submenu options.

49. The apparatus of claim 46 wherein the ordering on the display area of at least two of the submenu options displayed responsive to the intersection of the first location and the particular first selectable region is responsive to a common characteristic of each of the at least two submenu options.

50. The apparatus of claim 46 wherein the ordering on the display area of at least two of the submenu options displayed responsive to the intersection of the first location and the particular first selectable region is responsive to a length of each of the at least two submenu options.

51. The apparatus of claim 46 wherein the ordering on the display area of at least two of the submenu options displayed responsive to the intersection of the first location and the particular first selectable region is responsive to an alphabetic order of each of the at least two submenu options.

52. The apparatus of claim 45 wherein the location on the display area of the desired submenu option relative to the location of one of the other submenu options on the display area, the desired submenu option and the one other submenu option displayed on the display area before the intersection of the first location and the particular first selectable region occurs, indicates to the user the location of the particular second selectable region relative to the second selectable region associated with the one other submenu option.

53. For use with a computer system including a display, an apparatus for selecting a selectable region from a plurality of selectable regions, the apparatus comprising:

(a) a carrier readable by the computer system; and

(b) a program on the carrier, the program executable by the computer system, for;

(1) receiving a movement related signal responsive to movement of a body member of a user, the movement related signal including successive images of the body member of the user, each of the images indicating a location on the display;

(2) simultaneously displaying a plurality of selectable regions on the display, the plurality of selectable regions together at least partially circumscribing a region on the display; and

(3) selecting a particular one of the selectable regions in response to an intersection of a particular one of the successive locations and the particular selectable region and not in response to either one or both of;

(i) a quantity equalling or exceeding a predetermined quantity, the quantity being a function of the durations of one or more periods of intersection of two or more of the successive locations and the particular selectable region; and

(ii) an operation of a switch by the user at or near the time of the intersection of the particular location and the particular selectable region.

54. The apparatus of claim 53 wherein the carrier is selected from the group consisting of:

(a) a random access memory;

(b) a magnetic store; and

(c) an optical store.

55. The apparatus of claim 53 wherein the carrier is a communications network.

56. The apparatus of claim 53 further comprising the computer system.

57. The apparatus of claim 53 wherein the computer system is further capable of executing an application program;

wherein the particular selectable region is associated with an input for the application program; and

wherein the program is further operative to provide the input to the application program in response to the selection of the particular selectable region.

58. The apparatus of claim 53 wherein the program is further operative to indicate to the user the successive locations on the display.

59. For use with a display area, a user interface allowing a user, without touching a pointing device, to select a selectable region from a plurality of selectable regions, the user interface comprising:

(a) a receiver for receiving a movement related signal responsive to a movement of a body member of the user, the movement related signal including an image of the body member of the user, the image indicating a location on the display area; and

(b) a processor, coupled to the display area and the receiver, for;

(1) simultaneously displaying the plurality of selectable regions on the display area, the plurality of selectable regions together at least partially circumscribing a region on the display area; and

(2) selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region;

thereby allowing the user, without touching a pointing device, to select the particular selectable region from the plurality of selectable regions.

60. For use with a computer system including a display, an apparatus allowing a user of the apparatus, without touching a pointing device, to select a selectable region from a plurality of selectable regions, the apparatus comprising:

(a) a medium readable by the computer system; and

(b) a program on the medium, the program executable by the computer system, for;

(1) receiving a movement related signal responsive to a movement of a body member of the user, the movement related signal including an image of the body member of the user, the image indicating a location on the display;

(2) simultaneously displaying the plurality of selectable regions on the display, the plurality of selectable regions together at least partially circumscribing a region on the display; and

(3) selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region;

thereby allowing the user, without touching a pointing device, to select the particular selectable region from the plurality of selectable regions.

61. The apparatus of claim 60 further comprising the computer system.

62. The apparatus of claim 60 wherein the computer system is further capable of executing an application program;

wherein the particular selectable region is associated with an input for the application program; and

wherein the program is further operative to provide the input to the application program in response to the selection of the particular selectable region.

63. For use with a display area, a user interface allowing a user, without touching a pointing device, to select a selectable region from a plurality of selectable regions, the user interface comprising:

(a) a receiver for receiving an image of the body member of the user, the image indicating a location on the display area; and

(b) a processor, coupled to the display area and the receiver, for;

(1) simultaneously displaying the plurality of selectable regions on the display area, the plurality of selectable regions together at least partially circumscribing a region on the display area; and

(2) selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region;

thereby allowing the user, without touching a pointing device, to select the particular selectable region from the plurality of selectable regions.

64. For use with a display area, an apparatus for selecting a selectable region from a plurality of selectable regions, the plurality of selectable regions simultaneously displayed on the display area, the plurality of selectable regions together at least partially circumscribing a region on the display area, the apparatus comprising:

(a) movement related signal receiving means for receiving a movement related signal indicating a position of a body member of a user, the position indication including an image of the body member of the user, the image indicating a location on the display area; and

(b) selection means, operatively connected to the movement related signal receiving means, for selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region.

65. The apparatus of claim 64 wherein the movement related signal receiving means includes:

(1) a timer; and

(2) a detector, responsive to an expiration of the timer, for detecting the image of the body member of the user.

66. The apparatus of claim 64 wherein the selection means is further operative to represent the particular selectable region as an area on the display area;

and wherein the selection means further comprises a comparator for comparing the location on the display area and the area on the display area to determine whether the location on the display area intersects the area on the display area.

67. For use with a display area, a method of selecting a selectable region from a plurality of selectable regions, the plurality of selectable regions simultaneously displayed on the display area, the plurality of selectable regions together at least partially circumscribing a region on the display area, the method comprising:

(a) receiving a movement related signal including a first image of a body member of a user, the first image indicating a location on the display area; and

(b) selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user at or near the time of the intersection of the location and the particular selectable region.

68. The method of claim 67 wherein the movement related signal further includes a second image of the body member of the user;

and further comprising calculating a difference between the first and the second images.

69. The method of claim 67 wherein the movement related signal further includes a second image of the body member of the user;

and further comprising comparing the first and the second images to derive a movement of the body member of the user.

70. For use with a computer coupled to a display, a method for a user of the computer, without touching a pointing device, of selecting a selectable region from a plurality of selectable regions displayed on the display, the method comprising:

(a) receiving a movement related signal responsive to a movement of a body member of the user, the movement related signal including an image of the body member of the user, the image indicating a location on the display;

(b) simultaneously displaying a plurality of selectable regions on the display, the plurality of selectable regions together at least partially circumscribing a region on the display; and

(c) selecting a particular one of the selectable regions in response to an intersection of the location and the particular selectable region and not in response to either one or both of;

(i) a dwell event; and

(ii) an operation of a switch by the user substantially at the time of the intersection of the location and the particular selectable region.