Sound operated menu method and apparatus

• US 5,999,895 A
• Filed: 07/24/1995
• Issued: 12/07/1999
• Est. Priority Date: 07/24/1995
• Status: Expired due to Term

• Current Assignee: Donald K. Forest
• Original Assignee: Donald K. Forest
• Inventors: Forest, Donald K.
• Primary Examiner(s): Voeltz, Emanuel Todd
• Assistant Examiner(s): Alvarez, Raquel
• Application Number:
  US08/506,152
• Time in Patent Office: 1,597 Days
• Field of Search: 364/419.01, 364/419.03, 364/419.09, 395/2.6, 395/2.84, 395/2.63, 395/2.4, 395/758, 395/797, 395/798, 395/760, 434/167, 382/185, 707/535, 400/110, 704/1
• US Class Current: 704/1
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0237 using prediction or retrieval techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G09B 21/009 Teaching or communicating with deaf persons G09B21/04 takes precedence; deaf and dumb persons G09B21/00

Abstract

The apparatus and method of the invention relate to data entry and menu selection. Applications include: (a) data entry for ideographic languages, including Chinese, Japanese and Korean; (b) fast food ordering; (c) correction of documents generated by optical character recognition; and (d) computer access and speech synthesis by persons temporarily or permanently lacking normal motor capabilities. In a preferred embodiment, each option of a menu is associated respectively with a selectable region displayed adjacent an edge of a display, forming a perimeter menu and leaving a region in the center of the perimeter menu for the output of an application program. Selectable regions may be on the display, outside the display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold period or by a cursor path toward the selectable region, or by a combination thereof Remaining dwell time required to select a selectable region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored indicating regions and selection of an ideograph may be made by speaking the name of the associated color.

Claims

1. A method of selecting a sequence of one or more graphic symbols from a first plurality of sequences of one or more graphic symbols, one or more sequences of the first plurality of sequences including one or more ideographs in an ideographic language, said method comprising the steps of:

displaying on a display the first plurality of sequences, each of the first plurality of sequences having a first common characteristic in the ideographic language and each associated respectively with one of a first plurality of displayed sound indicators, each of the first sound indicators respectively indicating one of a first plurality of indicated sounds, each of the first indicated sounds differing from each other first indicated sound;

matching a first received sound to any one of the first indicated sounds; and

selecting the sequence associated with the matched first indicated sound.

2. The method of claim 1 wherein the first received sound is not a phonetic representation of the sequence associated with the matched first indicated sound.

3. The method of claim 1 wherein one of the first sound indicators is non-alphanumeric.

4. The method of claim 3 wherein the non-alphanumeric sound indicator is a hue.

5. The method of claim 3 wherein the non-alphanumeric indicator includes any one of a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch indicator, a change in volume indicator, a shape indicator, a color saturation indicator, a brightness indicator, a location indicator, and an object indicator.

6. The method of claim 1 further comprising the step of indicating the association of each of the first plurality of sequences with the associated sound indicator.

7. The method of claim 1 wherein each of the first plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.

8. The method of claim 1 wherein the first common characteristic includes a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation.

9. The method of claim 8 wherein the phonetic unit is specified by voice.

10. The method of claim 8 wherein each of the first sound indicators includes a hue of a region on the display each region having a hue different from the hue of all the regions of all other first sound indicators;

wherein each region is either adjacent or nearly adjacent another of the regions;

wherein the displaying step displays each sequence intersecting the region of the associated sound indicator; and

wherein the first received sound is the name of the hue of the sound indicator to be matched.

11. The method of claim 10 further comprising the step of inputting the selected sequence to an application program.

12. The method of claim 8 wherein the phonetic unit is specified by one or more keystrokes.

13. The method of claim 8 wherein the phonetic unit is a phonetic unit in a first dialect of the ideographic language, and wherein the matching step includes matching the phonetic unit to a phonetic unit in a second dialect of the ideographic language.

14. The method of claim 1 wherein the first common characteristic includes any one of:

(a) an intonation;

(b) a distinct sound;

(c) a part of speech;

(d) a meaning; and

(e) a meaning class.

15. The method of claim 1 wherein the first common characteristic includes any one of:

(a) a stroke used to draw an ideograph;

(b) a number of horizontal strokes;

(c) a number of vertical strokes;

(d) a number of total strokes;

(e) a stroke order;

(f) a radical;

(g) an ideograph;

(h) a kana;

(i) a diacritic;

(j) a classification of a part of an ideograph;

(k) a root;

(l) a graphic symbol;

(m) an appearance of a graphic symbol; and

(n) an attribute for optical recognition purposes.

16. The method of claim 1 wherein the displaying step precedes the matching step.

17. The method of claim 1 wherein the first plurality of sound indicators together at least partially circumscribe a region on the display.

18. The method of claim 17 wherein each of the first sound indicators is outside the display.

19. The method of claim 1 wherein each of the first sound indicators is on the display and either adjacent or nearly adjacent another of the first sound indicators.

20. The method of claim 1 further comprising the steps of:

receiving a signal; and

responsive to the received signal, displaying on the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the first common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators respectively indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound.

21. The method of claim 20 wherein each sequence of the first plurality of sequences has a frequency of use greater than each sequence of the second plurality of sequences.

22. The method of claim 1 further comprising the steps of:

first storing a representation of the first common characteristics; and

second storing a representation of a second common characteristic.

23. The method of claim 22 wherein the first storing step and the second storing step precede the displaying step.

24. The method of claim 23 followed by the steps of:

displaying oil the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the second common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound;

matching a second received sound to any one of the second indicated sounds; and

selecting the sequence of the second plurality of sequences associated with the matched second indicated sound.

25. The method of claim 1 wherein one of the first sound indicators is numeric.

26. The method of claim 1 wherein the displaying step displays each sequence of the first plurality of sequences in an order responsive to a second characteristic of the sequence other than the first common characteristic, the second characteristic including any one of:

(a) a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation;

(b) an intonation;

(c) a distinct sound;

(d) a stroke used to draw an ideograph;

(e) a number of horizontal strokes;

(f) a number of vertical strokes;

(g) a number of total strokes;

(h) a stroke order;

(i) a radical;

(j) a part of speech;

(k) an ideograph;

(l) a kana;

(m) a diacritic;

(n) a classification of a part of an ideograph;

(o) a meaning;

(p) a meaning class;

(q) a root;

(r) a graphic symbol;

(s) an appearance of a graphic symbol;

(t) an attribute for optical recognition purposes;

(u) a frequency of use of each of the first plurality of sequences;

(v) a number of graphic symbols; and

(w) a frequency of use of each of a second plurality of sequences, of graphic symbols, each of the second plurality of sequences including one of the first plurality of sequences.

27. The method of claim 1 further comprising the step of inputting the selected sequence to an application program.

28. The method of claim 1 further comprising the steps of:

inputting a first sequence of one or more graphic symbols including one or more ideographs to an application program; and

replacing the first sequence with the selected sequence.

29. The method of claim 1 wherein the first common characteristic includes a first graphic symbol and the displaying step displays only a single instance of the first graphic symbol indicating that the first graphic symbol is included in each of the first plurality of sequences.

30. The method of claim 29 wherein the first graphic symbol is an ideograph.

31. The method of claim 29 wherein the displaying step displays the single instance of the first graphic symbol in a region of the display separate from the display of at least part of one of the first plurality of sequences or in a manner visibly distinct from the display of at least part of one of the first plurality of sequences.

32. The method of claim 1 wherein one of the first sound indicators includes at least part of a particular graphic symbol of the associated sequence.

33. The method of claim 32 wherein the first received sound includes any one of:

(a) a pronunciation of at least part of the particular graphic symbol;

(b) a name of at least part of the particular graphic symbol;

(c) a meaning of at least part of the particular graphic symbol;

(d) a meaning class of at least part of the particular graphic symbol;

(e) a part of speech of at least part of the particular graphic symbol;

(f) a classification of at least part of the particular graphic symbol;

(g) a stroke used to draw at least part of the particular graphic symbol;

(h) a number of horizontal strokes used to draw the particular graphic symbol;

(i) a number of vertical strokes used to draw the particular graphic symbol; and

(j) a number of total strokes used to draw the particular graphic symbol.

34. The method of claim 1 wherein the number of sequences in the first plurality of sequences is less than or equal to a predetermined number thereby increasing the likelihood of matching of the first received sound to the first indicated sound intended by a user.

35. For use in a language rich in homophones, a method of selecting a sequence of one or more graphic symbols in the language from a plurality of sequences of one or more graphic symbols in the language, said method comprising the steps of:

(a) displaying the plurality of sequences, each of the plurality of sequences associated respectively with a sound which is not a phonetic representation of the associated sequence, each of the plurality of sounds differing from each other sound in the plurality of sounds;

(b) receiving a sound signal;

(c) matching the received sound signal to any one of the plurality of sounds; and

(d) selecting the sequence associated with the matched sound.

36. The method of claim 35 wherein at least two of the sequences are homophones.

37. The method of claim 35 wherein each sequence represents a word in the language.

38. The method of claim 35 further comprising the steps of:

(e) displaying a plurality of sound indicators, each sound indicator associated respectively with one of the plurality of sounds; and

(f) indicating the association of each sequence to the sound indicator associated with the sound associated with the sequence.

39. A method of editing a first sequence of two or more words in an ideographic language, said method comprising the steps of:

displaying on a display;

(a) the first sequence, the first sequence including a plurality of second sequences of one or more graphic symbols in the language, each of the second sequences associated respectively with a sound indicator, each of the sound indicators respectively indicating a sound, each of the indicated sounds differing from each of the other indicated sounds; and

(b) the sound indicators;

receiving a sound;

matching the received sound to any one of the indicated sounds; and

replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.

40. For use with a general purpose computer system including a display, an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including an ideograph in an ideographic language, said apparatus comprising:

(a) a computer readable medium; and

(b) a program, stored on the medium and executable by the general purpose computer system, for;

(1) displaying the plurality of sequences on the display, each of the sequences associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator;

(2) matching a received sound to the sound indicated by the particular sound indicator; and

(3) selecting the sequence associated with the particular sound indicator.

41. The apparatus of claim 40 wherein each of the plurality of sequences has a common characteristic in the ideographic language.

42. The apparatus of claim 40 wherein the medium may be coupled to and uncoupled from the general purpose computer system.

43. The apparatus of claim 40 wherein the medium includes any one of:

(a) a random access memory;

(b) a magnetic store; and

(c) an optical store.

49. The apparatus of claim 40 wherein the program is transferred from the medium to the general purpose computer system via a telecommunications or data communications network.

44. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:

(a) a display on which may be displayed the plurality of sequences, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and

(b) a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.

45. The apparatus of claim 44 wherein the common characteristic includes any one of the meaning classes:

(a) actions;

(b) amounts;

(c) animals;

(d) articles of clothing;

(e) bodily functions;

(f) buildings;

(g) business activities;

(h) cleaning activities;

(i) colors;

(j) communication activities;

(k) computer peripherals;

(l) days;

(m) devices used to maintain personal hygiene;

(n) directions;

(o) drinks;

(p) emergency conditions;

(q) emotions;

(r) financial activities;

(s) foods;

(t) government services;

(u) greetings;

(v) holidays;

(w) household appliances;

(x) illnesses;

(y) items of office equipment;

(z) jokes;

(aa) lengths;

(ab) locations, including locations frequented by a user of the apparatus;

(ac) meals;

(ad) means of transportation;

(ae) months;

(af) names;

(ag) numbers;

(ah) parts of the human body;

(ai) persons known to a user of the apparatus;

(aj) plants;

(ak) prosthetic device;

(al) recreational activities;

(am) rehabilitation activities;

(an) relative locations;

(ao) school activities;

(ap) shapes;

(aq) shopping activities;

(ar) sizes;

(as) smells;

(at) sports;

(au) tactile attributes;

(av) tastes;

(aw) telephone numbers;

(ax) temperatures;

(ay) times;

(az) topics of study;

(ba) utterances used as acknowledgements in conversation without conveying new substantive information;

(bb) utterances used to bid for a turn to speak in conversation;

(bc) weights; and

(bd) work activities.

46. The apparatus of claim 44 further comprising:

(a) a computer system including the display and a sound receiver for receiving the received sound; and

(b) a program, executable on the computer system;

and wherein the selection device is formed by the combination of the progam and the computer system.

47. The apparatus of claim 44 wherein the selection device includes a processor, a program executable oil the processor, and a medium readable by the processor;

wherein the processor, during execution of the program, compares a representation of the received sound to a representation of each of the indicated sounds, the representation of each of the indicated sounds stored on the medium and read by the processor therefrom; and

wherein the sound indicated by tile particular sound indicator is tile indicated sound whose representation is closest to the representation of the received sound.

48. The apparatus of claim 44 wherein the selection device includes a speech recognizer for comparing the received sound to the sound indicated by the particular sound indicator.

50. The apparatus of claim 44 wherein the sound indicated by the particular sound indicator is either one of:

(a) a sound found in normal speech; and

(b) a sound not found in normal speech.

51. The apparatus of claim 44 wherein each of the plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.

52. The apparatus of claim 44 wherein the plurality of sound indicators together at least partially circumscribe a region on the display.

53. The apparatus of claim 52 wherein each of the sound indicators is outside the display.

54. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of tile plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:

(a) means for displaying tile plurality of sequences, each of tile plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with tile particular sound indicator; and

(b) means for;

(1) matching a sound to the sound indicated by the particular sound indicator; and

(2) selecting the sequence associated with the particular sound indicator.

55. A device controller comprising:

(a) a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and

(b) a signal generating device, coupled to a controlled device, for;

(1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and

(2) generating a device control signal corresponding to the function of the controlled device.

56. The apparatus of claim 55 wherein the controlled device includes any one of:

(a) a computer peripheral;

(b) a device capable of playing previously recorded sound;

(c) a device capable of playing previously recorded video;

(d) a household appliance;

(e) a lamp;

(f) a microprocessor;

(g) a motorized transport device including either one of a scooter and a wheelchair;

(h) a radio;

(i) a robot;

(j) a security system;

(k) a television;

(l) a thermostat;

(m) a voice output device;

(n) a workstation;

(o) an alarm; and

(p) an office appliance.