Semantic user interface

• US RE39,090 E1
• Filed: 10/26/2001
• Issued: 05/02/2006
• Est. Priority Date: 07/03/1997
• Status: Expired due to Term

• Current Assignee: Activeword Systems, Inc.
• Original Assignee: Activeword Systems, Inc.
• Inventors: Armijo-Tamez, Jesus Roberto, Beauregard, Serge Pierre
• Primary Examiner(s): ALAM, SHAHID AL
• Application Number:
  US09/983,981
• Time in Patent Office: 1,649 Days
• Field of Search: 707/1, 707/2, 707 3- 6, 707/100, 707/101, 707/103.R, 707/104.1, 707/202, 707/10, 704/251, 704/252, 704/270, 704/271, 704/9, 704/10, 379/88.22, 345/781
• US Class Current: 1/1
• CPC Class Codes: G06F 9/45512 Command shells
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99936 Pattern matching access

Abstract

A system and method that allows a user to use their everyday language or user defined words to operate the computer in a highly efficient way. In short, every word, letter, control character and symbols is potentially actionable. A computer user'"'"'s productivity is dramatically increased by making available those functions that enable a user to produce most of his work through simple, language-based commands. The present invention provides an intuitive interface, referred to as a semantic user interface (SUI), that enhances the operation of the current standard window-based interface in a manner that is simple, richer and natural. By leveraging all of the richness and power inherent in a user'"'"'s language, the present invention provides an important tool that allows the personal computer to operate in a manner that is much closer to our natural way of interacting. A user is allowed to enter “commands” in his everyday natural language in order to control the operations of the computer. All commands are language-based and user-defined. These commands can be entered from any context of the user'"'"'s computer (e.g., any application or operating system workspace). The commands allows a user to launch applications and navigate within applications by using language rather than clicks from a pointing device such as a mouse. It also allows the replacement of keystrokes with stored words or keystrokes. The system keeps a complete archive record of all the text content the user provides as input, regardless of which application program or operating system window the user is operating in at the time. The combined set of all user defined commands and the memory of all the input text that is stored in the archive constitutes the personality profile and is transportable from one computer to another.

Claims

1. A system for permitting a user to implement functionality on a computer, the functionality being provided across a plurality of application programs or within an environment created by an operating system, the computer including a data entry device, comprising:

means for monitoring all data entered or selected by a user within and any one of the plurality of application programs or within an environment created by the operating system, said data including one or more alphabetic letters, symbols and/or words, wherein a certain combinations of data represents an action words ;

a wordbase having stored therein a plurality of item records, each wherein an item record having an corresponds to both said action word and one of a plurality of associated functions;

means for searching said wordbase for a match with an between said action word entered by the user and said item record corresponding thereto; and

means for performing said function associated with said action word.

2. The system of claim 1, wherein said data is entered via a microphone, selection device, or keyboard.

3. The system of claim 2, further comprising means for recognizing voice signals input via said microphone to produce recognizable data, wherein said recognizable data is used by said means for searching.

4. The system of claim 1, wherein said data entered by said user may be selected with a selection device by said user.

5. The system of claim 1, wherein a word entered by said user is a dual word, wherein said user diambiguates said dual word to indicate to said means for monitoring that said dual word is an action word.

6. The system of claim 1, wherein said data includes code words, dual words and content words.

7. The system of claim 6, further comprising means for providing feedback to said user when said user enters a dual word.

8. The system of claim 1, further comprising means for providing statistics regarding said user'"'"'s input activity, including the context said user was operating within at the time of said input activity.

9. The system of claim 1, wherein said data is captured to said means for monitoring prior to the operating system forwarding said data to one of the plurality of application programs.

10. The system of claim 1, further comprising a display device for displaying said data, wherein said data is erased if it represents an action word.

11. The system of claim 1, wherein said data is displayed within the data entry fields of one of the plurality of application programs.

12. The system of claim 1, further comprising means for forwarding said data to one of the plurality of application programs.

13. The system of claim 12, further comprising a state table that stores all information that has been forwarded to said one of the plurality of application programs.

14. The system of claim 1, wherein said action word is a dual word, said system further comprising means for providing a signal that a dual word has just been entered by said user.

15. The system of claim 1, wherein said function can be activated via a plurality of action words.

16. The system of claim 1, wherein said means for monitoring includes a virtual device driver.

17. The system of claim 1, wherein said means for monitoring monitors for a delineator, wherein an action word is always followed by a delineator.

18. The system of claim 17, wherein said delineator is a punctuation mark, a special character, entry of a space bar, or a click of a selection device.

19. The system of claim 17, wherein said delineator includes a context switch between application programs.

20. The system of claim 1, further comprising means for forwarding said data to the operating system.

21. The system of claim 1, wherein said wordbase includes a plurality of folders, wherein each folder has a priority associated therewith.

22. The system of claim 21, further comprising means for changing the priority of said plurality of folders.

23. The system of claim 1, further comprising means for displaying a charm box, said charm box having displayed therein information relating to said data entered by said user.

24. The system of claim 1, further comprising a mathematical application program for performing in-place arithmetic, said mathematical application program being triggered by an action word.

25. The system of claim 1, further comprising means for displaying a monitor, said monitor having a field for displaying said data and a field for displaying said function being performed.

26. The system of claim 1, wherein each item record within said wordbase includes a frequency count of dual word matches and code word matches.

27. The system of claim 1, wherein said function is performed by executing a script.

28. The system of claim 1, wherein said wordbase includes an archive of all data entered by said user.

29. The system of claim 28, wherein said archive includes a 7×

7 organization of said data.

30. The system of claim 1, wherein said function includes launching an application program, a file or a folder.

31. The system of claim 1, wherein said function includes text substitution, wherein said text is substituted at the position of a displayed curser cursor.

32. The system of claim 31, further comprising means for toggling between at least two choices for said text substitution.

33. The system of claim 1, wherein at least a subset of said action words are user defined, wherein said user can add, delete and modify said action words within said wordbase.

34. The system of claim 1, wherein said means for monitoring can be toggled between on and off.

35. The system of claim 1, wherein the environment is providing a graphical user interface (GUI).

36. The system of claim 35, further comprising means for selecting a block of data that is displayed via said GUI or one of the plurality of application programs, wherein said data can be entered by selecting said block of data.

37. The system of claim 1, wherein said associated function may include calling an agent.

38. The system of claim 1, wherein said wordbase is located on a server connected to a network.

39. The system of claim 1, further comprising means for providing said user with statistical feedback regarding said data.

40. The system of claim 1, wherein a single action word can activate two or more functions, the system further comprising means for selecting between said two or more functions when said single action word is entered by said user.

41. The system of claim 1, wherein said action word is formed by at least two natural language words.

42. The system of claim 1, further comprising means for generating and displaying statistical data regarding the productive use of said action words by said user.

43. The system of claim 1, further comprising a state table that includes a list of said data most recently entered by said user.

44. The system of claim 1, wherein said action words include code words and dual words, the system further comprising means for allowing said user to turn said code words and said dual words on and off within said wordbase.

69. The system of claim 1, further comprising means for defining at least two personal profiles, wherein each personal profile controls the computer with a different set of action words.

70. The system of claim 1, wherein said wordbase can be shared by more than one user.

71. The system of claim 1, wherein the computer is connected to a server.

72. The system of claim 1, wherein said associated function is selected from the group of:

navigation, information, substitution, and control.

73. The system of claim 1, wherein one of said plurality of associated functions includes launching an application program or Internet site.

45. A method for permitting a user to implement functionality on a computer having a graphical user interface and data entry device, comprising:

1) providing a wordbase having a plurality of item records, each item record having stored therein an action word, organizing said item records to define at least one personal profile, wherein said action word comprises natural language or code word or text strings;

2) associating a plurality of agents with said wordbase, wherein each agent performs one of a plurality of functions;

3) associating said action word stored in said wordbase with a function performed by one of said plurality of agents;

4) receiving a data string input by the user within an application program or an operating system environment or data selected by a user within said application program or said operating system environment, wherein said data string selected by the user is displayed on the graphical user interface;

5) determining if said data string input by the user or said data selected by the user is an action word stored in said wordbase; and

6) performing via one of said plurality of agents said function associated with said action word stored in said wordbase.

46. The method of claim 45, wherein performing said function includes launching an application program and passing data to the newly launched application.

47. The method of claim 45, further comprising the step of populating said wordbase with user defined action words.

74. The system of claim 45, wherein said function includes navigation, information, substitution and control.

48. A method for allowing a user to control a computer having an operating system that provides a graphical user interface (GUI), comprising the steps of:

(1) providing a semantic user interface (SUI) that complements the GUI, said semantic user interface seamlessly integrated with the operating system;

(2) allowing a user to enter keystrokes;

(3) monitoring for said keystroke by said SUI; and

(4) performing an action associated with said keystrokes, wherein said action can be performed within any application program running on the computer or within an environment created by the operating system.

53. The method of claim 48, wherein said action is selected from the group of:

navigation, information, substitution and control.

54. The method of claim 48, further comprising the steps of allowing a user to entervoice enter voice data via a speech recognition unit, monitored monitoring for said voice data by said SUI, and performing an action associated with said voice data, wherein said action can be performed within any application program running on the computer or within said environment created by the operating system .

55. The method of claim 48, wherein said keystrokes entered by said user can form a content word, a code word or a dual word, wherein the method further comprises disambiguating a dual word.

56. The method of claim 55, wherein said action is selected from the group of:

navigation, information, substitution and control.

57. The method of claim 56, further comprising the step of allowing said user to select said keystrokes that result in said action being performed.

58. The method of claim 55, further comprising the step of providing a signal to said user when said user enters a dual word.

59. The method of claim 55, further comprising the step of erasing a code word or dual word after it is entered by said user.

60. The method of claim 48, wherein said step of monitoring occurs prior to the operating system forwarding said keystrokes to any application program.

61. The method of claim 48, wherein said keystrokes can form two or more words that result in said action being performed.

62. The method of claim 48, wherein the step of monitoring monitors for entry of an action word followed by a delineator.

63. The method of claim 48, further comprising displaying a charm box, said charm box having displayed therein information relating to said keystrokes entered by said user.

64. The method of claim 48, wherein said step of providing said SUI further comprises the step of displaying a monitor, said monitor having a field for displaying said keystrokes and a field for displaying said action being performed.

65. The method of claim 48, wherein said action includes launching an application program, a file or a folder.

66. The method of claim 48, further comprising the steps of:

displaying at least two actions that can be performed;

allowing said user to toggle between said at least two actions; and

allowing said user to select one of said at least two actions.

67. The method of claim 48, wherein the step of monitoring can be toggled between on and off.

68. The method of claim 48, further comprising providing statistics regarding said user'"'"'s input activity, including the context said user was operating within at the time said keystrokes were entered.

49. A method for allowing a user to control a computer within a network, comprising the steps of:

(1) providing a semantic user interface (SUI);

(2) allowing a user to enter data, wherein said data can be entered via a microphone, selection device or keyboard;

(3) monitoring for said data by said SUI; and

(4) performing an action associated with said keystroke data, wherein said action can be performed within any application program running on the computer or within an environment created by an operating system, wherein said action is user definable.

50. The method of claim 49, further comprising the step of populating a wordbase, wherein said wordbase includes a plurality of item records, each item record having a one or more keystrokes and an associated action.

51. The method of claim 50, wherein said action is selected from the group of:

navigation, information, substitution and control.

52. The method of claim 49, wherein said network includes a server that executes an operating system that provides a graphical user interface (GUI), wherein said SUI complements said GUI.

75. A system for permitting a user to implement functionality on a computer, the functionality being provided across a plurality of application programs or within an environment created by an operating system, the computer including a data entry device, comprising:

means for monitoring all data entered or selected by a user within any one of the plurality of application programs or within an environment created by the operating system, said data including one or more alphabetic letters, symbols and/or words, wherein a certain combinations combination of data represent represents an action words word;

a wordbase having stored therein a plurality of item records, each wherein an item record having an corresponds to both said action word and one of a plurality of associated functions, wherein said associate associated functions are selected from the group of;

information, navigation, control and substitution;

means for searching said wordbase for a match with an between said action word entered by said the user and said item record corresponding thereto; and

means for performing said function associated with said action word.

76. The system of claim 75, wherein said data is entered via a microphone, selection device, or keyboard.

77. The system of claim 75, further comprising means for recognizing voice signals input via said microphone to produce recognizable data, wherein said recognizable data is used by said means for searching.

78. The system of claim 75, wherein said data entered by said user may be selected with a selection device by said user.

79. The system of claim 75, wherein a word entered by said user is a dual word, wherein said user disambiguates said dual word to indicate to said means for monitoring that said dual word is an action word.

80. The system of claim 75, wherein said data includes code words, dual words and content words.

81. The system of claim 80, further comprising means for providing feedback to said user when said user enters a dual word.

82. The system of claim 75, wherein said data is captured by said means for monitoring prior to the operating system forwarding said data to one of the plurality of application programs.

83. The system of claim 75, further comprising a display device for displaying said data, wherein said data is erased if it represents an action word.

84. The system of claim 75, wherein said function can be activated via a plurality of action words.

85. The system of claim 75, wherein said means for mounting monitors for a delineator, wherein an said action word is always followed by an said delineator.

86. The system of claim 85, wherein said delineator is a punctuation mark, a special character, entry of a space bar, or a click of a selection device.

87. The system of claim 85, wherein said delineator includes a context switch between application programs.

88. The system of claim 75, wherein said function includes launching an application program, a file or a folder.

89. The system of claim 75, wherein said function includes text substitution, wherein said text is substituted at the position of a displayed curser cursor.

90. The system of claim 89, further comprising means for toggling between at least two choices for said text substitution.

91. The system of claim 75, wherein at least a subset of said action words are user defined, wherein said user can add, delete and modify said action words within said wordbase.

92. The system of claim 75, wherein said means for monitoring can be toggled between on and off.

93. The system of claim 75, wherein said wordbase is located on a server connected to a network.

94. The system of claim 75, wherein a single action word can activate two or more functions, the system further comprising means for selecting between said two or more functions when said single action word is entered by said user.

95. The system of claim 75, wherein said action word is formed by at least two natural language words.

96. The system of claim 75, further comprising means for defining at least two personal profiles, wherein each personal profile controls the computer with a different set of action words.

97. The system of claim 75, wherein said wordbase can be shared by more than one user.

98. The system of claim 75, wherein the computer is connected to a server.

99. The system of claim 75, wherein one of said plurality of associated functions includes launching an application program or Internet site.

100. The system of claim 75, wherein said action words include code words and dual words, the system further comprising means for allowing said user to turn said code words and said dual words on and off within said wordbase.