Method and system for implementing a user interface for a device employing written graphical elements

• US 20060067577A1
• Filed: 01/12/2005
• Published: 03/30/2006
• Est. Priority Date: 03/17/2004
• Status: Active Grant

• Current Assignee: LeapFrog Enterprises, Inc. (VTech Holdings Ltd.)
• Original Assignee: LeapFrog Enterprises, Inc. (VTech Holdings Ltd.)
• Inventors: Marggraff, James, Edgecomb, Tracy L., Chisholm, Alexander
• Granted Patent: US 7,831,933 B2
• Time in Patent Office: Days
• US Class Current: 382/187
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/36 Matching; Classification

Abstract

A method and system for implementing a user interface for a device through user created graphical elements. The method includes recognizing a graphical element icon created by a user. Once recognized, a function related to the graphical element icon is accessed and an output in accordance with the function is provided. The function is persistently associated with the graphical element icon. Menu selection and navigation is implemented through interaction with the graphic element icon. A listing of options associated with the graphical element icon is audibly rendered. In response to a selection of one of the options, the selected option is invoked.

Claims

15. A method for interpreting user commands, comprising:

recognizing a created graphical element icon on a surface;

accessing a function related to the graphical element icon;

providing an audio output in accordance with the function; and

enabling a subsequent access of the function by storing an association of the function with the graphical element icon, wherein the storing of the association implements a persistent availability of the function for a predetermined amount of time, via interaction with the graphical element icon.

1. A method for interpreting user commands, comprising:

recognizing a created graphical element icon on a surface;

accessing a function related to the graphical element icon;

providing an output in accordance with the function; and

associating the function with the graphical element icon.

2. The method of claim 1, wherein the output comprises an audio output related to the function.

3. The method of claim 1, further comprising:

enabling a subsequent access of the function in response to a subsequent selection of the graphical element icon by storing the association of the function with the graphical element icon.

4. The method of claim 3, wherein the storing of the association of the function with the graphical element icon implements a persistent availability of the function, for a predetermined amount of time, via interaction with the graphical element icon.

5. The method of claim 1, wherein the graphical element icon is created by a pen device on the surface.

6. The method of claim 5, wherein the surface comprises a sheet of paper.

7. The method of claim 1, further comprising:

accessing one of a plurality of functions related to the graphical element icon by interpreting at least one actuation of the graphical element icon, wherein the at least one actuation selects the one of the plurality of functions.

8. The method of claim 7, wherein the at least one actuation comprises recognizing at least one tap of the graphical element icon.

9. The method of claim 7, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

10. The method of claim 7, wherein the plurality of functions comprises a predetermined menu of options.

11. The method of claim 7, wherein the plurality of functions comprises a plurality of configuration options of an application related to the graphical element icon.

12. The method of claim 11, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

13. The method of claim 1, further comprising:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical element icons.

14. The method of claim 13, further comprising:

recognizing at least one actuation of the graphical element icon to select a first hierarchical level function;

prompting the creation of a second graphical element icon;

recognizing at least one actuation of the second graphical element icon to select a second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with the second graphical element icon.

16. The method of claim 15, wherein the graphical element icon is created by a pen device on the surface.

17. The method of claim 16, wherein the surface comprises a sheet of paper.

18. The method of claim 15, further comprising:

accessing one of a plurality of functions related to the graphical element icon by interpreting at least one actuation of the graphical element icon, wherein the at least one actuation selects the one of the plurality of functions.

19. The method of claim 18, wherein the at least one actuation comprises recognizing at least one tap of the graphical element icon.

20. The method of claim 18, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

21. The method of claim 18, wherein the plurality of functions comprises a predetermined menu of options including one or more applications.

22. The method of claim 18, wherein the plurality of functions comprises a plurality of configuration options of an application related to the graphical element icon.

23. The method of claim 22, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

24. The method of claim 15, further comprising:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical element icons.

25. The method of claim 15, further comprising:

providing at least one audio tip related to the function to indicate an operating manner of the function.

25-1. The method of claim 24, further comprising:

recognizing at least one actuation of the graphical element icon to select a first hierarchical level function;

prompting the creation of a second graphical element icon;

recognizing at least one actuation of the second graphical element icon to select a second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with the second graphical element icon.

26. A device comprising:

an optical sensor for detecting images from a surface;

a processor coupled to the sensor; and

a memory coupled to the processor, the memory storing instructions that when executed cause the device to implement a method for interpreting user commands, comprising;

recognizing a created graphical element icon on the surface;

accessing a function related to the graphical element icon;

providing an output in accordance with the function; and

associating the function with the graphical element icon.

27. The device of claim 26, wherein the output comprises an audio output related to the function.

28. The device of claim 26, further comprising:

enabling a subsequent access of the function by storing the association of the function with the graphical element icon.

29. The device of claim 28, wherein the storing of the association of the function with the graphical element icon implements a persistent availability of the function, for a predetermined amount of time, via interaction with the graphical element icon.

30. The device of claim 26, further comprising a writing instrument and wherein the graphical element icon is created by the writing instrument on the surface.

31. The device of claim 30, wherein the surface comprises a sheet of paper.

32. The device of claim 26, wherein the method further comprises:

accessing one of a plurality of functions related to the graphical element icon by interpreting at least one actuation of the graphical element icon, wherein the at least one actuation selects the one of the plurality of functions.

33. The device of claim 32, wherein the at least one actuation comprises recognizing at least one tap of the graphical element icon.

34. The device of claim 32, wherein the method further comprises:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

35. The device of claim 32, wherein the plurality of functions comprises a predetermined menu of options.

36. The device of claim 32, wherein the plurality of functions comprises a plurality of configuration options of an application related to the graphical element icon.

37. The device of claim 36, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

38. The device of claim 26, wherein the method further comprises:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical element icons.

39. The device of claim 38, wherein the method further comprises:

recognizing at least one actuation of the graphical element icon to select a first hierarchical level function;

prompting the creation of a second graphical element icon;

recognizing at least one actuation of the second graphical element icon to select a second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with the second graphical element icon.

40. The device of claim 26, wherein the method further comprises:

providing at least one audio tip related to the function to indicate an operating manner of the function.

41. A method of interacting with a pen based computer system, said method comprising:

recognizing selection of a first graphical icon on a writable surface, said selection performed using a writing instrument of said pen based computer system;

in response to said selection, audibly rendering a listing of first options associated with said first graphical icon wherein said first options are operable to be invoked by said pen based computer system; and

in response to a selection of one of said first options, invoking said one of said first options.

42. A method as described in claim 41 wherein said first options comprise at least one application to be invoked.

43. A method as described in claim 41 wherein said one of said first options is an application program resident on said pen based computer system.

44. A method as described in claim 41 wherein said audibly rendering said listing of said first options comprises audibly rendering, one at a time, each of said first options in a round-robin fashion, in response to selections of said first graphical icon by said writing instrument.

45. A method as described in claim 44 further comprising identifying a selection of said one of said first options in response to said writing instrument selecting a portion of said first graphical icon after said one of said first options is audibly rendered.

46. A method as described in claim 45 wherein said portion of said first graphical icon is a symbol of a check mark.

47. A method as described in claim 41 wherein said first graphical icon is user written on said surface and further comprising automatically identifying said first graphical icon and wherein said automatically identifying said first graphical icon is performed using a processor of said pen based computer system.

48. A method as described in claim 41 wherein said first graphical icon is pre-printed on said surface.

49. A method as described in claim 41 wherein said first graphical icon is a menu item and wherein said first options are submenu items within a hierarchy of options operable to be invoked by said pen based computer system.

50. A method as described in claim 41 wherein said first options comprise an option having an associated second graphical icon and further comprising:

recognizing selection of said second graphical icon on said writable surface, said selection performed using said writing instrument of said pen based computer system;

in response to said selection, audibly rendering a listing of second options associated with said second graphical icon wherein said second options are operable to be invoked by said pen based computer system; and

in response to a selection of one of said second options, invoking said one of said second options.

51. A method as described in claim 50 wherein said second options comprise at least one application to be invoked.

52. A method as described in claim 50 wherein said one of said second options is an application program resident on said pen based computer system.

53. A method as described in claim 50 wherein said audibly rendering said listing of said second options comprises audibly rendering, one at a time, each of said second options in a round-robin fashion, in response to selections of said second graphical icon by said writing instrument.

54. A method as described in claim 53 further comprising identifying selection of said one of said second options by responding to said writing instrument selecting a portion of said second graphical icon after said one of said second options is audibly rendered.

55. A method as described in claim 50 wherein said second graphical icon is user written on said surface and further comprising automatically identifying said second graphical icon and wherein said automatically identifying said second graphical icon is performed using a processor of said pen based computer system.

56. A pen based computer system comprising:

a writing instrument;

a speaker;

a memory unit coupled to a bus;

an optical sensor coupled to said bus; and

a processor coupled to said bus and comprising instructions that when executed implement a method of interacting with a user comprising;

recognizing selection of a first graphical icon on a writable surface, said selection performed using said writing instrument;

in response to said selection, audibly rendering, via said speaker, a listing of first options associated with said first graphical icon wherein said first options are operable to be invoked by said pen based computer system; and

in response to a selection of one of said first options, invoking said one of said first options.

57. A system as described in claim 56 wherein said one of said first options is an application program resident on said pen based computer system.

58. A system as described in claim 56 wherein said audibly rendering said listing of said first options comprises audibly rendering, one at a time, each of said first options in a round-robin fashion, in response to selections of said first graphical icon by said writing instrument.

59. A system as described in claim 58 wherein said method further comprises identifying selection of said one of said first options in response to said writing instrument selecting a portion of said first graphical icon after said one of said first options is audibly rendered.

60. A system as described in claim 59 wherein said portion of said first graphical icon is a symbol of a check mark.

61. A system as described in claim 56 wherein said first graphical icon is user written on said surface and wherein said method further comprises automatically identifying said first graphical icon and wherein said automatically identifying said first graphical icon is performed using said processor of said pen based computer system.

62. A system as described in claim 56 wherein said first graphical icon is a menu item and wherein said first options are submenu items within a hierarchy of options operable to be invoked by said pen based computer system.

63. A system as described in claim 56 wherein said first options comprise an option having an associated second graphical icon and wherein said method further comprises:

recognizing selection of said second graphical icon on said writable surface, said selection performed using said writing instrument;

in response to said selection, audibly rendering, via said speaker, a listing of second options associated with said second graphical icon wherein said second options are operable to be invoked by said pen based computer system; and

in response to a selection of one of said second options, invoking said one of said second options.

64. A system as described in claim 63 wherein said one of said second options is an application program resident on said pen based computer system.

65. A system as described in claim 63 wherein said audibly rendering said listing of said second options comprises audibly rendering, one at a time, each of said second options in a round-robin fashion, in response to selections of said second graphical icon by said writing instrument.

66. A system as described in claim 65 wherein said method further comprises identifying a selection of said one of said second options by responding to said writing instrument selecting a portion of said second graphical icon after said one of said second options is audibly rendered.

67. A system as described in claim 63 wherein said second graphical icon is user written on said surface and wherein said method further comprises automatically identifying said second graphical icon and wherein said automatically identifying said second graphical icon is performed using said processor of said pen based computer system.

68. A method for accepting user input to implement a function, comprising:

recognizing a created graphical element icon on a surface;

accessing a function related to the graphical element icon;

providing an output in accordance with the function; and

associating the function with the graphical element icon.

69. The method of claim 68, wherein the output comprises an audio output related to the function.

70. The method of claim 69, wherein the audio output is provided via a pen device.

71. The method of claim 70, wherein the audio output is provided via a second device in communication with the pen device.

72. The method of claim 68, wherein the output comprises a visual output related to the function.

73. The method of claim 72, wherein the visual output is provided via a pen device.

74. The method of claim 73, wherein the visual output is provided via a second device in communication with the pen device.

75. The method of claim 68, wherein the graphical element icon is created by a pen device on the surface.

78. The method of claim 75, wherein the graphical element icon is created by:

recognizing a user written character on the surface; and

recognizing a user written circle enclosing the character.

79. The method of claim 78, further comprising:

initiating an OCR process on the character upon an up-stroke of the pen device after the user written circle.

80. The method of claim 78, further comprising:

recognizing a user written check mark within a predetermined distance to the graphical element icon; and

associating the check mark with the graphical element icon.

81. The method of claim 80, wherein the check mark is configured to invoke one of a plurality of functions selected by the graphical element icon.

82. The method of claim 68, wherein the surface comprises a sheet of paper.

83. The method of claim 68, wherein the surface comprises an electronically active surface.

84. The method of claim 83, wherein the electronically active surface comprises a display of a tablet PC.

85. The method of claim 68, further comprising:

accessing one of a plurality of functions related to the graphical element icon by interpreting at least one actuation of the graphical element icon, wherein the at least one actuation selects the one of the plurality of functions.

86. The method of claim 85, wherein in order of the plurality of functions is user configurable.

87. The method of claim 85, wherein at least one new function can be added to the plurality of functions via a software update or a firmware update.

88. The method of claim 85, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected, and wherein the audio output is user configurable.

89. The method of claim 68, further comprising:

accessing a plurality of functions by recognizing an interaction with a corresponding plurality of respective graphical element icons; and

maintaining a respective state for each of the plurality of functions, the respective state resulting from the interaction.

90. The method of claim 68, further comprising:

accessing a plurality of instances of a single function by recognizing an interaction with a corresponding plurality of respective graphical element icons corresponding to each instance; and

maintaining a respective independent state for each of the plurality of instances.

91. The method of claim 90, wherein at least one of the plurality of graphical element icons is preprinted and at least one of the plurality of graphical element icons is user created.

92. The method of claim 68, further comprising:

accessing a plurality of instances of a single function by recognizing an interaction with a corresponding plurality of respective graphical element icons corresponding to each instance; and

maintaining a coordinated state for each of the plurality of instances.

93. The method of claim 92, wherein at least one of the plurality of graphical element icons is preprinted and at least one of the plurality of graphical element icons is user created.

94. The method of claim 68, further comprising:

prompting the creation of a graphical element by providing a prompt related to a label for the graphical element;

recognizing a termination of the creation of the graphical element; and

associating the label with the graphical element.

95. The method of claim 94, further comprising:

associating a function with the graphical element, wherein the function is related to the label.

96. The method of claim 94, wherein the label is associated with the graphical element without performing an OCR (optical character recognition) process on the graphical element.

97. The method of claim 68, further comprising:

recognizing the created graphical element icon on the surface by using position determination means within a pen device to determine a location of the created graphical element icon on the surface, and wherein the position determination means is configured to determine the location without requiring position code on the surface.