Method and system for implementing a user interface for a device employing written graphical elements

• US 7,831,933 B2
• Filed: 01/12/2005
• Issued: 11/09/2010
• Est. Priority Date: 03/17/2004
• Status: Active Grant

• Current Assignee: LeapFrog Enterprises, Inc. (VTech Holdings Ltd.)
• Original Assignee: LeapFrog Enterprises, Inc. (VTech Holdings Ltd.)
• Inventors: Marggraff, James, Edgecomb, Tracy L., Chisholm, Alexander
• Primary Examiner(s): Theriault; Steven B
• Application Number:
  US11/034,491
• Publication Number:
  US 20060067577A1
• Time in Patent Office: 2,127 Days
• Field of Search: 715/863, 715/500.1, 715/776, 345/156, 345/173, 345179-180, 434/317
• US Class Current: 715/863
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/36 Matching; Classification

Abstract

A method and system for implementing a user interface for a device through user created graphical elements. The method includes recognizing a graphical element icon created by a user. Once recognized, a function related to the graphical element icon is accessed and an output in accordance with the function is provided. The function is persistently associated with the graphical element icon. Menu selection and navigation is implemented through interaction with the graphic element icon. A listing of options associated with the graphical element icon is audibly rendered. In response to a selection of one of the options, the selected option is invoked.

Claims

1. A pen device method for interpreting user commands, comprising:

said pen device prompting a user to write a first expected graphical element;

said pen device receiving data indicative of a first user created graphical element written on a surface and receiving a position on said surface of said first user created graphical element without recognizing a content of said first user created graphical element;

associating an interactive function with said position, wherein said interactive function is related to said first expected graphical element;

responsive to said pen device interacting with said position, said pen device executing said interactive function within said pen device, wherein said executing causes said pen device to render an output that is unique to said first expected graphical element regardless of said content of said first user created graphical element; and

enabling a subsequent access and execution of the interactive function in response to a subsequent selection of the first user created graphical element.

2. The method of claim 1, wherein the output comprises an audio output related to the interactive function.

3. The method of claim 1, wherein the storing of the association of the interactive function with said position of the first user created graphical element implements a persistent availability of the interactive function, for a predetermined amount of time, via interaction with said position of the first graphical element.

4. The method of claim 1, wherein the first user created graphical element is created by a pen device on the surface.

5. The method of claim 4, wherein the surface comprises a sheet of paper.

6. The method of claim 1, further comprising:

accessing one of a plurality of functions related to the first expected graphical element by interpreting at least one actuation of the first user created graphical element, wherein at least one actuation selects the one of the plurality of functions.

7. The method of claim 6, wherein the at least one actuation comprises recognizing at least one tap of the first user created graphical element.

8. The method of claim 6, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

9. The method of claim 6, wherein the plurality of functions comprises a predetermined menu of options.

10. The method of claim 6, wherein the plurality of functions comprises a plurality of configuration options of an application related to the first expected graphical element.

11. The method of claim 10, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

12. The method of claim 1, further comprising:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical elements.

13. The method of claim 12, further comprising:

recognizing at least one actuation of the first user created graphical element to select a first hierarchical level function;

prompting the creation of a second expected graphical element;

receiving data indicative of a second user created graphical element written on a surface and receiving a position on said surface of said second user created graphical element without recognizing a content of said second user created graphical element;

associating a second hierarchical level function with said position of said second user created graphical element, wherein said second hierarchical level function is related to said second expected graphical element;

recognizing at least one actuation of the second user created graphical element to select said second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with said position of the second graphical element.

14. The method of claim 1, further comprising:

said pen device recognizing a second graphical element on said surface;

said pen device tracking a state of said first user created graphical element to retain a location of said interactive function within a directory associated with said interactive function; and

said pen device tracking a state of said second graphical element to retain a location of a second interactive function within a directory associated with said second interactive function.

15. The method of claim 1 further comprising:

said pen device recognizing a second user created graphical element.

16. The method of claim 1, wherein said prompt is an audio instruction from said pen device.

17. A pen device method for interpreting user commands, comprising:

said pen device prompting a user to write a first expected graphical element;

said pen device receiving data indicative of a first user created graphical element written on a surface and receiving a position on said surface of said first user created graphical element without recognizing a content of said first user created graphical element;

associating an interactive function with said position, wherein said interactive function is related to said first expected graphical element;

responsive to said pen device interacting with said position, said pen device executing said interactive function within said pen device, wherein said executing causes said pen device to render an output that is unique to said first expected graphical element regardless of said content of said first user created graphical element; and

enabling a subsequent access and execution of the interactive function in response to a subsequent selection of said first user created graphical element by storing an association of the interactive function with said position of said first user created graphical element icon written on said surface, wherein the storing of the association implements a persistent availability of the interactive function for a predetermined amount of time, via interaction with the first user created graphical element.

18. The method of claim 17, wherein the first user created graphical element is created by a pen device on the surface.

19. The method of claim 18, wherein the surface comprises a sheet of paper.

20. The method of claim 17, further comprising:

accessing one of a plurality of functions related to the first expected graphical element by interpreting at least one actuation of the first user created graphical element, wherein at least one actuation selects the one of the plurality of functions.

21. The method of claim 20, wherein the at least one actuation comprises recognizing at least one tap of the first user created graphical element.

22. The method of claim 20, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

23. The method of claim 20, wherein the plurality of functions comprises a predetermined menu of options including one or more applications.

24. The method of claim 20, wherein the plurality of functions comprises a plurality of configuration options of an application related to the first expected graphical element.

25. The method of claim 24, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

26. The method of claim 17, further comprising:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical elements.

27. The method of claim 17, further comprising:

providing at least one audio tip related to the interactive function to indicate an operating manner of the interactive function.

28. The method of claim 26, further comprising:

recognizing at least one actuation of the first user created graphical element to select a first hierarchical level function;

prompting the creation of a second expected graphical element;

receiving data indicative of a second graphical element and receiving a position on said surface thereof without recognizing a content of said second graphical element;

associating a second hierarchical level function with said position of said second graphical element, wherein said second hierarchical level function is related to said second expected graphical element;

recognizing at least one actuation of the second graphical element to select said second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with said position of the second graphical element.

29. A pen device comprising:

an optical sensor for detecting images from a surface;

a processor coupled to the sensor; and

a memory coupled to the processor, the memory storing instructions that when executed cause the pen device to implement a method for interpreting user commands, comprising;

said pen device prompting a user to write a first expected graphical element;

receiving data indicative of a first user created graphical element written on the surface and receiving a position on said surface of said user created graphical element without recognizing a content of said first user created graphical element;

associating an interactive function with said position, wherein said interactive function is related to said first expected graphical element;

responsive to said pen device interacting with said position, executing said interactive function within said pen device;

responsive to said executing, rendering an output that is unique to said first expected graphical element regardless of said content of said first user created graphical element; and

enabling a subsequent access and execution of the interactive function in response to a subsequent selection of the first user created graphical element.

30. The device of claim 29, wherein the output comprises an audio output related to the interactive function.

31. The device of claim 29, wherein the association of the interactive function with the position of said first user created graphical element implements a persistent availability of the interactive function, for a predetermined amount of time, via interaction with the first user created graphical element.

32. The device of claim 29, further comprising a writing instrument and wherein the graphical element is created by the writing instrument on the surface.

33. The device of claim 32 wherein the surface comprises a sheet of paper.

34. The device of claim 29 wherein the method further comprises:

accessing one of a plurality of functions related to the first expected graphical element by interpreting at least one actuation of the first user created graphical element, wherein the at least one actuation selects the one of the plurality of functions.

35. The device of claim 34, wherein the at least one actuation comprises recognizing at least one tap of the first user created graphical element.

36. The device of claim 34, wherein the method further comprises:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected.

37. The device of claim 34, wherein the plurality of functions comprises a predetermined menu of options.

38. The device of claim 34, wherein the plurality of functions comprises a plurality of configuration options of an application related to the first expected graphical element.

39. The device of claim 38, wherein at least one of the plurality of configuration options comprises a default configuration of the application.

40. The device of claim 29, wherein the method further comprises:

implementing a hierarchy of functions; and

providing access to the hierarchy of functions via a corresponding hierarchy of graphical elements.

41. The device of claim 40, wherein the method further comprises:

recognizing at least one actuation of the first user created graphical element to select a first hierarchical level function;

prompting the creation of a second expected graphical element;

receiving data indicative of a second graphical element and receiving a position on said surface thereof without recognizing a content of said second graphical element;

associating a second hierarchical level function with said position of said second graphical element, wherein said second hierarchical level function is related to said second expected graphical element;

recognizing at least one actuation of the second graphical element to select said second hierarchical level function;

providing an audio output related to the second hierarchical level function; and

associating the second hierarchical level function with the position of said second graphical element.

42. The device of claim 29, wherein the method further comprises:

providing at least one audio tip related to the function to indicate an operating manner of the function.

43. A method of interacting with a pen based computer system, said method comprising:

said pen recognizing selection of a first graphical element on a writable surface without recognizing a content of said first graphical element, said selection performed using a writing instrument of said pen based computer system;

said pen in response to said selection, executing a first interactive function associated with a first expected graphical element at a position of said first graphical element regardless of said content of said first graphical element;

said pen in response to said executing said first interactive function, audibly rendering a listing of first options associated with said first expected graphical element, wherein said first options are operable to be invoked by said pen based computer system, wherein said audible rendering is unique to said first interactive function;

said pen in response to a selection of one of said first options, invoking said one of said first options; and

said pen enabling a subsequent access and execution of said first interactive function in response to a subsequent selection of said first graphical element.

44. A method as described in claim 43 wherein said first options comprise at least one application to be invoked.

45. A method as described in claim 43 wherein said one of said first options is an application program resident on said pen based computer system.

46. A method as described in claim 43 wherein said audibly rendering said listing of said first options comprises audibly rendering, one at a time, each of said first options in a round-robin fashion, in response to selections of said first graphical element by said writing instrument.

47. A method as described in claim 46 further comprising identifying a selection of said one of said first options in response to said writing instrument selecting a portion of said first graphical element after said one of said first options is audibly rendered.

48. A method as described in claim 47 wherein said portion of said first graphical element is a symbol of a check mark.

49. A method as described in claim 43 wherein said first graphical element is user written on said surface and further comprising automatically identifying said position of said first graphical element and wherein said automatically identifying said first graphical element is performed using a processor of said pen based computer system.

50. A method as described in claim 43 wherein said first graphical element is pre-printed on said surface.

51. A method as described in claim 43 wherein said first expected graphical element is a menu item and wherein said first options are submenu items within a hierarchy of options operable to be invoked by said pen based computer system.

52. A method as described in claim 43 wherein said first options comprise an option having an associated second expected graphical element and further comprising:

recognizing selection of a second graphical element on said writable surface without recognizing a content of said second graphical element, said selection performed using said writing instrument of said pen based computer system;

in response to said selection, executing a second interactive function related to said second expected graphical element;

said pen in response to said executing said second interactive function, audibly rendering a listing of second options associated with said second expected graphical element, wherein said second options are operable to be invoked by said pen based computer system, wherein said audible rendering is unique to said second interactive function; and

in response to a selection of one of said second options, invoking said one of said second options.

53. A method as described in claim 52 wherein said second options comprise at least one application to be invoked.

54. A method as described in claim 52 wherein said one of said second options is an application program resident on said pen based computer system.

55. A method as described in claim 52 wherein said audibly rendering said listing of said second options comprises audibly rendering, one at a time, each of said second options in a round-robin fashion, in response to selections of said second graphical element by said writing instrument.

56. A method as described in claim 55 further comprising identifying selection of said one of said second options by responding to said writing instrument selecting a portion of said second graphical element after said one of said second options is audibly rendered.

57. A method as described in claim 52 wherein said second graphical element is user written on said surface and further comprising automatically identifying a position of said second graphical element, and wherein said automatically identifying said position of said second graphical element is performed using a processor of said pen based computer system.

58. A pen based computer system comprising:

a writing instrument;

a speaker;

a memory unit coupled to a bus;

an optical sensor coupled to said bus; and

a processor coupled to said bus and comprising instructions that when executed implement a method of interacting with a user comprising;

said pen recognizing selection of a first graphical element on a writable surface without recognizing a content of said first graphical element, said selection performed using said writing instrument;

said pen in response to said selection, executing a first interactive function associated with a first expected graphical element at a position of said first graphical element regardless of said content of said first graphical element;

said pen in response to said executing said first interactive function, audibly rendering, via said speaker, a listing of first options associated with said first expected graphical element wherein said first options are operable to be invoked by said pen based computer system, wherein said audible rendering is unique to said first interactive function;

said pen in response to a selection of one of said first options, invoking said one of said first options; and

said pen enabling a subsequent access and execution of said first interactive function in response to a subsequent selection of said first graphical element.

59. A system as described in claim 58 wherein said one of said first options is an application program resident on said pen based computer system.

60. A system as described in claim 58 wherein said audibly rendering said listing of said first options comprises audibly rendering, one at a time, each of said first options in a round-robin fashion, in response to selections of said first graphical element by said writing instrument.

61. A system as described in claim 60 wherein said method further comprises identifying selection of said one of said first options in response to said writing instrument selecting a portion of said first graphical element after said one of said first options is audibly rendered.

62. A system as described in claim 61 wherein said portion of said first graphical element is a symbol of a check mark.

63. A system as described in claim 58, wherein said first graphical element is user written on said surface, and wherein said method further comprises automatically identifying said position of said first graphical element, and wherein said automatically identifying said position of said first graphical element is performed using said processor of said pen based computer system.

64. A system as described in claim 58, wherein said first graphical element is a menu item, and wherein said first options are submenu items within a hierarchy of options operable to be invoked by said pen based computer system.

65. A system as described in claim 58 wherein said first options comprise an option having an associated second expected graphical element and wherein said method further comprises:

recognizing selection of a second graphical element on said writable surface without recognizing a content of said second graphical element, said selection performed using said writing instrument;

in response to said selection, executing a second interactive function related to said second expected graphical item;

in response to said executing said second interactive function, audibly rendering, via said speaker, a listing of second options associated with said second expected graphical item, wherein said second options are operable to be invoked by said pen based computer system, wherein said audible rendering is unique to said second interactive function; and

in response to a selection of one of said second options, invoking said one of said second options.

66. A system as described in claim 65 wherein said one of said second options is an application program resident on said pen based computer system.

67. A system as described in claim 65 wherein said audibly rendering said listing of said second options comprises audibly rendering, one at a time, each of said second options in a round-robin fashion, in response to selections of said second graphical element by said writing instrument.

68. A system as described in claim 67 wherein said method further comprises identifying a selection of said one of said second options by responding to said writing instrument selecting a portion of said second graphical element after said one of said second options is audibly rendered.

69. A system as described in claim 65 wherein said second graphical item is user written on said surface, and wherein said method further comprises automatically identifying a position of said second graphical item, and wherein said automatically identifying said position of said second graphical icon is performed using said processor of said pen based computer system.

70. A pen device method for accepting user input to implement a function, comprising:

said pen device prompting a user to write an expected graphical element;

said pen device receiving data indicative of a user created graphical element written on a surface and receiving a position on said surface of said user created graphical element without recognizing a content of said user created graphical element, wherein said position is a user selected position;

associating an interactive function with said position, wherein said interactive function is related to said expected graphical element, and wherein said associating corresponds said interactive function to said user created graphical element, and wherein said associating enables a subsequent access and execution of said interactive function responsive to a subsequent selection of said user created graphical element; and

responsive to said pen device interacting with said position, said pen device executing said interactive function within said pen device, wherein said executing causes said pen device to render an output that is unique to said expected graphical element regardless of said content of said user created graphical element.

71. The method of claim 70, wherein the output comprises an audio output related to the interactive function.

72. The method of claim 70, wherein the output is provided via a second device in communication with the pen device.

73. The method of claim 70, wherein the output comprises a visual output related to the interactive function.

74. The method of claim 73, wherein the visual output is provided via a second device in communication with the pen device.

75. The method of claim 70, wherein the user created graphical element icon is created by a pen device on the surface.

76. The method of claim 75, wherein a graphical element is created by:

recognizing a user written character on the surface; and

recognizing a user written circle enclosing the character.

77. The method of claim 76, further comprising:

initiating an OCR process on the character upon an up-stroke of the pen device after the user written circle.

78. The method of claim 76, further comprising:

recognizing a user written check mark within a predetermined distance to the graphical element; and

associating the check mark with the graphical element.

79. The method of claim 78, wherein the check mark is configured to invoke one of a plurality of functions selected by the graphical element.

80. The method of claim 70, wherein the surface comprises a sheet of paper.

81. The method of claim 70, wherein the surface comprises an electronically active surface.

82. The method of claim 81, wherein the electronically active surface comprises a display of a tablet PC.

83. The method of claim 70, further comprising:

accessing one of a plurality of functions related to the expected graphical element by interpreting at least one actuation of the user created graphical element, wherein the at least one actuation selects the one of the plurality of functions.

84. The method of claim 83, wherein an order of the plurality of functions is user configurable.

85. The method of claim 83, wherein at least one new function can be added to the plurality of functions via a software update or a firmware update.

86. The method of claim 83, further comprising:

providing one of a plurality of audio outputs when the one of the plurality of functions is selected, and wherein the audio output is user configurable.

87. The method of claim 70, further comprising:

said pen device accessing a plurality of functions by recognizing an interaction with a corresponding plurality of respective graphical elements; and

said pen device maintaining a respective state for each of the plurality of functions, the respective state resulting from the interaction.

88. The method of claim 70, further comprising:

said pen device accessing a plurality of instances of a single function by recognizing an interaction with a corresponding plurality of respective graphical elements corresponding to each instance; and

said pen device maintaining a respective independent state for each of the plurality of instances.

89. The method of claim 88, wherein at least one of the plurality of graphical elements is preprinted and at least one of the plurality of graphical elements is user created.

90. The method of claim 70, further comprising:

said pen device accessing a plurality of instances of a single function by recognizing an interaction with a corresponding plurality of respective graphical elements corresponding to each instance; and

said pen device maintaining a coordinated state for each of the plurality of instances.

91. The method of claim 90, wherein at least one of the plurality of graphical elements is preprinted and at least one of the plurality of graphical elements is user created.

92. The method of claim 70, further comprising:

said pen device prompting the creation of a graphical element by providing a prompt related to a label for the graphical element;

said pen device recognizing a termination of the creation of the graphical element; and

said pen device associating the label with the graphical element.

93. The method of claim 92, further comprising:

said pen device associating a function with the graphical element, wherein the function is related to the label.

94. The method of claim 92, wherein the label is associated with the graphical element without performing an OCR (optical character recognition) process on the graphical element.

95. The method of claim 70, further comprising:

said pen device recognizing said position of the created graphical element on the surface by using position determination means within a pen device, and wherein the position determination means is configured to determine the location without requiring position code on the surface.