MULTI-MODAL SMARTPEN COMPUTING SYSTEM
First Claim
Patent Images
1. A smart pen device for interacting with a user using a plurality of modalities, the system comprising:
- a processor;
a gesture capture system coupled to the processor and configured to capture handwriting data;
one or more microphones coupled to the processor and configured to capture audio data;
an onboard memory coupled to the processor and configured to store the captured handwriting data or captured audio data responsive to the processor identifying the input;
a display system coupled to the processor and configured to output display data stored in the onboard memory responsive to the processor identifying an output associated with the captured handwriting data or with the captured audio data;
an audio output system coupled to the processor and configured to play stored audio data responsive to the processor identifying an output associated with the captured handwriting data or with the captured audio data; and
computer program code stored on a memory and configured to be executed by the processor, the computer program code including instructions for identifying an input associated with the captured handwriting data or with the captured audio data, instructions for providing an output as display data to the display system or audio data to the audio output system.
4 Assignments
0 Petitions
Accused Products
Abstract
In a pen-based computing system, a smart pen allows user interaction with the pen-based computing system using multiple modalities. Generally, the modalities are categorized as input (or command) modalities and output (or feedback) modalities. Examples of input modalities for the smart pen include writing with the smart pen to provide written input and/or speaking or otherwise providing sound to give audio input to the smart pen. Examples of output modalities for the smart pen include reading visual information displayed by the system, receiving haptic feedback and/or listening to sound played by the system
-
Citations
19 Claims
-
1. A smart pen device for interacting with a user using a plurality of modalities, the system comprising:
-
a processor; a gesture capture system coupled to the processor and configured to capture handwriting data; one or more microphones coupled to the processor and configured to capture audio data; an onboard memory coupled to the processor and configured to store the captured handwriting data or captured audio data responsive to the processor identifying the input; a display system coupled to the processor and configured to output display data stored in the onboard memory responsive to the processor identifying an output associated with the captured handwriting data or with the captured audio data; an audio output system coupled to the processor and configured to play stored audio data responsive to the processor identifying an output associated with the captured handwriting data or with the captured audio data; and computer program code stored on a memory and configured to be executed by the processor, the computer program code including instructions for identifying an input associated with the captured handwriting data or with the captured audio data, instructions for providing an output as display data to the display system or audio data to the audio output system. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9)
-
-
10. A method for a pen-based computing system interacting with a user using a plurality of modalities, the method comprising:
-
receiving an interaction from the user; associating a command with the interaction; responsive to associating a voice capture command with the interaction, recording audio data and storing the recorded audio data in a smart pen; responsive to associating a text capture command with the interaction, capturing handwriting data or text data proximate to the smart pen and storing the captured handwriting data or text data in the smart pen; responsive to associating an audio playback command with the interaction, aurally presenting data to the user; and responsive to associating a visual playback command with the interaction, visually presenting data to the user. - View Dependent Claims (11, 12, 13, 14, 15)
-
-
16. A pen-based computing system for interacting with a user using a plurality of modalities, the system comprising:
-
a smart pen device configured to receive one or more user interactions; computer program code stored on a memory and configured to be executed by a processor coupled to the smart pen device, the computer program code including instructions for identifying an input associated with a subset of the captured handwriting or with a subset of the captured audio data; a storage device coupled to the processor and the smart pen, the storage device the onboard memory storing data associated with the smart pen device responsive to the processor identifying the input associated with a user interaction with the smart pen device; and an output module coupled to the processor and to the storage device, the output module presenting stored data to the user responsive to the processor associating an output with the subset of the one or more user interactions with the smart pen device. - View Dependent Claims (17, 18, 19)
-
Specification