Cyclical creation, transfer and enhancement of multi-modal information between paper and digital domains

• US 8,416,218 B2
• Filed: 05/29/2008
• Issued: 04/09/2013
• Est. Priority Date: 05/29/2007
• Status: Active Grant

• Current Assignee: Livescribe Incorporated (Anoto Group AB)
• Original Assignee: Livescribe Incorporated (Anoto Group AB)
• Inventors: Marggraff, Jim, Edgecomb, Tracy L.
• Primary Examiner(s): Beck, Alexander S
• Assistant Examiner(s): BOGALE, AMEN W
• Application Number:
  US12/129,193
• Publication Number:
  US 20090021493A1
• Time in Patent Office: 1,776 Days
• Field of Search: 345/179, 382/173, 382/181
• US Class Current: 345/179
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 40/171 by use of digital ink

Abstract

In a pen-based computing system, multi-modal data is transferred between a paper domain and a digital domain. Data initially generated in the paper domain is captured by a smart pen and a digital file including the captured data is generated. For example, a computing system coupled to the smart pen generates a digital file including the captured data. A paper representation of the digital file is subsequently generated. The digital file can subsequently be modified by editing the paper representation of the digital file using the smart pen. Edits to the paper representation of the digital file are captured by the smart pen and converted to the digital domain where they are used to edit the content of the digital file.

Claims

1. A pen-based computing system for transferring data between a paper domain and a digital domain, the system comprising:

a smart pen device interacting with a paper representation, the paper representation containing printed textual data representing textual data of a digital file, the smart pen device configured to;

capture written editing data, the written editing data modifying the printed textual data on the paper representation;

recognize drawing of a delete button on the paper representation;

recognize drawing of an erase box on the paper representation, the erase box separate from the delete button, and the erase box enclosing a subset of the printed textual data; and

responsive to detecting that the delete button has been invoked, exclude the enclosed subset of the printed textual data from data transferred by the smart pen device to a computing device; and

a non-transitory computer-readable storage medium containing computer program code for execution on the computing device, the computer program code containing instructions to;

receive, from the smart pen device, the written editing data captured from the paper domain; and

modify the digital file to generate a modified digital file in accordance with the written editing data.

2. The pen-based computing system of claim 1, wherein excluding the enclosed subset of the printed textual data from data transferred by the smart pen device to the computing device comprises:

deleting the subset of the printed textual data from a memory of the smart pen.

3. The pen-based computing system of claim 1, wherein the digital file further comprises at least one of:

image content, audio content, and video content.

4. The pen-based computing system of claim 1 wherein the smart pen further captures at least one of:

image content, data or sensed motion of the smart pen.

5. The pen-based computing system of claim 1, wherein:

the smart pen device further captures annotation data, the annotation data associated with a subset of the paper representation; and

the computing system receives the annotation data and associates the annotation data with a subset of the digital file corresponding to the subset of the paper representation.

6. The pen-based computing system of claim 1, wherein the smart pen device is further configured to:

capture written annotation data associated with a subset of the paper representation of the digital file; and

associate the written annotation data with a subset of the digital file corresponding to the subset of the paper representation of the digital file.

7. A non-transitory computer-readable storage medium storing computer program code for transferring data between a paper domain and a digital domain, the computer program code when executed by a processor causes the processor to perform steps including:

capturing data written on a first physical document using a smart pen;

transferring the captured data from the smart pen to a computer system, the computer system storing the captured data as a digital document;

identifying a paper representation of the digital document, the paper representation containing printed data corresponding to the captured data;

capturing written editing data written on the paper representation using the smart pen, the written editing data modifying the printed data on the paper representation;

capturing a gesture identifying a section of the printed data on the paper representation;

recording an audio comment related to the identified section of the printed data on the paper representation;

recognizing drawing of a delete button using the smart pen on the paper representation of the digital document;

recognizing drawing of an erase box using the smart pen on the paper representation, the erase box separate from the delete button, and the erase box enclosing a subset of the printed data; and

responsive to invoking the delete button using the smart pen, excluding the enclosed subset of the printed data from data transferred from the smart pen to the computer system;

transferring the captured written editing data from the smart pen to the computer system, the computer system generating a modified digital document in accordance with the captured written editing data; and

transferring the recorded audio comment from the smart pen to the computer system, the computer system storing the recorded audio comment in association with a section of the modified digital document corresponding to the identified section of the printed data on the paper representation.

8. A non-transitory computer-readable storage medium storing computer program code for transferring data between a paper domain and a digital domain, the computer program code when executed by a processor causing the processor to perform steps including:

identifying a paper representation of a digital document, the paper representation containing printed data;

capturing written editing data written on the paper representation using a smart pen, the written editing data modifying the printed data on the paper representation;

recognizing drawing of a delete button using the smart pen on the paper representation;

recognizing drawing of an erase box using the smart pen on the paper representation, the erase box separate from the delete button, and the erase box enclosing a subset of the printed data;

responsive to invoking the delete button using the smart pen, excluding the enclosed subset of the printed data from data transferred by the smart pen to a computer system; and

transferring the captured written editing data from the smart pen to the computer system, the computer system generating a modified digital document in accordance with the captured written editing data.

9. The non-transitory computer-readable storage medium of claim 8, wherein excluding the enclosed subset of the printed data from data transferred by the smart pen to the computer system comprises:

deleting the subset of the printed data from a memory of the smart pen.

10. A pen-based computing system for transferring data between a paper domain and a digital domain, the system comprising:

a smart pen device interacting with a paper representation, the paper representation containing printed textual data representing textual data of a digital file, the smart pen device configured to;

capture written editing data, gesture data, and audio comments from the paper domain, the written editing data modifying the printed textual data on the paper representation, and the gesture data identifying a section of the printed textual data on the paper representation to be associated with each of the audio comments;

recognize drawing of a delete button on the paper representation;

recognize drawing of an erase box on the paper representation, the erase box separate from the delete button, and the erase box enclosing a subset of the printed textual data for deletion; and

responsive to detecting that the delete button has been invoked, exclude the enclosed subset of the printed textual data from data transferred by the smart pen device to a computing device; and

a non-transitory computer-readable storage medium containing computer program code for execution on the computing device, the computer program code containing instructions to;

receive, from the smart pen device, the written editing data, the gesture data, and the audio comments captured from the paper domain;

modify the digital file to generate a modified digital file in accordance with the written editing data; and

store each of the audio comments in association with a section of the modified digital file corresponding to the section of the printed textual data identified by the corresponding gesture data.