Correlation of Written Notes to Digital Content

• US 20140122490A1
• Filed: 10/24/2013
• Published: 05/01/2014
• Est. Priority Date: 10/26/2012
• Status: Active Grant

• Current Assignee: Livescribe Incorporated (Anoto Group AB)
• Original Assignee: Livescribe Incorporated (Anoto Group AB)
• Inventors: Black, David Robert, Halle, Brett Reed, Van Schaack, Andrew J.
• Granted Patent: US 9,195,697 B2
• Time in Patent Office: Days
• US Class Current: 707/737
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 3/0317 in co-operation with a patterned surface, e.g. absolute position or relative movement detection for an optical mouse or pen positioned with respect to a coded surface
• CPC Class Codes: G06F 3/0321 by optically sensing the absolute position with respect to a regularly patterned surface forming a passive digitiser, e.g. pen optically detecting position indicative tags printed on a paper sheet constructional details of pen-shaped pointing devices G06F3/03545, G06F3/03542, G06F3/037
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry

Abstract

A system and method receives a plurality of data streams from a smart pen device and a computing device, and indexes the data streams to a synchronized time index. A processor receives a first data stream representing gesture data captured by a smart pen device, and a second data stream representing a sequence of states associated with applications executing on a computing device, such that each state identifies content displayed by a computing device while the gesture data is captured. For example, a state could be a particular page of a digital document displayed by the computing device. After receiving the first and second data streams, the processor indexes the data streams to a synchronized time index, and stores the indexed data streams in a memory.

Claims

1. A computer-implemented method comprising:

receiving, by a processor, a first data stream representing gesture data captured by a gesture capture system of a smart pen device interacting with a writing surface;

receiving, by the processor, a second data stream representing a sequence of states associated with one or more applications executing on a computing device, each state identifying content being displayed by a computing device while the gesture capture system captures the gesture data;

indexing, by the processor, the first and second data streams to a synchronized time index; and

storing the indexed first and second data streams to a memory.

2. The computer-implemented method of claim 1, further comprising:

receiving, by the processor, a third data stream representing audio data captured by an audio capture system;

indexing, by the processor, the third data stream to the synchronized time index; and

storing the indexed third data stream to the memory.

3. The computer-implemented method of claim 1, wherein one state in the sequence of states comprises a particular page of a digital document displayed by the computing device at a given time.

4. The computer-implemented method of claim 1, wherein one state in the sequence of states comprises a particular zoomed in portion of a page of a digital document displayed by the computing device at a given time.

5. The computer-implemented method of claim 1, wherein one state in the sequence of states comprises one or more user inputs stored by the computing device at a given time.

6. The computer-implemented method of claim 1, wherein one state in the sequence of states comprises a playback frame of a video played by the computing device at a given time.

7. The computer-implemented method of claim 1, wherein the sequence of states comprises one or more state changes, and wherein the state changes are temporally correlated to the synchronized time index.

8. The computer-implemented method of claim 1, wherein the indexing comprises temporally synchronizing the first and second data streams based on the time the data streams were generated.

9. The computer-implemented method of claim 1, further comprising:

retrieving at least one of the stored indexed data streams upon a user'"'"'s request for playback; and

replaying a portion of the retrieved data streams, wherein the replaying comprises displaying the gesture data or the sequence of states in correlation with the synchronized time index.

10. The computer-implemented method of claim 9, wherein the replaying further comprises:

identifying a portion of a previously captured gesture based on an interaction between the smart pen and the writing surface;

correlating the identified portion of the previously captured gesture to a particular time point in the time index; and

replaying the identified portion of the previously captured gesture beginning from the particular time point.

11. The computer-implemented method of claim 9, wherein the replaying further comprises:

identifying a portion of a previously captured state based on content displayed by the computing device;

correlating the identified portion of the previously captured state to a particular time point in the time index; and

replaying the identified portion of the previously captured state beginning from the particular time point.

12. The computer-implemented method of claim 1, further comprising:

identifying a portion of the first data stream;

receiving a request to associate the portion of the first data stream with a portion of a digital document; and

storing an association between the portion of the first data stream and the portion of the digital document.

13. The computer-implemented method of claim 12, further comprising:

identifying a portion of a third data stream representing audio data captured by an audio capture system;

receiving a request to associate the portion of the third data stream with a portion of a selected region of the writing surface; and

storing an association between the portion of the third data stream and the selected region of the writing surface.

14. A smart pen computing system comprising:

a non-transitory computer-readable storage medium storing computer program code and integrated within the smart pen device, the computer program code configured to be executed by the processor, the computer program code including instructions for;

receiving, by a processor, a first data stream representing gesture data captured by a gesture capture system of a smart pen device interacting with a writing surface;

receiving, by the processor, a second data stream representing a sequence of states associated with one or more applications executing on a computing device, each state identifying content being displayed by a computing device while the gesture capture system captures the gesture data;

indexing, by the processor, the first and second data streams to a synchronized time index; and

storing the indexed first and second data streams to a memory.

15. The smart pen computing system of claim 14, further comprising:

a smart pen, wherein the processor is integrated within the smart pen; and

a gesture capture system integrated within the smart pen, the gesture capture system coupled to the processor and the gesture capture system for capturing written data.

16. The smart pen computing system of claim 14, the computer program code including further instructions for:

receiving, by the processor, a third data stream representing audio data captured by an audio capture system;

indexing, by the processor, the third data stream to the synchronized time index; and

storing the indexed third data stream to the memory.

17. The smart pen computing system of claim 14, wherein one state in the sequence of states comprises a particular page of a digital document displayed by the computing device at a given time.

18. The smart pen computing system of claim 14, wherein one state in the sequence of states comprises one or more user inputs stored by the computing device at a given time.

19. The smart pen computing system of claim 14, wherein the indexing comprises temporally synchronizing the first and second data streams based on the time the data streams were generated.

20. The smart pen computing system of claim 14, the computer program code including further instructions for:

retrieving at least one of the stored indexed data streams upon a user'"'"'s request for playback; and

replaying a portion of the retrieved data streams, wherein the replaying comprises displaying the gesture data or the sequence of states in correlation with the synchronized time index.

21. The smart pen computing system of claim 14, the computer program code including further instructions for:

identifying a portion of the first data stream;

receiving a request to associate the portion of the first data stream with a portion of a digital document; and

storing an association between the portion of the first data stream and the portion of the digital document.