Multimedia word processor

• US 6,802,041 B1
• Filed: 01/20/2000
• Issued: 10/05/2004
• Est. Priority Date: 01/20/1999
• Status: Expired due to Fees

• Current Assignee: Perfectnotes Corporation
• Original Assignee: Perfectnotes Corporation
• Inventors: Rehm, Peter H
• Primary Examiner(s): Hong, Stephen S.
• Assistant Examiner(s): LUDWIG, MATTHEW J
• Application Number:
  US09/488,424
• Time in Patent Office: 1,720 Days
• Field of Search: 715/500.1, 715/512, 715/500, 709/216, 709/328, 709/217, 725/87, 345/723, 345/731, 345/728
• US Class Current: 715/201
• CPC Class Codes: G06F 16/40 of multimedia data, e.g. slideshows comprising image and additional audio data retrieval of still image data G06F16/50; retrieval of audio data G06F16/60; retrieval of video data G06F16/70
• CPC Class Codes: G06F 16/48 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

A word processor optimized for students taking notes automatically opens the correct document for each course according to a class schedule. It makes digital audio or video recordings of classes and establishes various kinds of links between the notes being taken and the point in the recording at the time the notes were taken. A special microphone is optimized for distance and reduced interference. After permitting the word processor to break the silence, any incomplete notes can be completed by using the links to quickly find the portion of the recording relevant to the incomplete notes. The search is improved by providing configurable preplay times based on the type of link and by playing back the recording at increased speed with pitch correction. Transcription of part of the recording is facilitated by playing this part in a series of short overlapping segments and transcribing each segment before playing the next.

Claims

1. A method of taking notes while listening to an audio event and thereafter improving these notes, using a computing device having a memory, a display and at least when needed a microphone and at least when needed an audio output device, said method comprising the steps of:

(a) said computing device making a recording of an audio event as it happens by recording the sounds picked up by said microphone and storing them in said memory as a digital audio recording;

(b) during said making a recording, said computing device inputting a plurality of user notes and storing said user notes in said memory, each user note comprising at least one of the set of (1) keystrokes, (2) pen strokes converted to characters, and (3) pen strokes stored as sketches;

(c) during said making a recording and said inputting a plurality of user notes, said computing device establishing a plurality of audio links in said memory, each audio link linking a user note that was input and the point in said digital audio recording that was substantially the point being recorded as latter-said user note was being input;

(d) said computing device terminating said step of making a recording;

(e) said computing device providing random access to said user notes and said audio links by said computing device inputting navigation commands to selectively display any portion of said user notes on said display;

(f) after said terminating, said computing device receiving at least one play command, each said play command selecting at least one of said audio links;

(g) said computing device obtaining a starting point from an audio link by subtracting a predetermined substantial preplay time from said audio link'"'"'s point in said digital audio recording;

(h) responsive to said play command, said computing device playing back said digital audio recording to said audio output device starting from the starting point indicated by the audio link that was selected by said play command;

(i) after receiving at least one said play command, said computing device inputting editing commands for editing said user notes proximate to the audio link that was selected by latter-said play command;

(j) during said computing device playing back said digital audio recording, said computing device increasing the speed of playback to a preconfigured amount substantially faster than normal speed of playback.

2. The method of claim 1 additionally comprising the step of said computing device correcting the pitch of playback by an amount sufficient to compensate for said increasing speed of playback.

3. The method of claim 1 wherein said preconfigured amount is in the range of 125% to 200%, inclusive, of normal speed of playback.

4. The method of claim 1 wherein said computing device includes a silent mode and said method additionally comprises the steps of:

at the commencement of said making a recording, said computing device entering said silent mode, said silent mode persisting past the termination of said step of making a recording;

responsive to said play command, said computing device first checking whether said computing device is in said silent mode and if said computing device is in said silent mode then said computing device refusing to play back the recording and if said computing device is not in said silent mode then said computing device proceeding to play back the recording;

said computing device inputting permission to break the silence and responsive to said permission, said computing device leaving said silent mode.

5. The method of claim 1 additionally comprising the steps of during said computing device playing back said digital audio recording and during said computing device inputting editing command for editing said user notes, said computing device inputting additional user notes and establishing additional audio links to link said additional user notes with said digital audio recording, each additional audio link linking a user note that was input with the point in said digital audio recording that was substantially the point being played back as latter-said user note was being input.

6. The method of claim 1 additionally comprising the steps of after said step of said computing device inputting editing commands, said computing device inputting an optional command to delete said digital audio recording and responsive to said optional command to delete, said computer deleting said digital audio recording and also deleting all of said audio links that link to a point in said digital audio recording, and retaining said user notes.

7. The method of claim 1 additionally comprising the steps of:

said computing device establishing a timer with a predetermined expiration time,

responsive to the inputting of every user note, said computing device restarting said timer,

responsive to each expiration of said timer, said computing device inserting a symbol in said user notes and establishing a periodic audio link that links said symbol and the point in said digital audio recording that was being recorded when said timer expired, and said computer restarting said timer;

said computing device inputting commands to hide and show the symbols and responsive to said commands, said computing device hiding or showing the symbols on said display;

whereby a series of one or more symbols in the user notes indicates a period of idleness during which time no user notes were input and said computing device preserves convenient access to said digital audio recording that was recorded during said period of idleness.

8. A method of taking notes while listening to an audio event and thereafter improving these notes, using a computing device having a memory, a display, a real time clock and at least when needed a microphone and at least when needed an audio output device, said method comprising the steps of:

(a) said computing device inputting a schedule of events that are to be recorded and storing said schedule in said memory as a list of scheduled events, said schedule comprising a plurality of groups, each group comprising a plurality of similar events;

(b) said computing device associating a plurality of word processor files with said plurality of groups, such that each word processor file is associated with one of said groups;

(c) said computing device checking its real time clock and said list of scheduled events to determine whether one of the scheduled events is happening, and if so, said computing device automatically recording this scheduled event and said computing device opening the word processor file that is associated with the group of which said scheduled event is a member;

(d) said computing device making a recording of said scheduled event as it happens by recording the sounds picked up by said microphone and storing them in said memory as a digital audio recording;

(e) during said making a recording, said computing device inputting a plurality of user notes and storing said user notes in said word processor file that was opened, each user note comprising at least one of the set of (1) keystrokes, (2) pen strokes converted to characters, (3) pen strokes stored as sketches, and (4) pressing a button provided for establishing a timestamp;

(f) during said making a recording and said inputting a plurality of user notes, said computing device establishing a plurality of audio links, each audio link linking a user note that was input and the point in said digital audio recording that was substantially the point being recorded as latter-said user note was being input;

(g) said computing device terminating said step of making a recording;

(h) said computing device providing random access to said user notes and said audio links by said computing device inputting navigation commands to selectively display any portion of said user notes on said display;

(i) after said terminating, said computing device receiving at least one play command, each said play command selecting at least one of said audio links;

(j) said computing device obtaining a starting point from an audio link by subtracting a predetermined substantial preplay time from said audio link'"'"'s point in said digital audio recording;

(k) responsive to said play command, said computing device playing back said digital audio recording to said audio output device starting from the starting point indicated by the audio link that was selected by said play command;

(l) after receiving at least one said play command, said computing device inputting editing commands for editing said user notes proximate to the audio link that was selected by latter-said play command.

9. The method of claim 8 additionally comprising the step of, when said computer device checks its real time clock, initiating recording starting a few minutes prior to the beginning of a scheduled event that is about to happen.

10. The method of claim 8 additionally comprising the steps of:

said computing device saving said word processor file as a hypertext markup language format file, wherein at least a plurality of said audio links are represented in said hypertext markup language format file as hyperlinks referencing a plurality of audio files;

said computing device saving at least some of said digital audio recording in said plurality of audio files, wherein each audio file of said plurality of audio files contains a copy of a portion of said digital audio recording, and wherein each audio file starts approximately at a point in said digital audio recording that is linked by at least one link of said plurality of links;

whereby said hypertext markup language format file and said audio files are accessible by hypertext markup language browser software and

whereby said hyperlinks each reference an audio file that approximately starts with the portion of said digital audio recording that was pointed to in the audio link from which said hyperlink arose.

11. The method of claim 8 additionally comprising the steps of:

said computing device saving said word processor file as an enhanced hypertext markup language format file, wherein at least a plurality of said links are represented in said enhanced hypertext markup language format file as hyperlinks referencing at least one audio file and wherein each said hyperlink further references a point within said at least one audio file, each reference substantially corresponding in data content to the audio link from which said reference arose;

said computing device saving at least some of said digital audio recording in said at least one audio file;

whereby said hypertext markup language format file and said at least one multimedia file are accessible by hypertext markup language browser software that is enhanced with the capability of referencing and playing an audio file starting at any specified point within the referenced audio file.

12. A method of both taking and improving notes while listening to an audio event, using a computing device having a memory, a display capable of displaying a plurality of windows, and at least when needed a microphone and at least when needed an audio output device, said method comprising the steps of:

(a) said computing device making a recording of an audio event as it happens by recording the sounds picked up by said microphone and storing them in-said memory as a digital audio recording;

(b) during said making a recording, said computing device inputting a plurality of user notes and storing said user notes in said memory and displaying said user notes in a first window, each user note comprising at least one of the set of (1) keystrokes, (2) pen strokes converted to characters, and (3) pen strokes stored as sketches;

(c) during said making a recording and said inputting a plurality of user notes, said computing device establishing a plurality of audio links in said memory, each audio link linking a user note that was input and the point in said digital audio recording that was substantially the point being recorded as latter-said user note was being input;

(d) said computing device providing random access to said user notes and said audio links by said computing device inputting navigation commands to selectively display any portion of said user notes on said display;

(e) said computing device inputting a command to open a second window and displaying at least a portion of a word processor file, said word processor file including audio links to a digital audio recording in said computing device'"'"'s memory;

(f) said computing device inputting selective play commands in said second window, said selective play commands selecting at least a particular point in the word processor file in said second window that is proximate to an audio link;

(g) said computing device, responsive to said selective play commands, following said link to a particular point in the digital audio recording and initiating playback to an earphone starting approximately from said particular point in the digital audio recording;

(h) said computing device recording audio to the portion of the word processor file displayed in said first window and simultaineously playing audio using the portion of the word processor file displayed in said second window;

(i) said first window, responsive to user input to said first window, establishing audio links according to predetermined rules;

whereby a user can divide his or her attention between taking notes of an event in real time in said first window and reviewing said multimedia data and improving notes in said second window without any risk of missing something important in the real time event.

13. A method of taking notes while listening to an audio event and thereafter improving these notes, using a computing device having a memory, a display and at least when needed a microphone and an audio output device, said method comprising the steps of:

(a) said computing device making a recording of an audio event as it happens by recording the sounds picked up by said microphone and storing them in said memory as a digital audio recording;

(b) during said making a recording, said computing device inputting a plurality of user notes, each user note comprising at least one of the set of (1) keystrokes, (2) pen strokes converted to characters, and (3) pen strokes stored as sketches;

(c) during said making a recording and said inputting a plurality of user notes, said computing device establishing a plurality of audio links, each audio link linking a user note that was input and the point in said digital audio recording that was substantially the point being recorded as latter-said user note was being input;

(d) said computing device terminating said step of making a recording;

(e) providing random access to said user notes and said audio links by said computing device inputting navigation commands to selectively display any portion of said user notes on said display at any time;

(f) after said terminating, said computing device receiving at least one lockstep play-next-segment command;

(g) responsive to each said lockstep play-next-segment command, said computing device playing back a segment of said digital audio recording to said audio output device for a predetermined segment play length time and then stopping play back, said predetermined segment play length time being sufficiently long for there to be some audio to transcribe and sufficiently short to act on it completely with only one listening;

(h) responsive to successive lockstep play-next-segment commands, said computing device slightly overlapping the segments of the digital audio recording played back by a predetermined segment overlap time, said predetermined segment overlap time being substantially shorter than said predetermined segment play length time;

(i) after receiving at least one lockstep play-next-segment command, said computing device inputting editing commands for editing said user notes.

14. The method of claim 13 wherein said predetermined segment play length time is about 3.5 seconds and said predetermined segment overlap time is about 0.7 seconds.

15. The method of claim 13 wherein said step of inputting editing commands additionally comprises the steps of:

after a lockstep play-next-segment command, said computing device inputing additional user, said additional user notes comprising a plurality of ink-printing characters;

responsive to the first ink-printing character input after said lockstep play-next-segment command, said computing device establishing a lockstep audio link in said memory, said lockstep audio link linking said ink-printing character and the point in said digital audio recording that was at the beginning of the segment of said digital audio recording that was last played;

after said establishing a lockstep audio link, said computer inputting a play command that selects said lockstep audio link and responsive to said play command, said computing device playing said digital audio recording starting substantially from the point in said digital audio recording pointed to by said lockstep audio link;

whereby the lockstep audio links tightly define a correspondence between the user notes and points in the digital audio recording.

16. A computer-readable memory storage medium comprising a substrate from which a computer can read program codes and data, said computer having a display and at least when needed a microphone and at least when needed an audio output device, said substrate containing a computer program comprising:

(a) means for directing said computer to make a recording of an audio event as it happens by recording the sounds picked up by said microphone and storing them in said computer'"'"'s memory as a digital audio recording;

(b) means for directing said computer, during said making a recording, to input a plurality of user notes and to store said user notes in said computer'"'"'s memory, each user note comprising at least one of the set of (1) keystrokes, (2) pen strokes converted to characters, and (3) pen strokes stored as sketches;

(c) means for directing said computer, during said making a recording and said inputting a plurality of user notes, to establish a plurality of audio links in said memory, each audio link linking a user note that was input and the point in said digital audio recording that was substantially the point recorded as latter-said user note was input;

(d) means for directing said computer to device terminate said step of making a recording;

(e) means for directing said computer to provide random access to said user notes and said audio links by directing said computer to input-navigation commands to selectively display any portion of said user notes on said display;

(f) means for directing said computer, to receive at least one play command after said terminating, each said play command selecting at least one of said audio links;

(g) means for directing said computer to obtain a starting point from an audio link by subtracting a predetermined substantial preplay time from said audio link'"'"'s point in said digital audio recording;

(h) means for directing said computer to respond to said play command by said computer playing back said digital audio recording to said audio output device starting from the starting point indicated by the audio link that was selected by said play command;

(i) means for directing said computer, after receiving at least one said play command, to input editing commands for editing said user notes proximate to the audio link that was selected by latter-said play command;

(j) means for directing said computer, during said computer playing back said digital audio recording, to increase the speed of playback by a preconfigured amount substantially faster than normal speed of playback.