Enhanced audio recording for smart pen computing systems

• US 8,284,951 B2
• Filed: 05/29/2008
• Issued: 10/09/2012
• Est. Priority Date: 05/29/2007
• Status: Active Grant

• Current Assignee: Livescribe Incorporated (Anoto Group AB)
• Original Assignee: Livescribe Incorporated (Anoto Group AB)
• Inventors: Van Schaack, Andy, Canova, Frank
• Primary Examiner(s): BARRERA, RAMON M
• Application Number:
  US12/129,575
• Publication Number:
  US 20090022332A1
• Time in Patent Office: 1,594 Days
• Field of Search: 381/26, 381/91, 381/334, 345/179
• US Class Current: 381/91
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2420/07 Applications of wireless loudspeakers or wireless microphones
• CPC Class Codes: H04R 2430/20 Processing of the output signals of the acoustic transducers of an array for obtaining a desired directivity characteristic H04R2203/12 takes precedence
• CPC Class Codes: H04R 3/005 for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04R 5/027 Spatial or constructional arrangements of microphones, e.g. in dummy heads
• CPC Class Codes: H04R 5/033 Headphones for stereophonic communication details thereof, e.g. relating to batteries, cables or control elements H04R1/10

Abstract

In a pen-based computing system, a pair of microphones captures audio concurrently with a smart pen device capturing handwriting gestures. The audio captured by the pair of microphones is processed to produce an enhanced audio recording. Noise cancellation techniques may be applied to reduce noise generated by the smart pen device interacting with the writing surface to produce a higher quality audio recording. In addition, beam steering techniques may be applied to adjust the gain of portions of the audio signal from different directions and allow the user to focus between multiple audio sources.

Claims

1. A method for recording audio in a smart pen computing system, the method comprising:

capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

capturing handwriting gestures by a smart pen device of the smart pen computing system concurrently with capturing the audio;

processing the audio captured by the first and second microphones, wherein processing the audio comprises canceling noise generated from interaction between the smart pen device and a writing surface; and

synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

2. The method of claim 1, further comprising:

transferring the captured audio to an external computing system; and

processing the audio on the external computing system.

3. A method for recording audio in a smart pen computing system, the method comprising:

capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

capturing handwriting gestures by a smart pen device of the smart pen computing system concurrently with capturing the audio;

processing the audio captured by the first and second microphones, wherein processing the audio comprises;

detecting movement of the first and second microphone during the capturing of the audio; and

processing the audio to reduce aurally perceivable distortion caused by the movement; and

synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

4. A method for recording audio in a smart pen computing system, the method comprising:

capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

capturing handwriting gestures by a smart pen device of the smart pen computing system concurrently with capturing the audio;

processing the audio captured by the first and second microphones, wherein the audio is processed on the smart pen device; and

synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

5. The method of claim 4, wherein processing the audio comprises beam steering to adjust a gain of a first portion of the audio received from a first direction relative to a second portion of the audio received from a second direction.

6. The method of claim 4, wherein processing the audio comprises:

processing first audio originating from a first audio source to obtain a first audio channel;

processing second audio originating from a second audio source to obtain a second audio channel;

combining the first audio channel and the second audio channel to obtain an enhanced stereo channel.

7. The method of claim 4, wherein processing the audio comprises:

identifying an audio source based on characteristics of the captured audio;

determining a direction of the identified audio source, wherein the determined direction is updated as the audio source changes location; and

applying beam steering to the captured audio to adjust a gain of a first portion of the audio received from the direction of the identified audio source relative to a second portion of the audio.

8. The method of claim 4, wherein the first microphone is located near a left ear of a user operating the smart pen device, and the second microphone is located near a right ear of the user.

9. The method of claim 4, wherein the first microphone is located at a first position on a surface of the smart pen device and the second microphone is located at a second position on the surface of a smart pen device at a distance from the first position.

10. The method of claim 4, wherein processing the audio comprises processing the audio in real-time.

11. The method of claim 4, further comprising outputting the processed audio to a speaker.

12. A computer program product for recording audio in a smart pen computing system, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code that comprises:

an audio capture module for capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

a gesture capture module for capturing handwriting gestures, the handwriting gestures captured concurrently with capturing the audio;

an audio processing module for processing the audio captured by the first and second microphones, wherein the audio processing module comprises a noise cancellation module to cancel noise generated from interaction between the smart pen device and a writing surface; and

a synchronization module for synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

13. A computer program product for recording audio in a smart pen computing system, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code that comprises:

an audio capture module for capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

a gesture capture module for capturing handwriting gestures, the handwriting gestures captured concurrently with capturing the audio;

an audio processing module for detecting movement of the first and second microphone during the capturing of the audio and processing the audio captured by the first and second microphone to reduce aurally perceivable distortion caused by the movement; and

a synchronization module for synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

14. A pen-based computing system for recording audio, the system comprising:

a gesture capture system on a smart pen device for capturing handwriting gestures of a user interacting with a writing surface;

a first microphone located at a first position on a surface of the smart pen device, the first microphone communicatively coupled to the smart pen device for recording audio concurrently with capturing the handwriting gestures; and

a second microphone located at a second position on the surface of the smart pen device at a distance from the first position, the second microphone communicatively coupled to the smart pen device for recording the audio concurrently with capturing the handwriting gestures;

a processor programmed to process the audio captured by the first and second microphones and to synchronize the processed audio with the handwriting gestures captured concurrently with the audio;

a memory for storing the processed audio and the handwriting gestures; and

an interface for transferring the processed audio and the captured handwriting gestures to an external computing system.

15. The system of claim 14, wherein the processor processes the audio to cancel noise in the recorded audio generated from interaction between the smart pen device and the writing surface.

16. The system of claim 14, wherein the processor applies a beam steering function to the audio, the beam steering function adapted to adjust a gain of a first portion of the audio received from a first direction relative to a second portion of the audio received from a second direction.

17. A pen-based computing system for recording audio, the system comprising:

a gesture capture system on a smart pen device for capturing handwriting gestures of a user interacting with a writing surface;

a first microphone placed at a first position and communicatively coupled to the smart pen device for recording audio concurrently with capturing the handwriting gestures;

a left ear bud housing the first microphone and adapted to be worn on a left ear;

a second microphone placed at a second position and communicatively coupled to the smart pen device for recording the audio concurrently with capturing the handwriting gestures;

a right ear bud housing the second microphone and adapted to be worn on a right ear;

an ear bud interface for communicatively coupling the left ear bud and the right ear bud to the smart pen device;

a processor programmed to process the audio captured by the first and second microphones and to synchronize the processed audio with the handwriting gestures captured concurrently with the audio;

a memory for storing the processed audio and the handwriting gestures; and

an interface for transferring the processed audio and the captured handwriting gestures to an external computing system.

18. The system of claim 17, further comprising:

a left speaker housed in the left ear bud and adapted to play back a first portion of the recorded audio; and

a right speaker housed in the right ear bud and adapted to play back a second portion of the recorded audio.

19. A computer program product for recording audio in a smart pen computing system, the computer program product comprising a non-transitory computer-readable storage medium containing computer program code that comprises:

an audio capture module for capturing audio using a first microphone and a second microphone, the first and second microphones located at different positions in the smart pen computing system;

a gesture capture module for capturing handwriting gestures using a smart pen device, the handwriting gestures captured concurrently with capturing the audio;

an audio processing module for processing the audio captured by the first and second microphones, wherein the audio processing module is stored on the smart pen device and causes a processor on the smart pen device to process the audio on the smart pen device;

a synchronization module for synchronizing the processed audio with the handwriting gestures captured concurrently with the audio.

20. The computer program product of claim 19, wherein the audio processing module further comprises a beam steering module to adjust a gain of a first portion of the audio received from a first direction relative to a second portion of the audio received from a second direction.

21. The computer program product of claim 19, wherein the audio processing module further executes steps including:

processing first audio originating from a first audio source to obtain a first audio channel;

processing second audio originating from a second audio source to obtain a second audio channel;

combining the first audio channel and the second audio channel to obtain an enhanced stereo channel.

22. The computer program product of claim 19, wherein the audio processing module further executes steps including:

identifying an audio source based on characteristics of the captured audio;

determining a direction of the identified audio source, wherein the determined direction is updated as the audio source moves; and

applying beam steering to the captured audio to adjust a gain of a first portion of the audio received from the direction of the identified audio source relative to a second portion of the audio.

23. The computer program product of claim 19, wherein the audio processing module processes the audio comprises processing the audio in real-time.

24. The computer program product of claim 19, further comprising an audio output module for outputting the processed audio to a speaker.