System and Method for Whiteboard and Audio Capture

• US 20070126755A1
• Filed: 11/30/2006
• Published: 06/07/2007
• Est. Priority Date: 06/19/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, Zhang, Zhengyou, Liu, Zicheng, He, Li-wei, Cutler, Ross
• Granted Patent: US 7,770,116 B2
• Time in Patent Office: Days
• US Class Current: 345/619
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a digital camera and a microphone. The system can be retrofit to any existing whiteboard. It computes the time stamps of pen strokes on the whiteboard by analyzing the sequence of captured snapshots. It also automatically produces a set of key frames representing all the written content on the whiteboard before each erasure. The whiteboard content serves as a visual index to efficiently browse the audio meeting. The system not only captures the whiteboard content, but also helps the users to view and manage the captured meeting content efficiently and securely.

Claims

1. A graphical user interface for viewing and playing back event data comprising:

a main display pane displaying a composition of a raw image of content written on a whiteboard and a key frame image summarizing content written on a whiteboard.

2. The graphical user interface of claim 1 further comprising:

a graphical representations of at least one key frame image summarizing content written on a whiteboard.

3. The graphical user interface of claim 1 wherein the raw image displayed in the main display pane depicts content written on a whiteboard at some point in an event playback timeline, said event playback timeline representing playback of a recorded event timeline, and wherein the key frame image displays content written on a whiteboard at some later point in the event playback timeline than the raw image.

4. The graphical user interface of claim 3 wherein audio is associated with the raw image and key frame image are correlated with associated audio signals recorded at the time the content was written.

5. The graphical user interface of claim 3 further comprising a background transparency slider control that allows a user to adjust the image composition displayed in the main display pane from the raw input image to the key frame image.

6. The graphical user interface of claim 3 further comprising displaying current pen strokes representing strokes that have already been written in an event playback timeline;

and future pen strokes that represent strokes that have not yet been written in the event playback timeline.

7. The graphical user interface of claim 6 wherein current pen strokes, are rendered darker and more clearly than future strokes.

8. The graphical user interface of claim 6 wherein future strokes, strokes that have not yet been written in the event playback timeline, are shown in ghost-like style.

9. The graphical user interface of claim 1 wherein the image composition is determined by

rendering a first original rectified image of current whiteboard content using an image at a given time and time stamp information;

rendering a second key frame image of future pen strokes relative to said first image, converting said future strokes to gray scale and blurring said future pen strokes using a Gaussian filter;

adding said first and second images to obtain a resultant image; and

alpha-blending said resultant image with said rectified image from an input sequence to achieve an alpha-blended composition image.

10. The graphical user interface of claim 9 wherein a user can control the alpha value used for alpha-blending with a slider control.

11. The graphical user interface of claim 9 wherein the user can move the slider from 0, wherein only the key frame image is shown, to 1, showing exactly the original rectified image.

12. The graphical user interface of claim 8 further comprising a VCR and standard timeline control that allows the user to do at least one of:

sequence backwards in an event playback timeline;

sequence forwards in an event playback timeline; and

stop in an event playback timeline.

13. The graphical user interface of claim 8 further comprising a timeline bar graphically displaying an event playback timeline as a bar.

14. The graphical user interface of claim 13 further comprising numerical values of the start time, end time and current time of the event timeline playback.

15. The graphical user interface of claim 14 further comprising a pointer on said bar that can be selected and dragged forwards and backwards along the timeline bar to linearly sequence forwards and backwards in an image and audio sequence.

16. The graphical user interface of claim 4 wherein a user can jump to a starting point of audio corresponding to the key frame image in a meeting playback timeline by selecting said key frame thumbnail.

17. The graphical user interface of claim 4 wherein a user can access corresponding recorded audio at a time a current pen stroke or a future pen stroke in the event playback timeline was written by selecting either current or future pen strokes.

18. The graphical user interface of claim 17 wherein said time that a current pen stroke or a future pen stroke was written is the earliest time when that current pen stroke or future pen stroke appeared.

20. The graphical user interface of claim 2 wherein selecting a key frame thumbnail displays the corresponding key frame image to the main display pane.

21. The graphical user interface of claim 20 wherein a user can do at least one of:

zoom in and out using zoom control buttons,

read text and diagrams in the image displayed in the main display pane,

print the image displayed in the main pane; and

cut and paste portions of the image displayed in the main display pane to other documents.

19. The graphical user interface of claim 55 wherein a cursor of an input device is changed to a symbol indicating that it is selectable when the cursor is hovering over a current stroke or future stroke in the main window.