System and method for whiteboard and audio capture

• US 7,770,116 B2
• Filed: 11/30/2006
• Issued: 08/03/2010
• Est. Priority Date: 06/19/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, Zhang, Zhengyou, Liu, Zicheng, He, Li-wei, Cutler, Ross
• Primary Examiner(s): BAUTISTA, XIOMARA L
• Application Number:
  US11/565,596
• Publication Number:
  US 20070126755A1
• Time in Patent Office: 1,342 Days
• Field of Search: 715/716, 715/719, 715/720, 715/721, 715/727, 715/751, 715/753, 715/759, 715/764, 715/765, 715/769, 715/770, 715/781, 715/786, 715/835, 715/838, 715/856, 715859-862, 715/863, 715/973, 715/974, 715/978, 715/704, 345/156, 345/157, 345/173, 345/179, 345/619
• US Class Current: 715/716
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a digital camera and a microphone. The system can be retrofit to any existing whiteboard. It computes the time stamps of pen strokes on the whiteboard by analyzing the sequence of captured snapshots. It also automatically produces a set of key frames representing all the written content on the whiteboard before each erasure. The whiteboard content serves as a visual index to efficiently browse the audio meeting. The system not only captures the whiteboard content, but also helps the users to view and manage the captured meeting content efficiently and securely.

Claims

1. A computer-implemented process for viewing and playing back event data, comprising:

displaying a graphical user interface for viewing and playing back event data further comprising;

displaying a main display pane displaying a composition of a raw image of content written on a whiteboard and a key frame image summarizing content written on a whiteboard and;

displaying current pen strokes representing strokes that have already been written in an event playback timeline; and

future pen strokes that represent strokes that have not yet been written in the event playback timeline.

2. The computer-implemented process of claim 1 further comprising:

a key frame thumbnail graphical representation of at least one key frame image summarizing content written on a whiteboard.

3. The computer-implemented process of claim 2 wherein selecting a key frame thumbnail displays the corresponding key frame image to the main display pane.

4. The computer-implemented process of claim 3 wherein a user can do at least one of:

zoom in and out using zoom control buttons,

read text and diagrams in the image displayed in the main display pane,

print the image displayed in the main pane; and

cut and paste portions of the image displayed in the main display pane to other documents.

5. The computer-implemented process of claim 1 wherein the raw image displayed in the main display pane depicts content written on a whiteboard at some point in an event playback timeline, said event playback timeline representing playback of a recorded event timeline, and wherein the key frame image displays content written on a whiteboard at some later point in the event playback timeline than the raw image.

6. The computer-implemented process of claim 5 wherein audio is associated with the raw image and key frame image are correlated with associated audio signals recorded at the time the content was written.

7. The computer-implemented process of claim 6 wherein a user can jump to a starting point of audio corresponding to the key frame image in a meeting playback timeline by selecting a key frame thumbnail.

8. The computer-implemented process of claim 6 wherein a user can access corresponding recorded audio at a time a current pen stroke or a future pen stroke in the event playback timeline was written by selecting either current or future pen strokes.

9. The computer-implemented process of claim 8 wherein said time that a current pen stroke or a future pen stroke was written is the earliest time when that current pen stroke or future pen stroke appeared.

10. The computer-implemented process of claim 8 wherein a cursor of an input device is changed to a symbol indicating that it is selectable when the cursor is hovering over a current stroke or future stroke in the main window.

11. The computer-implemented process of claim 5 further comprising a background transparency slider control that allows a user to adjust the image composition displayed in the main display pane from the raw input image to the key frame image.

12. The computer-implemented process of claim 1 wherein current pen strokes, are rendered darker and more clearly than future strokes.

13. The computer-implemented process of claim 1 wherein future strokes, strokes that have not yet been written in the event playback timeline, are shown in ghost-like style.

14. The computer-implemented process of claim 13 further comprising a VCR and standard timeline control that allows the user to do at least one of:

sequence backwards in an event playback timeline;

sequence forwards in an event playback timeline; and

stop in an event playback timeline.

15. The computer-implemented process of claim 13 further comprising a timeline bar graphically displaying an event playback timeline as a bar.

16. The computer-implemented process of claim 15 further comprising numerical values of the start time, end time and current time of the event timeline playback.

17. The computer-implemented process of claim 16 further comprising a pointer on said bar that can be selected and dragged forwards and backwards along the timeline bar to linearly sequence forwards and backwards in an image and audio sequence.

18. The computer-implemented process of claim 1 wherein the image composition is determined by

rendering a first original rectified image of current whiteboard content using an image at a given time and time stamp information;

rendering a second key frame image of future pen strokes relative to said first image, converting said future strokes to gray scale and blurring said future pen strokes using a Gaussian filter;

adding said first and second images to obtain a resultant image; and

alpha-blending said resultant image with said rectified image from an input sequence to achieve an alpha-blended composition image.

19. The computer-implemented process of claim 18 wherein a user can control the alpha value used for alpha-blending with a slider control.

20. The computer-implemented process of claim 18 wherein the user can move the slider from 0, wherein only the key frame image is shown, to 1, showing exactly the original rectified image.