System and method for whiteboard and audio capture

• US 7,260,257 B2
• Filed: 06/19/2002
• Issued: 08/21/2007
• Est. Priority Date: 06/19/2002
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, Zhang, Zhengyou, Liu, Zicheng, He, Li-wei, Cutler, Ross
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/178,443
• Publication Number:
  US 20030234772A1
• Time in Patent Office: 1,889 Days
• Field of Search: 382/165, 382224-225, 382/162, 382/305, 382/164, 382/176, 382202-203, 382/278, 382/282, 348699-700, 348/14.1, 348/273, 348/254, 348/335, 348/239, 348/36, 3482111-21114, 348/207.99, 348/222.1, 348 1401- 1416, 386 69- 70, 386/124, 386111-112, 715/700, 715722-723, 715/835, 715/741, 715/727, 715/776, 358/518, 358/1.9, 358/1.15, 358/538, 345/173, 345/427, 345/582, 345/589, 709224-232, 709217-219, 709/204, 359/725, 359/94, 359/403, 434322-323, 434350-362
• US Class Current: 382/165
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a digital camera and a microphone. The system can be retrofit to any existing whiteboard. It computes the time stamps of pen strokes on the whiteboard by analyzing the sequence of captured snapshots. It also automatically produces a set of key frames representing all the written content on the whiteboard before each erasure. The whiteboard content serves as a visual index to efficiently browse the audio meeting. The system not only captures the whiteboard content, but also helps the users to view and manage the captured meeting content efficiently and securely.

Claims

1. A process for summarizing and indexing audiovisual content, comprising the following process actions:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content;

wherein said analyzing process action comprises the following process actions;

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells, each cell representing a cell image;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

2. The process of claim 1 further comprising correlating said audio recordings with said key frames.

3. The process of claim 2 wherein said audio signals are correlated with said sequence of images by time stamps associated with both the recorded audio and the sequence of images.

4. The process of claim 3 wherein correlating the audio signals with said sequence of images comprises the process actions of:

time stamping said sequence of images with a common clock at the time the images are captured;

time stamping said audio signals with a common clock at the time the audio signals are recorded; and

correlating the sequence of images and audio signals using the time stamps of the common clock.

5. The process of claim 3 further comprising accessing said sequence of images and said correlated audio signals at a desired point in said sequence of images.

6. The process of claim 5 wherein said key frames are used to select said desired point in said sequence.

7. The process of claim 1 further comprising the process action of filtering cell images spatially to refine classification results.

8. The process of claim 1 further comprising the process action of filtering the cell images temporally to refine classification results.

9. The process of claim 1 further comprising color balancing the key frame images to improve image quality.

10. The process of claim 1 wherein rectifying said whiteboard further comprises:

specifying the four corners of said whiteboard in an image;

cropping any non-whiteboard region of each image; and

performing a bi-linear warp to a rectangular shape for each whiteboard region using bi-cubic interpolation to obtain a cropped and rectified whiteboard image in each captured image of the sequence of images.

11. The process of claim 1 wherein the process action for extracting whiteboard background color comprises:

determining whiteboard cells with the brightest luminance and smallest variance;

determining any holes in whiteboard color; and

filling each hole by searching whiteboard cells around said hole and setting color to nearest cell that is not a hole.

12. The process of claim 1 wherein the process action for determining any holes in the whiteboard comprises the process actions of:

specifying a range of colors that a whiteboard cell can be;

examining each whiteboard cell for being outside said range of colors; and

designating any cell having a color outside said range as a hole.

13. The process of claim 1 wherein the process action for extracting whiteboard background color comprises:

histogramming whiteboard image luminance;

determining peak whiteboard luminance;

designating color corresponding to peak luminance as initial whiteboard color;

determining any whiteboard color outliers, said outliers representing erroneous data;

marking any outliers as holes; and

filling each hole by searching cells around hole and setting color to that of nearest cell that is not a hole.

14. The process of claim 13 further comprising filtering whiteboard color image after filling each hole.

15. The process of claim 14 wherein outliers are detected using a least-median squares technique.

16. The process of claim 15 wherein the least-median squares technique comprises the following process actions:

fitting a plane in the luminance Y or RGB space by minimizing the median of the squared errors; and

designating cells that do not follow this model as outliers.

17. The process of claim 14 wherein filtering said whiteboard image comprises fitting a plane in red, green, blue (RGB) color space.

18. The process of claim 1 wherein the process action for dividing each image up into cells comprises dividing each image into rectangular cells.

19. The process of claim 1 wherein the process action for dividing each image into cells comprises dividing each image into cells wherein cell size is approximately the same size of a single written character on the board.

20. The process of claim 18 wherein the cell size corresponds to a region of 1.5 inches by 1.5 inches in the whiteboard.

21. The process of claim 18 wherein the cell size is determined by camera resolution and whiteboard size.

22. The process of claim 14 wherein the cell size is 1 pixel by 1 pixel.

23. The process of claim 1 wherein the process action for clustering cell images comprises clustering cell images that are considered to be the same over time.

24. The process of claim 23 wherein the process action of clustering cell images that are considered to be the same over time comprises the action of determining if cell images are the same by using a normalized cross-correlation technique to compare two cells at a time.

25. The process of claim 24 wherein the cross-correlation score ranges from −

1, for two images not similar at all, to 1, for two identical images.

26. The process of claim 25 further comprising a process action of applying a Mahalanobis distance test to determine if two cells are the same.

27. The process of claim 26 wherein the Mahalanobis distance test given by d=|Ī

−

Ī

′

|/(σ

+σ

′

);

where i is the first cell image and I′

is the second cell image, Ī

is the mean color of the first cell image, Ī

′

is the mean color of the second cell image, σ

is the standard deviation from Ī and

σ

′

is the standard deviation from Ī

′ and

wherein I and I′

are the two cell images are considered to be identical if and only if d<

T_d and c>

T_c, and wherein T_d=2 and T_c=0.707 in one particular embodiment.

28. The process of claim 1 wherein the cell classifying process action further comprises:

designating a cell image as a whiteboard cell if the RGB values are approximately the same;

designating a cell image as a stroke cell if said cell is mostly white or gray with one or two primary colors mixed in; and

designating a cell image as a foreground cell if it does not have the characteristics of a whiteboard cell or a stroke cell.

29. The process of claim 1 wherein the cell classifying process action further comprises:

determining the color distribution of a current cell image;

determining the color distribution of the corresponding whiteboard cell and

determining if the color distribution of the current cell image and the corresponding whiteboard cell are the same, not the same but having a strong similarity or are totally different.

30. The process of claim 28 wherein the cell image is classified as a whiteboard cell if and only if |Ī

−

Ī

_w|/(σ

+σ

_w)<

T_w and σ

/σ

_w<

T _σ

where Ī

_w is the whiteboard color, σ

_w is the standard deviation from Ī

_w, Ī

is the mean color of the current cell and o is the standard deviation of Ī

;

wherein T_w=2 and T_σ

=2.

otherwise, it is classified as a foreground object cell.

31. The process of claim 7 wherein the process action of spatially filtering the cell images comprises the following process actions:

identifying isolated foreground cells;

reclassifying isolated foreground cells as stroke cells;

identifying stroke cells that are connected to foreground cells; and

reclassifying the stroke cells that are connected to foreground cells are foreground cells.

32. The process of claim 8 wherein the process action of temporally filtering the cell images comprises the following process actions:

evaluating each cell over time to determine if it contained the same stroke cell before and after it was classified as a foreground cell; and

if a cell was classified as the same stroke cell before and after it was classified as a stroke cell, classifying it as a stroke cell.

33. The process of claim 1 wherein the key frame extracting process action comprises:

inputting the classification results;

counting the stroke cells for each frame in the sequence of images;

using the stroke count for the frames to determine the peaks and valleys of the stroke count;

if the difference between each adjacent peak and valley exceeds a prescribed threshold, designating the data between valleys as chapters and the peak within each chapter as the key frame representing the chapter.

34. The process of claim 33 wherein the prescribed threshold is 20 percent of the total stroke count.

35. The process of claim 33 further comprising the process action of reconstructing said key frame images.

36. The process of claim 35 wherein said process action of reconstructing said key frame images further comprises the following process actions:

inputting the classified cell images of the sequence of images and key frames divided into cell images;

if a key frame cell image is classified as a whiteboard image or a stroke image, its image is rendered;

if a key frame foreground cell image is within the span of a stroke, this cell image is rendered with the stroke cell image from neighboring images in the sequence;

if the key frame cell image is not classified as a whiteboard image, a stroke image or a foreground cell within the span of a stroke, it is filled rendered as a whiteboard image.

37. The process of claim 9 wherein the color balancing of the key frame images to improve image quality further comprises the process actions of:

making the background uniformly white and increasing the color saturation of pen strokes by using mean whiteboard color to scale the color of each pixel in a cell; and

reducing image noise.

38. A system for capturing the audio and video content of a meeting comprising:

a capture system that captures a sequence of images of data written on a whiteboard and audio signals corresponding to sounds that occur during a meeting;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames wherein the analysis server identifies the key data frames by;

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells of cell images;

clustering cell images that are similar throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results; and

a browsing module for viewing the analyzed meeting key data frames and correlated audio.

39. The system of claim 38 wherein the capture system further comprises:

a camera positioned to capture sequence of images;

a microphone to record the audio signals; and

a computer for recording the sequence of images and the audio signals.

40. The system of claim 39 wherein the camera is at least one of:

a still camera; and

a video camera.

41. The system of claim 39 wherein the camera is zoomed in as close to the whiteboard as possible to maximize resolution.

42. The system of claim 39 wherein the camera is aligned as parallel to the whiteboard as possible to minimize scene depth.

46. The system of claim 38 further comprising a privacy feature in said capture unit that allows a user erase at least one of:

portions of the sequence images; and

portions of the audio.

47. The system of claim 46 wherein said privacy feature is activated by pressing either a graphical user interface button or a physical button.

48. The system of claim 38 wherein the capture system is portable.

43. A system for capturing the audio and video content of a meeting comprising:

a capture system that captures a sequence of images of data written on a whiteboard and audio signals corresponding to sounds that occur during a meeting, wherein

one or more users register a user identifier at the capture unit before recording starts;

if at least one user registers at the capture unit, the analysis server generates an access token after event recording and analysis;

the access token and computer memory location of analyzed meeting data is provided to the registered user identifiers; and

the one or more users access the computer memory location of the analyzed event data to review the analyzed meeting data;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames; and

a browsing module for viewing the analyzed meeting key data frames and correlated audio.

44. The system of claim 43 wherein the user identifier is an email address.

45. The system of claim 43 wherein the computer memory location of the analyzed event data is an address of an Internet web site.

49. A process for distilling event content, comprising the following process actions:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content by;

rectifying a view of the whiteboard in each image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

50. The process of claim 49 wherein the analyzing process action further comprises using Optical Character Recognition techniques to provide editable text from key frame images.

51. The process of claim 49 wherein the analyzing process action further comprises using voice recognition techniques to convert the audio to text.

52. The process of claim 49 further comprising using gesture recognition techniques.

53. The process of claim 49 wherein said capturing process action further comprises:

installing a color patch of known color characteristics above the whiteboard;

capturing said color patch in an image in said sequence of imdages;

determining the color exhibited by the color patch in the image;

capturing said color patch in the next image in said sequence of images;

determining the color exhibited by the color patch in said next image;

determining if the color exhibited by the patch has changed between the image and the next image by more than a prescribed threshold; and

if the color exhibited exceeds the threshold adjusting the exposure parameters of the camera to a setting which would have made the color of said color patch in the next image, had the camera been set to said exposure setting, match the color of the color patch exhibited in the image.

54. The process of claim 53 wherein said adjusting process action is performed on every image captured.

55. A system for distilling the content of a meeting comprising:

a capture system that captures a sequence of data written on a whiteboard, said capture system tracking pen location to infer content written on a whiteboard and recording audio signals correlating to said content written on a whiteboard;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames, wherein said analysis server performs the following actions;

dividing each region of the whiteboard into cells;

clustering cells that are the same throughout the sequence of data written for each cell over time;

classifying each cell as a stroke or a whiteboard cell; and

extracting key frame images using the classification results.

56. The system of claim 55 wherein the whiteboard cells are divided into cells that are approximately the size of one written character.

57. A computer-readable medium having computer-executable instructions for providing non-linear access to data recorded during a meeting, said computer executable instructions comprising program modules for:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content by;

rectifying a view of the whiteboard in each image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.