System and method for whiteboard and audio capture

• US 20030234772A1
• Filed: 06/19/2002
• Published: 12/25/2003
• Est. Priority Date: 06/19/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Li-Wei, Gupta, Anoop, Zhang, Zhengyou, Liu, Zicheng, Cutler, Ross
• Granted Patent: US 7,260,257 B2
• Time in Patent Office: Days
• US Class Current: 345/177
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a digital camera and a microphone. The system can be retrofit to any existing whiteboard. It computes the time stamps of pen strokes on the whiteboard by analyzing the sequence of captured snapshots. It also automatically produces a set of key frames representing all the written content on the whiteboard before each erasure. The whiteboard content serves as a visual index to efficiently browse the audio meeting. The system not only captures the whiteboard content, but also helps the users to view and manage the captured meeting content efficiently and securely.

Claims

1. A process for summarizing and indexing audiovisual content, comprising the following process actions:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content.

2. The process of claim 1 further comprising correlating said audio recordings with said key frames.

3. The process of claim 2 wherein said audio signals are correlated with said sequence of images by time stamps associated with both the recorded audio and the sequence of images.

4. The process of claim 3 wherein correlating the audio signals with said sequence of images comprises the process actions of:

time stamping said sequence of images with a common clock at the time the images are captured;

time stamping said audio signals with a common clock at the time the audio signals are recorded; and

correlating the sequence of images and audio signals using the time stamps of the common clock.

5. The process of claim 3 further comprising accessing said sequence of images and said correlated audio signals at a desired point in said sequence of images.

6. The process of claim 5 wherein said key frames are used to select said desired point in said sequence.

7. The process of claim 1 wherein said analyzing process action comprises the following process actions:

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells, each cell representing a cell image;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

8. The process of claim 7 further comprising the process action of filtering cell images spatially to refine classification results.

9. The process of claim 7 further comprising the process action of filtering the cell images temporally to refine classification results.

10. The process of claim 7 further comprising color balancing the key frame images to improve image quality.

11. The process of claim 7 wherein rectifying said whiteboard further comprises:

specifying the four corners of said whiteboard in an image;

cropping any non-whiteboard region of each image; and

performing a bi-linear warp to a rectangular shape for each whiteboard region using bi-cubic interpolation to obtain a cropped and rectified whiteboard image in each captured image of the sequence of images.

12. The process of claim 7 wherein the process action for extracting whiteboard background color comprises:

determining whiteboard cells with the brightest luminance and smallest variance;

determining any holes in whiteboard color; and

filling each hole by searching whiteboard cells around said hole and setting color to nearest cell that is not a hole.

13. The process of claim 7 wherein the process action for determining any holes in the whiteboard comprises the process actions of:

specifying a range of colors that a whiteboard cell can be;

examining each whiteboard cell for being outside said range of colors; and

designating any cell having a color outside said range as a hole.

14. The process of claim 7 wherein the process action for extracting whiteboard background color comprises:

histogramming whiteboard image luminance;

determining peak whiteboard luminance;

designating color corresponding to peak luminance as initial whiteboard color;

determining any whiteboard color outliers, said outliers representing erroneous data;

marking any outliers as holes; and

filling each hole by searching cells around hole and setting color to that of nearest cell that is not a hole.

15. The process of claim 14 further comprising filtering whiteboard color image after filling each hole.

16. The process of claim 15 wherein outliers are detected using a least-median squares technique.

17. The process of claim 16 wherein the least-median squares technique comprises the following process actions:

fitting a plane in the luminance Y or RGB space by minimizing the median of the squared errors; and

designating cells that do not follow this model as outliers.

18. The process of claim 15 wherein filtering said whiteboard image comprises fitting a plane in red, green, blue (RGB) color space.

19. The process of claim 7 wherein the process action for dividing each image up into cells comprises dividing each image into rectangular cells.

20. The process of claim 7 wherein the process action for dividing each image into cells comprises dividing each image into cells wherein cell size is approximately the same size of a single written character on the board.

21. The process of claim 20 wherein the cell size corresponds to a region of 1.5 inches by 1.5 inches in the whiteboard.

22. The process of claim 19 wherein the cell size is determined by camera resolution and whiteboard size.

23. The process of claim 15 wherein the cell size is 1 pixel by 1 pixel.

24. The process of claim 7 wherein the process action for clustering cell images comprises clustering cell images that are considered to be the same over time.

25. The process of claim 24 wherein the process action of clustering cell images that are considered to be the same over time comprises the action of determining if cell images are the same by using a normalized cross-correlation technique to compare two cells at a time.

26. The process of claim 25 wherein the cross-correlation score ranges from −

1, for two images not similar at all, to 1, for two identical images.

27. The process of claim 26 further comprising a process action of applying a Mahalanobis distance test to determine if two cells are the same.

28. The process of claim 27 wherein the Mahalanobis distance test given by d=|{overscore (I)}−

{overscore (I)}′

|/(σ

+σ

′

);

where I is the first cell image and I′

is the second cell image, {overscore (I)} is the mean color of the first cell image, {overscore (I)}′

is the mean color of the second cell image, σ

is the standard deviation from {overscore (I)} and σ

′

is the standard deviation from {overscore (I)}′ and

wherein I and I′

are the two cell images are considered to be identical if and only if d<

T_d and c>

T_c, and wherein T_d=2 and T_c=0.707 in one particular embodiment.

29. The process of claim 7 wherein the cell classifying process action further comprises:

designating a cell image as a whiteboard cell if the RGB values are approximately the same;

designating a cell image as a stroke cell if said cell is mostly white or gray with one or two primary colors mixed in; and

designating a cell image as a foreground cell if it does not have the characteristics of a whiteboard cell or a stroke cell.

30. The process of claim 7 wherein the cell classifying process action further comprises:

determining the color distribution of a current cell image;

determining the color distribution of the corresponding whiteboard cell and

determining if the color distribution of the current cell image and the corresponding whiteboard cell are the same, not the same but having a strong similarity or are totally different.

31. The process of claim 29 wherein the cell image is classified as a whiteboard cell if and only if |{overscore (I)}−

{overscore (I)}_w|/(σ

+σ

_w)<

T_w and σ

/σ

_w<

T_σ

where {overscore (I)}_w is the whiteboard color, σ

_w is the standard deviation from {overscore (I)}_w, {overscore (I)} is the mean color of the current cell and σ

is the standard deviation of {overscore (I)};

wherein T_w=2 and T_σ

=2.

otherwise, it is classified as a foreground object cell.

32. The process of claim 8 wherein the process action of spatially filtering the cell images comprises the following process actions:

identifying isolated foreground cells;

reclassifying isolated foreground cells as stroke cells;

identifying stroke cells that are connected to foreground cells; and

reclassifying the stroke cells that are connected to foreground cells are foreground cells.

33. The process of claim 9 wherein the process action of temporally filtering the cell images comprises the following process actions:

evaluating each cell over time to determine if it contained the same stroke cell before and after it was classified as a foreground cell; and

if a cell was classified as the same stroke cell before and after it was classified as a stroke cell, classifying it as a stroke cell.

34. The process of claim 7 wherein the key frame extracting process action comprises:

inputting the classification results;

counting the stroke cells for each frame in the sequence of images;

using the stroke count for the frames to determine the peaks and valleys of the stroke count;

if the difference between each adjacent peak and valley exceeds a prescribed threshold, designating the data between valleys as chapters and the peak within each chapter as the key frame representing the chapter.

35. The process of claim 34 wherein the prescribed threshold is 20 percent of the total stroke count.

36. The process of claim 34 further comprising the process action of reconstructing said key frame images.

37. The process of claim 36 wherein said process action of reconstructing said key frame images further comprises the following process actions:

inputting the classified cell images of the sequence of images and key frames divided into cell images;

if a key frame cell image is classified as a whiteboard image or a stroke image, its image is rendered;

if a key frame foreground cell image is within the span of a stroke, this cell image is rendered with the stroke cell image from neighboring images in the sequence;

if the key frame cell image is not classified as a whiteboard image, a stroke image or a foreground cell within the span of a stroke, it is filled rendered as a whiteboard image.

38. The process of claim 10 wherein the color balancing of the key frame images to improve image quality further comprises the process actions of:

making the background uniformly white and increasing the color saturation of pen strokes by using mean whiteboard color to scale the color of each pixel in a cell; and

reducing image noise.

39. A graphical user interface for viewing and playing back event data comprising:

a main display pane displaying a composition of a raw image of content written on a whiteboard and a key frame image summarizing content written on a whiteboard.

40. The graphical user interface of claim 39 further comprising:

a graphical representations of at least one key frame image summarizing content written on a whiteboard.

41. The graphical user interface of claim 39 wherein the raw image displayed in the main display pane depicts content written on a whiteboard at some point in an event playback timeline, said event playback timeline representing playback of a recorded event timeline, and wherein the key frame image displays content written on a whiteboard at some later point in the event playback timeline than the raw image.

42. The graphical user interface of claim 41 wherein audio is associated with the raw image and key frame image are correlated with associated audio signals recorded at the time the content was written.

43. The graphical user interface of claim 41 further comprising a background transparency slider control that allows a user to adjust the image composition displayed in the main display pane from the raw input image to the key frame image.

44. The graphical user interface of claim 41 further comprising displaying current pen strokes representing strokes that have already been written in an event playback timeline;

and future pen strokes that represent strokes that have not yet been written in the event playback timeline.

45. The graphical user interface of claim 44 wherein current pen strokes, are rendered darker and more clearly than future strokes.

46. The graphical user interface of claim 44 wherein future strokes, strokes that have not yet been written in the event playback timeline, are shown in ghost-like style.

47. The graphical user interface of claim 39 wherein the image composition is determined by

rendering a first original rectified image of current whiteboard content using an image at a given time and time stamp information;

rendering a second key frame image of future pen strokes relative to said first image, converting said future strokes to gray scale and blurring said future pen strokes using a Gaussian filter;

adding said first and second images to obtain a resultant image; and

alpha-blending said resultant image with said rectified image from an input sequence to achieve an alpha-blended composition image.

48. The graphical user interface of claim 47 wherein a user can control the alpha value used for alpha-blending with a slider control.

49. The graphical user interface of claim 47 wherein the user can move the slider from 0, wherein only the key frame image is shown, to 1, showing exactly the original rectified image.

50. The graphical user interface of claim 42 further comprising a VCR and standard timeline control that allows the user to do at least one of:

sequence backwards in an event playback timeline;

sequence forwards in an event playback timeline; and

stop in an event playback timeline.

51. The graphical user interface of claim 42 further comprising a timeline bar graphically displaying an event playback timeline as a bar.

52. The graphical user interface of claim 51 further comprising numerical values of the start time, end time and current time of the event timeline playback.

53. The graphical user interface of claim 52 further comprising a pointer on said bar that can be selected and dragged forwards and backwards along the timeline bar to linearly sequence forwards and backwards in an image and audio sequence.

54. The graphical user interface of claim 42 wherein a user can jump to a starting point of audio corresponding to the key frame image in a meeting playback timeline by selecting said key frame thumbnail.

55. The graphical user interface of claim 42 wherein a user can access corresponding recorded audio at a time a current pen stroke or a future pen stroke in the event playback timeline was written by selecting either current or future pen strokes.

56. The graphical user interface of claim 55 wherein said time that a current pen stroke or a future pen stroke was written is the earliest time when that current pen stroke or future pen stroke appeared.

57. The graphical user interface of claim 55 wherein a cursor of an input device is changed to a symbol indicating that it is selectable when the cursor is hovering over a current stroke or future stroke in the main window.

58. The graphical user interface of claim 40 wherein selecting a key frame thumbnail displays the corresponding key frame image to the main display pane.

59. The graphical user interface of claim 58 wherein a user can do at least one of:

zoom in and out using zoom control buttons,

read text and diagrams in the image displayed in the main display pane,

print the image displayed in the main pane; and

cut and paste portions of the image displayed in the main display pane to other documents.

60. A system for capturing the audio and video content of a meeting comprising:

a capture system that captures a sequence of images of data written on a whiteboard and audio signals corresponding to sounds that occur during a meeting;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames; and

a browsing module for viewing the analyzed meeting key data frames and correlated audio.

61. The system of claim 60 wherein the capture system further comprises:

a camera positioned to capture sequence of images;

a microphone to record the audio signals; and

a computer for recording the sequence of images and the audio signals.

62. The system of claim 61 wherein the camera is at least one of:

a still camera; and

a video camera.

63. The system of claim 61 wherein the camera is zoomed in as close to the whiteboard as possible to maximize resolution.

64. The system of claim 61 wherein the camera is aligned as parallel to the whiteboard as possible to minimize scene depth.

65. The system of claim 60 wherein the analysis server identifies the key data frames by:

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells of cell images;

clustering cell images that are similar throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

66. The system of claim 60 wherein the analysis server identifies the key data frames by:

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

clustering pixels that are similar throughout the sequence of images for each cell over time;

classifying each pixel as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

67. The system of claim 60 wherein one or more users register a user identifier at the capture unit before recording starts;

if at least one user registers at the capture unit, the analysis server generates an access token after event recording and analysis;

the access token and computer memory location of analyzed meeting data is provided to the registered user identifiers; and

the one or more users access the computer memory location of the analyzed event data to review the analyzed meeting data.

68. The system of claim 67 wherein the user identifier is an email address.

69. The system of claim 67 wherein the computer memory location of the analyzed event data is an address of an Internet web site

70. The system of claim 60 further comprising a privacy feature in said capture unit that allows a user erase at least one of:

portions of the sequence images; and

portions of the audio.

71. The system of claim 70 wherein said privacy feature is activated by pressing either a graphical user interface button or a physical button.

72. The system of claim 60 wherein the capture system is portable.

73. A process for distilling event content, comprising the following process actions:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content by;

rectifying a view of the whiteboard in each image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

74. The system of claim 73 wherein the analyzing process action further comprises using Optical Character Recognition techniques to provide editable text from key frame images.

75. The system of claim 73 wherein the analyzing process action further comprises using voice recognition techniques to convert the audio to text.

76. The system of claim 73 further comprising using gesture recognition techniques.

77. The system of claim 73 wherein said capturing process action further comprises:

installing a color patch of known color characteristics above the whiteboard;

capturing said color patch in an image in said sequence of images;

determining the color exhibited by the color patch in the image;

capturing said color patch in the next image in said sequence of images;

determining the color exhibited by the color patch in said next image;

determining if the color exhibited by the patch has changed between the image and the next image by more than a prescribed threshold; and

if the color exhibited exceeds the threshold adjusting the exposure parameters of the camera to a setting which would have made the color of said color patch in the next image, had the camera been set to said exposure setting, match the color of the color patch exhibited in the image.

78. The system of claim 77 wherein said adjusting process action is performed on every image captured.

79. A system for distilling the content of a meeting comprising:

a capture system that captures a sequence of data written on a whiteboard, said capture system tracking pen location to infer content written on a whiteboard and recording audio signals correlating to said content written on a whiteboard;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames.

80. The system of claim 79 wherein said analysis server performs the following actions:

dividing each region of the whiteboard into cells;

clustering cells that are the same throughout the sequence of data written for each cell over time;

classifying each cell as a stroke or a whiteboard cell; and

extracting key frame images using the classification results.

81. The system of claim 80 wherein the whiteboard cells are divided into cells that are approximately the size of one written character.

82. A computer-readable medium having computer-executable instructions for providing non-linear access to data recorded during a meeting, said computer executable instructions comprising program modules for:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content by;

rectifying a view of the whiteboard in each image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells;

clustering cell images throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.