System and Method for Whiteboard and Audio Capture

• US 20070269105A1
• Filed: 08/31/2006
• Published: 11/22/2007
• Est. Priority Date: 06/19/2002
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Gupta, Anoop, Zhang, Zhengyou, He, Li-wei, Cutler, Ross, Lui, Zicheng
• Application Number:
  US11/469,450
• Publication Number:
  US 20070269105A1
• Time in Patent Office: Days
• US Class Current: 382/165
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06Q 10/1095 Meeting or appointment
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/806 with processing of the sound signal

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a digital camera and a microphone. The system can be retrofit to any existing whiteboard. It computes the time stamps of pen strokes on the whiteboard by analyzing the sequence of captured snapshots. It also automatically produces a set of key frames representing all the written content on the whiteboard before each erasure. The whiteboard content serves as a visual index to efficiently browse the audio meeting. The system not only captures the whiteboard content, but also helps the users to view and manage the captured meeting content efficiently and securely.

Claims

1-59. -59. (canceled)

60. A system for capturing the audio and video content of a meeting comprising:

a capture system that captures a sequence of images of data written on a non-electronic whiteboard and audio signals corresponding to sounds that occur during a meeting;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames; and

a browsing module for viewing the analyzed meeting key data frames and correlated audio.

61. The system of claim 60 wherein the capture system further comprises:

a camera positioned to capture sequence of images;

a microphone to record the audio signals; and

a computer for recording the sequence of images and the audio signals.

62. The system of claim 61 wherein the camera is at least one of:

a still camera; and

a video camera.

63. The system of claim 61 wherein the camera is zoomed in as close to the whiteboard as possible to maximize resolution.

64. The system of claim 61 wherein the camera is aligned as parallel to the whiteboard as possible to minimize scene depth.

65. The system of claim 60 wherein the analysis server identifies the key data frames by:

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

dividing each image of the sequence of images into cells of cell images;

clustering cell images that are similar throughout the sequence of images for each cell over time;

classifying each cell image as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

66. The system of claim 60 wherein the analysis server identifies the key data frames by:

rectifying a view of the whiteboard in every image in the sequence of images;

extracting whiteboard background color;

clustering pixels that are similar throughout the sequence of images for each cell over time;

classifying each pixel as a stroke, a foreground object or whiteboard cell; and

extracting key frame images using the classification results.

67. The system of claim 60 wherein

one or more users register a user identifier at the capture unit before recording starts;

if at least one user registers at the capture unit, the analysis server generates an access token after event recording and analysis;

the access token and computer memory location of analyzed meeting data is provided to the registered user identifiers; and

the one or more users access the computer memory location of the analyzed event data to review the analyzed meeting data.

68. The system of claim 67 wherein the user identifier is an email address.

69. The system of claim 67 wherein the computer memory location of the analyzed event data is an address of an Internet web site.

70. The system of claim 60 further comprising a privacy feature in said capture unit that allows a user erase at least one of:

portions of the sequence images; and

portions of the audio.

71. The system of claim 70 wherein said privacy feature is activated by pressing either a graphical user interface button or a physical button.

72. The system of claim 60 wherein the capture system is portable.

73-78. -78. (canceled)

79. A system for distilling the content of a meeting comprising:

a capture system that captures a sequence of data written on a non-electronic whiteboard, said capture system tracking pen location to infer content written on a whiteboard and recording audio signals correlating to said content written on a whiteboard;

an analysis server for analyzing the sequence of images that extracts key data frames written on the whiteboard and correlates the audio signals to the key data frames.

80. The system of claim 79 wherein said analysis server performs the following actions:

dividing each region of the whiteboard into cells;

clustering cells that are the same throughout the sequence of data written for each cell over time;

classifying each cell as a stroke or a whiteboard cell; and

extracting key frame images using the classification results.

81. The system of claim 80 wherein the whiteboard cells are divided into cells that are approximately the size of one written character.

82. (canceled)

83. A process for summarizing and indexing audiovisual content, comprising the following process actions:

capturing a sequence of images of content written on a non-electronic white board with a camera;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to isolate key frames summarizing key points of said board content.

84. The process of claim 83 further comprising correlating said audio recordings with said key frames.

85. The process of claim 84 wherein said audio signals are correlated with said sequence of images by time stamps associated with both the recorded audio and the sequence of images.

86. The process of claim 85 wherein correlating the audio signals with said sequence of images comprises the process actions of:

time stamping said sequence of images with a common clock at the time the images are captured;

time stamping said audio signals with a common clock at the time the audio signals are recorded; and

correlating the sequence of images and audio signals using the time stamps of the common clock.

87. The process of claim 85 further comprising accessing said sequence of images and said correlated audio signals at a desired point in said sequence of images.

88. The process of claim 87 wherein said key frames are used to select said desired point in said sequence.