System and method for real-time whiteboard capture and processing

• US 7,426,297 B2
• Filed: 03/21/2007
• Issued: 09/16/2008
• Est. Priority Date: 11/18/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, He, Liwei
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US11/689,171
• Publication Number:
  US 20070156816A1
• Time in Patent Office: 545 Days
• Field of Search: 382162-167, 382224-225, 382/236, 382/103, 382/173, 358518-540, 358/1.9, 358/1.15, 345/173, 345589-604, 3482111-21114, 348/207.99, 348/222.1, 348 1401- 1416, 348/273, 348/254, 709/231, 386/124, 386 69- 70, 386111-112, 434/323, 434350-362
• US Class Current: 382/165
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a video camera and records or transmits them in real-time. The Real-Time Whiteboard Capture captures pen strokes on whiteboards in real time using an off-the-shelf video camera. Unlike many existing tools, the RTWCS does not instrument the pens or the whiteboard. It analyzes the sequence of captured video images in real time, classifies the pixels into whiteboard background, pen strokes and foreground objects (e.g., people in front of the whiteboard), and extracts newly written pen strokes. This allows the RTWCS to transmit whiteboard contents using very low bandwidth to remote meeting participants. Combined with other teleconferencing tools such as voice conference and application sharing, the RTWCS becomes a powerful tool to share ideas during online meetings.

Claims

1. A system for transferring the content of data written on a whiteboard to at least one remote meeting participant, comprising:

a capture system that captures a sequence of images of content written on a whiteboard and audio signals corresponding to sounds that occur during a meeting;

an analysis server for analyzing the sequence of images and transmitting whiteboard content without foreground obstructions and associated audio to a remote meeting participant in real-time wherein the analysis server comprises modules for;

dividing each of said images into cells of cell images;

for each image,

determining if the cell images have changed from the last frame;

computing the background color of the cell images which represent the color of blank whiteboard;

updating a whiteboard background model by integrating the color of the blank whiteboard computed from the cell images;

classifying the cell images into foreground or whiteboard cells using the whiteboard color model;

extracting changed whiteboard content; and

outputting changed whiteboard content to at least one remote participant.

2. The system of claim 1 further comprising a module for enhancing the color of whiteboard content.

3. The system of claim 1 wherein each image frame is processed in Bayer color format.

4. The system of claim 3 wherein the sequence of images are captured in Bayer color format.

5. The system of claim 1 wherein the analysis server processes and outputs changed whiteboard content in real-time.