System and method for real-time whiteboard capture and processing

• US 7,260,278 B2
• Filed: 03/19/2004
• Issued: 08/21/2007
• Est. Priority Date: 11/18/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Zhang, Zhengyou, He, Liwei
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/804,311
• Publication Number:
  US 20050104864A1
• Time in Patent Office: 1,250 Days
• Field of Search: 382/162, 382/165, 382/305, 382/164, 382/176, 382/224, 382/225, 382/278, 382/282, 382202-203, 358/518, 358/1.9, 358/538, 358/1.15, 345/173, 345/427, 345/582, 345/589, 386 69- 70, 386111-112, 386/124, 715/835, 715/722, 715/776, 709224-232, 709/204, 359/725, 359/94, 359/403, 434322-323, 434350-362, 348/273, 348/254, 348/335, 348/239, 348/36, 3482111-21114, 348/207.99, 348/222.1, 348 1401- 1416
• US Class Current: 382/305
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A system that captures both whiteboard content and audio signals of a meeting using a video camera and records or transmits them in real-time. The Real-Time Whiteboard Capture captures pen strokes on whiteboards in real time using an off-the-shelf video camera. Unlike many existing tools, the RTWCS does not instrument the pens or the whiteboard. It analyzes the sequence of captured video images in real time, classifies the pixels into whiteboard background, pen strokes and foreground objects (e.g., people in front of the whiteboard), and extracts newly written pen strokes. This allows the RTWCS to transmit whiteboard contents using very low bandwidth to remote meeting participants. Combined with other teleconferencing tools such as voice conference and application sharing, the RTWCS becomes a powerful tool to share ideas during online meetings.

Claims

1. A computer-implemented process for outputting whiteboard content, comprising the following process actions:

inputting a sequence of image frames of content written on a whiteboard in real-time;

dividing each of said image frames into cells;

for each cell in each image frame,

determining if there is a change in the cell image compared to a correspondingly located cell the immediately preceding image frame in said sequence of image frames;

whenever it is determined that there is a color change, setting a cell age to a prescribed minimum value, and if there is no change increasing cell age by a prescribed increment value;

determining if cell age is greater than a specified threshold value;

whenever the cell age is not greater than the threshold value, not processing said cell any further;

if said cell age is greater than the threshold value,

computing the background color of the cell;

updating a whiteboard color model;

classifying each cell image as a foreground or whiteboard cell using said whiteboard color model;

whenever a cell image is classified as a foreground cell not processing that cell any further; and

whenever a cell image is classified as a whiteboard cell, outputting the cell image whenever it exhibits strokes not present in the correspondingly located cell in the preceding image frames or is missing strokes found in the preceding image frames in real-time.

2. The computer-implemented process of claim 1 wherein the image frames are input in Bayer format.

3. The computer-implemented process of claim 1 further comprising the process action of outputting the image frame cells so that the strokes on the whiteboard are displayed in an enhanced manner using said cell classification.

4. The computer-implemented process of claim 3 wherein the process action of outputting the frame cells so that the strokes on the whiteboard are displayed in an enhanced manner comprises the process actions of:

making the whiteboard background color for cells displaying whiteboard background in said sequence of image frames more uniform;

increasing the stroke saturation of the whiteboard content to make the strokes more vivid and legible; and

reducing image noise in said sequence of image frames.

5. The computer-implemented process of claim 4 wherein the process action of outputting a cell comprises outputting only changed whiteboard cells.

6. The computer-implemented process of claim 1, wherein said cell classifying process action comprises:

comparing the input image frame cell to a previous image in a corresponding cell location by comparing the difference between the background color of the cell image and the corresponding color of the whiteboard model;

if the difference exceeds a given threshold, designating the cell as a foreground cell, and if the difference does not exceed a given threshold designating the cell as a whiteboard cell;

determining if the cell location was a whiteboard cell in a previous output, and if not designating the cell as a new whiteboard cell and outputting its cell image;

if the cell location was not a whiteboard cell in the previous output, determining if the cell image in the same location of the output is the same and if not outputting its cell image.

7. The computer-implemented process of claim 6 wherein the difference between a background color of the cell image and the corresponding color in the whiteboard color model is determined by determining the Euclidean distance between the cell image and the corresponding cell location in the whiteboard color model.

8. The computer-implemented process of claim 6 further comprising the process action of verifying that said cell classified as a whiteboard cell is a whiteboard cell, comprising the process actions of:

determining whether said whiteboard cell is connected to any foreground cells; and

if said whiteboard cell is determined to be connected to any foreground cells, classifying it as a foreground cell.

9. The computer-implemented process of claim 6 further comprising the process action of verifying that said cell classified as a foreground cell is a foreground cell, comprising the process actions of:

determining whether said foreground cell is connected to other foreground cells; and

if a cell is determined to be a foreground cell and not connected to other cells, classifying it as a whiteboard cell.

10. The computer-implemented process of claim 1 wherein the process action of updating the whiteboard color model for the whiteboard and stroke cells comprises the process actions of:

dividing the image of the whiteboard into cells;

sorting the pixels in each cell according to their luminance value; and

assigning the highest luminance value in each cell as the resulting whiteboard color of that cell.

11. The computer-implemented process of claim 10 further comprising the process action of filtering the colors of the cells comprising the following process actions:

subjecting the resulting cell colors to a least-median-square error procedure, which fits a global plane over the colors and throws away the cells that are foreground cells;

designating the remaining cells as whiteboard background cells;

using said remaining whiteboard background cells to update the whiteboard background;

and filling the holes in the whiteboard color model created by throwing out the cells that were foreground cells with known colors from neighboring cells with whiteboard background colors.

12. The computer-implemented process of claim 1 wherein the process action of updating the whiteboard color model comprises the process actions of:

dividing the image of the whiteboard into cells;

sorting the pixels in each cell according to their luminance value; and

averaging the top 10% values of the luminance values in each cell and assigning this value as the resulting whiteboard color of that cell.

13. The computer-implemented process of claim 1 further comprising correlating said audio recordings with said video.

14. The computer-implemented process of claim 1 wherein determining if there is a change in said cell image is computed in Bayer color space.

15. The computer-implemented process of claim 1 wherein the background color of the cell is computed in Bayer color space.

16. The computer-implemented process of claim 1 wherein the whiteboard color model is computed in Bayer color space.

17. The computer-implemented process of claim 1 wherein classifying cell images is computed in Bayer color space.

18. The computer-implemented process of claim 1 wherein determining if cell images are the same comprises using a normalized cross-correlation technique to compare two cells at a time.

19. The computer-implemented process of claim 18 wherein the cross-correlation score ranges from −

1, for two images not similar at all, to 1, for two identical images.

20. The computer-implemented process of claim 19 further comprising a process action of applying a Mahalanobis distance test to determine if two cells are the same.

21. The computer-implemented process of claim 20 wherein the Mahalanobis distance test given by d=|Ī

−

Ī

′

|/(σ

+σ

′

);

where I is the first cell image and I′

is the second cell image, Ī

is the mean color of the first cell image, Ī

′

, is the mean color of the second cell image, σ

is the standard deviation from Ī and

σ

′

is the standard deviation from Ī

′ and

wherein I and I′

are the two cell images are considered to be identical if and only if d<

T_d and C>

T_c, and wherein T_d=2 and T_c=0.707.

22. The computer-implemented process of claim 1 wherein the whiteboard color model is dynamically updated by incorporating the background color from the current frame into the existing whiteboard color model.

23. The computer-implemented process of claim 22 wherein the background color from the whiteboard color model is updated using a Kalman filter technique.

24. The computer-implemented process of claim 1 further comprising the process action of analyzing the sequence of images to isolate key frames summarizing key points of said whiteboard content.

25. A computer-readable medium having computer-executable instructions for transferring whiteboard content captured during a meeting in real-time, said computer executable instructions comprising program modules for:

capturing a sequence of images of content written on a non-electronic whiteboard in Bayer color format;

recording audio signals correlated with the sequence of images; and

analyzing the sequence of images to transmit said images of the whiteboard and audio signals to at least one remote meeting participant by;

dividing each of said image frames into cells of cell images;

determining if there is a change in the cell image compared to a correspondingly located cell the immediately preceding image frame in said sequence of image frames;

whenever it is determined that there is a color change, setting a cell age to a prescribed minimum value, and if there is no change increasing cell age by a prescribed increment value;

determining if cell age is greater than a specified threshold value;

whenever the cell age is not greater than the threshold value, not processing said cell any further;

if said cell age is greater than the threshold value,

computing the background color of the cell;

updating a whiteboard color model;

classifying each cell image as a foreground or whiteboard cell using said whiteboard color model;

whenever a cell image is classified as a foreground cell not processing that cell any further; and

whenever a cell image is classified as a whiteboard cell, outputting the cell image whenever it exhibits strokes not present in the correspondingly located cell in the preceding image frames or is missing strokes found in the preceding image frames in real-time.

26. The computer-readable medium of claim 25 further comprising a module for correcting for exposure if more than a prescribed percentage of the cells have changed.

27. The computer-readable medium of claim 26 wherein the prescribed percentage of cells is 90 percent.

28. The computer-readable medium of claim 26 wherein the module for correcting for exposure automatically sets the appropriate amount of light to be captured to form each image frame in order to avoid over or underexposing resulting image frame.

29. The computer-readable medium of claim 26 wherein the module for correcting for exposure comprises sub-modules for:

initializing the lower and upper limits of a search range of a search algorithm to be the entire range of the exposure setting;

setting the search range to the average of the lower and upper limits;

measuring over-exposure and under-exposure for each incoming image frame;

exiting the module for correcting exposure if a given incoming image frame is neither over-exposed or under-exposed;

if an incoming image frame is over exposed, setting the upper searching limit to the current exposure setting;

if an incoming image frame is under exposed, setting the lower searching limit to the current exposure setting; and

if an incoming image frame is either over exposed or under exposed, setting the exposure of the incoming image frame to the average of the updated lower and upper limits.

30. A computer-implemented process for creating an enhanced data stream content written on a piece of paper, comprising the following process actions:

inputting a sequence of image frames of content written on the document in real-time;

dividing each of said image frames into cells of cell images;

for each cell location in each image frame,

determining if there is a change in said cell image from the cell in the same location in the previous image frame in said sequence of image frames;

if there is a change, setting cell age to a minimum value, and if there is no changes increasing cell age by a prescribed value;

determining if cell age is greater than a specified value;

if cell age is not greater than a specified value, not processing said cell further for said image frame;

if said cell age is greater than a specified value, computing the background color of the cell;

updating a document color model;

classifying cell images into foreground or document cells using said document color model;

if a cell image is classified as a foreground cell not processing that cell any further for that input image frame; and

if a cell image is not classified as a foreground cell, outputting any cell image frames with newly appeared strokes or newly deleted strokes in real-time.