System and method for real-time whiteboard streaming

• US 20040165768A1
• Filed: 06/17/2003
• Published: 08/26/2004
• Est. Priority Date: 02/24/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: He, Li-Wei, Zhang, Zhengyou
• Granted Patent: US 7,224,847 B2
• Time in Patent Office: Days
• US Class Current: 382/162
• CPC Class Codes: H04L 65/1101 Session protocols
• CPC Class Codes: H04L 65/4046 with distributed floor control
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A system and method for streaming whiteboard content to computing devices in a networked environment. The invention is an extension of whiteboard image generation technology to provide network-based collaboration of a target meeting. In one aspect, each networked client can receive audio content and whiteboard content (video images). In another aspect, each networked client can transmit audio content and annotation content which is displayed separately or generated on the whiteboard image. The streaming content is built on external collaboration frameworks.

Claims

1. A computer-implemented process for creating an enhanced data stream of whiteboard content, comprising the process actions of:

inputting a sequence of image frames of a whiteboard;

dividing each of said image frames of said whiteboard into cells;

determining if there are significant lighting changes in said cells in said sequence of images;

if there are significant lighting changes, performing a whiteboard color matching process to determine if the color of the whiteboard reflects a color of the whiteboard in a whiteboard color model database, said whiteboard color being used to define the whiteboard background color;

if there are no significant lighting changes, classifying each of the cells of the image frames as foreground, whiteboard background or stroke cells of content written on the whiteboard;

enhancing the sequence of image frames so that the strokes on the whiteboard are displayed in an enhanced manner using said cell classification; and

outputting a sequence of enhanced image frames of said whiteboard in a data stream.

2. The computer-implemented process of claim 1 wherein the process action of enhancing the sequence of image frames so that the strokes on the whiteboard are displayed in an enhanced manner comprises the process actions of:

making the whiteboard background color for cells displaying whiteboard background in said sequence of image frames more uniform;

reducing image noise in said sequence of image frames; and

increasing the stroke saturation of the whiteboard content to make the strokes more vivid and legible for stroke cells.

3. The computer-implemented process of claim 1 wherein the process action of outputting a sequence of enhanced image frames of said whiteboard in a data stream comprises using only enhanced whiteboard cells and enhanced stroke cells, and no foreground cells.

4. The computer-implemented process of claim 1 wherein the process action of outputting a sequence of enhanced image frames of said whiteboard in a data stream comprises using enhanced whiteboard cells, enhanced stroke cells and foreground cells.

5. The computer-implemented process of claim 1, wherein said whiteboard color matching process action comprises:

determining whether or not the color of the whiteboard background cells in the new lighting condition match a whiteboard color model in a whiteboard color model database;

if the color of the whiteboard background cells in the new lighting condition corresponds to a whiteboard color model reflecting the color of the whiteboard background cells in said whiteboard color model database, using this whiteboard color model for subsequent cell classification;

if the color of the whiteboard background cell in the new lighting condition is not found is said whiteboard color model database, creating a new whiteboard color model reflecting the color of the whiteboard background cells of the whiteboard in the new lighting condition.

6. The computer-implemented process of claim 5 further comprising the process action of:

adding said new whiteboard color model to the whiteboard color model database.

7. The computer-implemented process of claim 1 further comprising the process action of determining if strokes have been added or subtracted from the whiteboard.

8. The computer-implemented process of claim 7 wherein the process action of determining whether strokes have been added or subtracted comprises the process action of determining whether or not the stroke content in each cell increases or decreases compared to correspondingly located cells in said sequence of image frames.

9. The computer-implemented process of claim 7 further comprising the process action of outputting a cell if there are changes in said cell to said sequence of enhanced images by displaying only enhanced stroke cells and enhanced whiteboard background cells and no foreground cells.

10. The computer-implemented process of claim 9 wherein enhanced whiteboard and enhanced stroke cells are output in said sequence of enhanced images until it is determined that strokes have been added or subtracted, and wherein the enhanced stroke cells corresponding to the strokes that have been added or subtracted are then output when changes to stroke content occur.

11. The computer-implemented process of claim 1 wherein the process action of classifying each of the cells of the image frames as foreground, whiteboard background or stroke cells of content written on the whiteboard, comprises the following process actions:

for all image frame cells in an image frame in said sequence of images,

inputting an image frame'"'"'s cell;

comparing said image frame'"'"'s cell against the cells in the corresponding location in previous frames;

if the cell age based on the number of frames that the cell is unchanged is not greater than an age threshold, classifying the cell as a foreground cell;

if the cell age is greater than an age threshold, determining if there is any significant difference in cell color,

if there is not any significant difference in cell color, classifying the cell as a foreground cell;

if there is significant difference in cell color, determining whether said cell contains edges;

if said cell does contain edges, classifying said cell as a stroke cell; and

if said cell does not contain edges, classifying said cell as a whiteboard background cell.

12. The computer-implemented process of claim 11 further comprising the process action of verifying that said cell classified as a foreground cell is a foreground cell, comprising the process actions of:

determining whether said foreground cell is connected to other foreground cells; and

if a cell is determined to be a foreground cell and not connected to other cells, reverting its classification to unknown.

13. The computer-implemented process of claim 11 wherein the process action of determining if there is any significant difference in cell color, comprises the process action of:

examining the Y, U, V channels of the cell;

if the Y, U, V channels of said cell color are not within 15, 5, and 5 intensity levels from the correspondingly located cells in said whiteboard color model respectively, specifying that there is a significant difference in cell color.

14. The computer-implemented process of claim 12 wherein a foreground cell is not connected to other foreground cells if in a 5×

5 cell neighborhood, there are less than 6 foreground cells.

19. The computer-implemented process of claim 1 further comprising a process action of determining whether a whiteboard background color model should be updated after the process action of classifying each of the cells of the image frames as foreground, whiteboard background or stroke cells of content written on the whiteboard.

20. The computer-implemented process of claim 19 wherein the process action of determining whether the whiteboard background color model should be updated comprises the process actions of:

identifying any gradual changes in the whiteboard color by examining the average color of the corresponding cells in the past sequence of image frames leading up to the current frame of a whiteboard; and

if gradual changes are identified in the whiteboard color, determining whether the current whiteboard color model in a whiteboard color model database of all whiteboard color models available should be updated.

21. The computer-implemented process of claim 20 wherein if average cell color is almost the same, but not the same, as in previous frames, further comprising the process actions of:

determining that a gradual lighting change has occurred; and

updating the current whiteboard background color model for the whiteboard background and stroke cells having a whiteboard background color.

22. The computer-implemented process of claim 21 wherein the process action of updating the whiteboard color model for the whiteboard and stroke cells comprises the process actions of:

dividing the image of the whiteboard into cells;

sorting the pixels in each cell according to their luminance value; and

assigning the highest luminance value in each cell as the resulting whiteboard color of that cell.

23. The computer-implemented process of claim 22 further comprising the process action of filtering the colors of the cells comprising the following process actions:

subjecting the resulting cell colors to a least-median-square error procedure, which fits a global plane over the colors and throws away the cells that are foreground cells;

designating the remaining cells as whiteboard background cells;

using said remaining whiteboard background cells to update the whiteboard background;

and filling the holes in the whiteboard color model created by throwing out the cells that were foreground cells with known colors from neighboring cells with whiteboard background colors.

24. The computer-implemented process of claim 22 wherein the cell size should is approximately the same as what the size of a single character on the whiteboard is expected to be.

25. The computer-implemented process of claim 21 wherein the process action of updating the whiteboard color model for the whiteboard and stroke cells comprises the process actions of:

dividing the image of the whiteboard into cells;

sorting the pixels in each cell according to their luminance value; and

averaging the top 10% values of the luminance values in each cell and assigning this value as the resulting whiteboard color of that cell.

26. The computer-implemented process of claim 25 wherein the cell size should is approximately the same as what the size of a single character on the whiteboard is expected to be.

27. The computer-implemented process of claim 21 wherein the process action of updating the whiteboard background color comprises:

taking a greater percentage of the last whiteboard background color model and a smaller percentage of the new whiteboard background color and summing these to obtain a new whiteboard background color.

28. The computer-implemented process of claim 27 wherein the greater percentage of the last whiteboard background color model is 90% and the small percentage of the new whiteboard background color is 10%.

29. The computer-implemented process of claim 1 wherein the process action of determining if there are significant lighting changes in said cells comprises the process action of:

determining if a majority of the cells exhibit a color change and if so designating that a lighting change has occurred.

30. The computer-implemented process of claim 29 wherein as majority is defined as 95% percent of the cells exhibit a color change.

15. The computer-implemented process of 12 wherein the process action of determining whether a cell contains edges comprises the process actions of:

for each frame'"'"'s cells,

running a Sobel filter to identify horizontal edges in said cell to obtain a first filtered Sobel image;

running a Sobel filter to identify vertical edges in said cell to obtain a second filtered Sobel image;

determining the number of edgels that exist in each frame wherein an edgel is defined to be the sum of the absolute values of the corresponding pixels in the two Sobel filtered images;

for each cell, that is stationary for N frames, wherein

a cell is stationary from successive frames N to N+1 when all pixels in the cell of frame N and N+1 are the same,

and contains an edgel number whose value is greater than a prescribed threshold,

considering it to be a stroke cell.

18. The computer-implemented process of claim 15 wherein said edgels in said stroke cells define the edges of the stroke.

16. The computer-implemented process of 15 wherein all the pixels in a cell of frame N and N+1 are the same if the average absolute difference of all pixels is below a prescribed threshold.

17. The computer-implemented process of claim 16 wherein the prescribed threshold is defined in terms of an intensity level.

31. A system for streaming whiteboard data content, the system comprising;

a classification module for classifying each cell of a sequence of whiteboard images that are divided into cells, as a stroke cell, a whiteboard background cell or a foreground cell;

a dynamic whiteboard background and initialization module that is used to determine the color of the whiteboard background color in said cells;

an image enhancement module for enhancing images comprised of whiteboard background, foreground and stroke cells; and

a module for adding and erasing whiteboard stroke data for creating images of the whiteboard using whiteboard background and stroke cells.

32. The system of claim 31 further comprising a module for streaming whiteboard image data wherein said whiteboard image data is comprised of whiteboard background cells, foreground cells and stroke cells.

33. The system of claim 31 further comprising a module for streaming whiteboard content data, wherein said whiteboard content data is comprised of whiteboard background cells and stroke cells.

34. The system of claim 31 wherein the module for classifying whiteboard content data comprises sub-modules for:

for all cells in said sequences of images of a whiteboard,

inputting an image frame cell;

comparing said image frame cell against the image of cells in the same location in previous frames;

if the cell age is not greater than an age threshold classifying the cell as a foreground cell;

designating all the cells that have been stationary for more than N frames as background candidates;

determining if there is any significant difference in cell color and if there is any significant difference in the cell color, classifying the cell as a foreground cell;

determining whether a cell classified as a foreground cell is connected to other foreground cells;

if a cell is classified as a foreground cell and not connected to other cells, reverting its classification to unknown;

determining whether said cell contains edges;

if said cell does contain edges, classifying said cell as a stroke cell; and

if said cell does not contain edges, classifying said cell as a whiteboard background cell.

35. The system of claim 31 wherein some but not all images recorded in a video stream of a whiteboard are used to create said sequence of images.

36. The system of claim 31 wherein all images recorded in a video stream of a whiteboard are used to create said sequence of images.

37. A computer-readable medium having computer-executable instructions for transmitting a video stream of a whiteboard, said computer executable instructions comprising:

inputting a sequence of image frames of a whiteboard;

dividing each of said image frames of said whiteboard into cells;

determining if there are significant lighting changes in said cells in said sequence of images;

if there are significant lighting changes, performing a whiteboard color matching process to determine the color of the whiteboard;

if there are no significant lighting changes, classifying each of the cells of the image frames as foreground, whiteboard background or stroke cells of content written on the whiteboard;

enhancing the sequence of image frames so that only the strokes on the whiteboard are displayed in an enhanced manner using said cell classification.

38. The computer-readable medium of claim 37 further comprising a computer-executable instruction for outputting said enhanced sequence of images using only said whiteboard background and stroke cells.

39. The computer-readable medium of claim 37 further comprising a computer-executable instruction for outputting said enhanced sequence of images using only said whiteboard background, foreground cells and stroke cells.

40. A system for transmitting a video stream of enhanced whiteboard images, the system comprising:

a sender, comprising,

an input module for inputting a sequence of captured whiteboard images and audio synchronized with said sequence of captured whiteboard images, and dividing each of said images into cells;

a classification module for classifying whiteboard image data as stroke cells, whiteboard background cells or foreground cells;

a dynamic whiteboard background and initialization module;

an image enhancement module fore enhancing strokes in said stroke cells and making whiteboard background cell color more uniform; and

a module for adding and erasing whiteboard stroke data;

an transmission module that transmits a first data stream of enhanced whiteboard images that include foreground objects using said foreground cells, whiteboard background cells and said stroke cells, and a second data stream of enhanced whiteboard images that do not include foreground objects, using said stroke cells and whiteboard background cells but no foreground cells to a receiver over a network.

41. The system of claim 40, further comprising a receiver, said receiver comprising:

a receiving module that receives said enhanced whiteboard images and synchronized audio sent from said sender over said network;

a speaker module that plays said synchronized audio; and

a display module that displays said enhanced whiteboard images at the receiver.

42. The system of claim 41, wherein said receiver, further comprises a module for selecting which of the first data stream or the second data stream said receiver wants to.display on said display module.

43. The system of claim 41, wherein said receiver, further comprises a module for sending annotations to said whiteboard data stream back to said sender.

44. The system of claim 43, wherein said sender displays said annotations sent by said receiver on said whiteboard by superimposing said annotations on said whiteboard by projecting said annotations onto said whiteboard with a projector.

45. The system of claim 43, wherein said sender displays said annotations on said whiteboard by superimposing said annotations on said whiteboard on a separate display, wherein both said whiteboard image and said annotations are displayed.

46. The system of claim 40 wherein said sender further comprises a module for archiving said first and second data streams for later viewing or transmission.

47. The system of claim 40 wherein said dynamic whiteboard background estimation and initialization module, estimates whiteboard color by:

for a first input image,

initializing whiteboard background color for each pixel, and associating each pixel with a covariance estimate that characterizes the accuracy of said pixel color, designating the initialized whiteboard color as the estimated whiteboard color;

given an input image, comparing estimated color of the whiteboard and the actual observed color in said input image at each pixel, and

if the estimated color and the observed color are very different, the observed pixel is a part of either a stroke or a foreground object, and the color of that pixel is discarded,

otherwise, the color of that pixel is an observation of the whiteboard color, to be used in updating the whiteboard color; and

updating the color of those pixels that were not discarded under a probabilistic framework, wherein each pixel is modified based on how reliable the current observed pixel is with respect to the estimated color, how different the current observed pixel is from the estimated color, and how different the observed color of the pixel is from the colors in the neighboring pixels.

48. The system of claim 47 wherein the covariance matrix of the color estimate at each pixel is also updated when the pixel colors are updated.

49. The system of claim 47 wherein said dynamic whiteboard background estimation and initialization module, estimates whiteboard color by:

for a subsequently input image after said first input image,

comparing the updated estimated color of the whiteboard and the actual observed color in said input image at each pixel, and

if the updated estimated color and the observed color are very different, the observed pixel is a part of either a stroke or a foreground object, and the color of that pixel will be discarded;

otherwise, the color of that pixel is an observation of the whiteboard color, to be used in updating the whiteboard color; and

updating the color of those pixels that were not discarded under a probabilistic framework, wherein each pixel is modified based on how reliable the current observed pixel is with respect to the estimated color, how different the current observed pixel is from the estimated color, and how different the observed color of the pixel is from the colors in the neighboring pixels.

50. The system of claim 40 wherein said dynamic whiteboard background estimation and initialization module, estimates whiteboard color by:

for a first input image,

dividing a whiteboard into cells;

initializing whiteboard background color for each cell, and associating each cell with a covariance estimate that characterizes the accuracy of said cell color, designating the initialized whiteboard color as the estimated whiteboard color;

given an input image, comparing estimated color of the whiteboard and the actual observed color in said input image at each cell, and

if the estimated color and the observed color are very different, the observed cell is a part of either a stroke or a foreground object, and the color of that cell is discarded;

otherwise, the color of that cell is an observation of the whiteboard color, to be used in updating the whiteboard color; and

updating the color of those cells that were not discarded under a probabilistic framework, wherein each cell is modified based on how reliable the current observed cell is with respect to the estimated color, how different the current observed cell is from the estimated color, and how different the observed color of the cell is from the colors in the neighboring cells.

51. The system of claim 50 wherein the covariance matrix of the color estimate at each cell is also updated when the cell colors are updated.

52. The system of claim 50 wherein said dynamic whiteboard background estimation and initialization module, estimates whiteboard color by:

for a subsequently input image after said first input image,

comparing the updated estimated color of the whiteboard and the actual observed color in said input image at each cell, and

if the updated estimated color and the observed color are very different, the observed cell is a part of either a stroke or a foreground object, and the color of that cell will be discarded;

otherwise, the color of that cell is an observation of the whiteboard color, to be used in updating the whiteboard color; and

updating the color of those cells that were not discarded under a probabilistic framework, wherein each cell is modified based on how reliable the current observed cell is with respect to the estimated color, how different the current observed cell is from the estimated color, and how different the observed color of the cell is from the colors in the neighboring cells.

53. A computer-implemented process for estimating whiteboard color dynamically from a sequence of images in real time, comprising the process actions of:

defining the state variable x_i,j as the color of the whiteboard at each cell (i, j) to be estimated, and the observation variable z_i,j as the color of the whiteboard at each cell (i;

j) observed from an image;

calculating the color in each cell as,

x_i,j^t=({circumflex over (P)}_i,j^t\t-1)<sup>−

1</sup>+(Q_i,j^t)<sup>−

1</sup>+(S_i,j^t)<sup>−

1</sup>)<sup>−

1</sup>({circumflex over (P)}_i,j^t\t-1)<sup>−

1</sup>{circumflex over (x)}_i,j^t\t-1+(Q_i,j^t)<sup>−

1</sup>z_i,j^t+(S_i,j^t)<sup>−

1</sup>{overscore (x)}_i,j^t

where for a first iteration, the mean of the predictions is used

$\frac{1}{\mathrm{\Omega <br><br>}}<br><br>\underset{\left(k,l\right)\ne <br><br>\left(i,j\right)}{\sum <br><br>}<br><br>x_{k,i}^{t<br><br>t-1}$

as {overscore (x)}_i,j^t where Ω

is the number of neighboring cells, k∈

[i−

1,i+1], l∈

[j−

1, j+1], (k,l)≠

(i,j), and in subsequent iterations, the mean of the estimates from the previous iteration are used; and

wherein the covariance of x_i,j^t is given by

P_i,j^t=(({circumflex over (P)}_i,j^t\t-1)<sup>−

1</sup>+(Q_i,j^t)<sup>−

1</sup>+(S_i,j^t)<sup>−

1</sup>)<sup>−

1</sup>)(2I+(S_i,j^t)<sup>−

1</sup>{overscore (P)}_i,j^t)

where I is the identity matrix, and

${\stackrel{\_}{P}}_{i,j}^t=\underset{\left(k,l\right)\ne <br><br>\left(i,j\right)}{\sum <br><br>}<br><br>P_{k,l}^t$

and the mean of the predictions,

$\frac{1}{\mathrm{\Omega <br><br>}}<br><br>\underset{\left(k,l\right)\ne <br><br>\left(i,j\right)}{\sum <br><br>}<br><br>{\hat{P}}_{k,l}^{t<br><br>t-1}$

is used as {overscore (P)}_i,j^t in the first iteration, and where the observation covariance is Q_i,j^t, the smoothness covariance matrix is S_i,j^t, the state transition noise is η

_i,j^t and its covariance matrix is R_i,j^t.