System and method for whiteboard collaboration

• US 8,982,177 B2
• Filed: 07/08/2013
• Issued: 03/17/2015
• Est. Priority Date: 07/08/2013
• Status: Active Grant

• Current Assignee: Arlington Technologies, LLC (Dominion Harbor Enterprises, LLC)
• Original Assignee: Avaya Incorporated
• Inventors: Modai, Ori, Wiener, Yair, Zilbershtein, Itai E.
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US13/936,927
• Publication Number:
  US 20150009278A1
• Time in Patent Office: 617 Days
• Field of Search: 297/135, 348/14.01, 348/14.05, 348/14.08, 348/14.09, 348/14.1, 348/14.12, 370/216, 370/260, 434/362, 705/7.12, 705/301, 707/723, 709/204, 709/205, 709/231, 715/733, 715/751, 715/753, 715/757, 719/323
• US Class Current: 348/14.08
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Disclosed is a method and tool for collaborating in a video conference using a non-electronically enhance whiteboard and without the use of electronically enhanced writing and/or pointing tools. An embodiment may set up a front camera to capture an overall video stream that is split into at least two video feeds, a whiteboard feed and presenter feeds. The front camera may be set up to have a depth of field and focus to capture the whiteboard and presenter as well as specific video capture settings for each feed to optimize the whiteboard or the presenter. An embodiment may capture the two feeds according to the appropriate settings and deliver the two feeds to other video conference member devices. Embodiments may add additional cameras to permit additional optimization such as eliminating whiteboard occluded areas and/or automatically selecting a presenter from a plurality of people who are in the video capture frame.

Claims

1. A method to collaborate in a video conference when a presenter uses a non-electronically enhanced writing surface, referred to herein as a whiteboard, and without using electronically enhanced writing and/or pointing tools, using a front camera that is operatively connected to a video processing-communication unit and is placed in front of said whiteboard in order to capture video of said whiteboard and said presenter using said whiteboard, said video conference collaboration method comprising:

setting by said video processing-communication unit a depth of field and a focus for said front camera such that said front camera captures an image of said whiteboard and a predefined distance in front of said whiteboard as a presenter'"'"'s area;

setting by said video processing-communication unit said front camera to capture an overall front camera video stream as at least two video streams, a whiteboard video stream and a presenter video stream such that said whiteboard video stream and said presenter video stream are portions of said overall front camera video stream;

setting by said video processing-communication unit whiteboard video capture settings of said whiteboard video stream on said front camera to optimize capture of said whiteboard;

setting by said video processing-communication unit presenter video capture settings of said presenter video stream on said front camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard;

capturing by said video processing-communication unit said whiteboard video stream and said presenter video stream using said front camera according to said depth of field, said focus, said whiteboard video capture settings, and said presenter video capture settings; and

delivering by said video processing-communication unit a final conference whiteboard video stream based on said whiteboard video stream and a final conference presenter video stream based on said presenter video stream to member devices of said video conference.

2. The method of claim 1 wherein said process of capturing by said video processing-communication unit said whiteboard video stream and said presenter video stream of said presenter using said front camera further comprises designating a whiteboard video stream portion of said overall front camera video stream as said whiteboard video stream and a presenter video stream portion of said overall front camera video stream as said presenter video stream such that when said front camera is capturing said whiteboard video stream portion of said overall front camera video stream, said front camera operates according to said whiteboard video capture settings, and when said front camera is capturing said presenter video stream portion of said overall front camera video stream, said front camera operates according to said presenter video capture settings.

3. The method of claim 1 wherein at least one additional camera is operatively connected to said video processing-communication unit, said at least one additional camera is placed such that said at least one additional camera captures video of said whiteboard and said presenter from a different angle than said front camera, said method of claim 1 further comprising:

setting by said video processing-communication unit an additional camera depth of field and an additional camera focus for said at least additional camera such that said at least one additional camera captures an image of said whiteboard and said predefined distance in front of said whiteboard as said presenter'"'"'s area;

setting by said video processing-communication unit said at least one additional camera to capture an overall additional camera video stream as at least two video streams, an additional camera whiteboard video stream and an additional camera presenter video stream such that said additional camera whiteboard video stream and said additional camera presenter video stream are portions of said overall additional camera video stream;

setting by said video processing-communication unit additional camera whiteboard video capture settings of said additional camera whiteboard video stream on said at least one additional camera to optimize capture of said whiteboard;

setting by said video processing-communication unit additional camera presenter video capture settings of said additional camera presenter video stream on said at least one additional camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard;

capturing by said video processing-communication unit said additional camera whiteboard video stream and said additional camera presenter video stream using said at least one additional camera according to said additional camera depth of field, said additional camera focus, said additional camera whiteboard video capture settings, and said additional camera presenter video capture settings;

aggregating by said video processing-communication unit said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one additional camera in order to produce an optimized whiteboard video stream;

aggregating by said video processing-communication unit said additional camera presenter video stream with said presenter video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one additional camera in order to produce an optimized presenter video stream; and

wherein said process of delivering by said video processing-communication unit said whiteboard video stream and said presenter video stream to member devices of said video conference delivers said optimized whiteboard video stream as said final conference whiteboard video stream and said optimized presenter video stream as said final conference presenter video stream.

4. The method of claim 1 further comprising:

optimizing said whiteboard video stream by said video processing-communication unit prior to said process of delivering by said video processing-communication unit said whiteboard video stream and said presenter video stream as said final conference whiteboard video stream and said final conference presenter video stream to member devices of said video conference;

wherein said optimization is at least one of a group consisting of;

image processing to transform captured images of said whiteboard to a rectangular shape, enhancing said captured images of said whiteboard to improve lines and shapes contrast, and balancing colors of said captured images of said whiteboard to obtain an authentic whiteboard color.

5. The method of claim 1 further comprising:

detecting by said video processing-communication unit that portions of said whiteboard in said whiteboard video stream are occluded; and

replacing by said video processing-communication unit said detected occluded portions of said whiteboard in said whiteboard video stream with previously composed frames of said whiteboard video stream not having said detected occluded portions of said whiteboard in order to complete missing portions of said whiteboard in said whiteboard video stream.

6. The method of claim 3 further comprising:

detecting by said video processing-communication unit that portions of said whiteboard in said whiteboard video stream are occluded; and

wherein said process of aggregating by said video processing-communication unit said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one side video camera in order to produce an optimized whiteboard video stream further comprises replacing by said video processing-communication unit said detected occluded portions of said whiteboard in said whiteboard video stream with frames of said additional camera whiteboard video stream that show said portions of said whiteboard detected as being occluded in order to complete missing portions of said whiteboard in said whiteboard video stream.

7. The method of claim 3 further comprising:

detecting by said video processing-communication unit that portions of said whiteboard in said whiteboard video stream are distorted by specular light and reflections; and

wherein said process of aggregating by said video processing-communication unit said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one side video camera in order to produce an optimized whiteboard video stream further comprises replacing by said video processing-communication unit said detected specular light and reflections distorted portions of said whiteboard in said whiteboard video stream with frames of said additional camera whiteboard video stream that show said specular light and reflections distorted portions of said whiteboard without distortions in order to compensate for said y said detected specular light and reflections in said whiteboard video stream.

8. The method of claim 3 wherein said process of aggregating by said video processing-communication unit said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one side video camera in order to produce an optimized whiteboard video stream further comprises:

aligning said whiteboard video stream and said additional camera whiteboard stream according to borders of said whiteboard; and

composing said optimized whiteboard video stream based on said alignment of said borders of said whiteboard between said whiteboard video stream and said additional camera whiteboard video stream.

9. The method of claim 1 further comprising:

automatically selecting by said video processing-communication unit said presenter from a plurality of people whose image is captured by at least said front camera; and

wherein at least said process of setting by said video processing-communication unit presenter video capture settings of said presenter video stream on said front camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard optimizes for said automatically selected presenter.

10. The method of claim 9 wherein said process of automatically selecting by said video processing-communication unit said presenter from said plurality of people whose image is captured by at least said front camera further comprises:

detecting faces of each person of said plurality of people whose image is captured by at least said front camera using face detection analysis;

extending face detection of each person of said plurality of people whose image is captured by at least said front camera to include an entire available image of each person of said plurality of people using edge detection analysis;

estimating proximity to said whiteboard of each detected person of said plurality of people whose image is captured by at least said front camera;

analyzing gaze direction for each detected person of said plurality of people whose image is captured by at least said front camera;

analyzing gesture direction for each detected person of said plurality of people whose image is captured by at least said front camera; and

determining a person of said plurality of people whose image is captured by at least said front camera is said presenter based on at least one of a group of characteristics for each person of said plurality of people consisting of;

proximity to said whiteboard, gaze direction, gesture direction, and persistency as said presenter.

11. The method of claim 10 wherein said process of estimating proximity to said whiteboard of each detected person of said plurality of people whose image is captured by at least said front camera further comprises using a depth sensing camera to obtain said estimate of proximity to said whiteboard.

12. The method of claim 10 wherein said process of estimating proximity to said whiteboard of each detected person of said plurality of people whose image is captured by at least said front camera further comprises:

identifying a specific location point on a contour of each detected person of said plurality of people whose image is captured by at least said front camera;

projecting said specific location point to a plane on said whiteboard for each detected person of said plurality of people whose image is captured by at least said front camera;

comparing projected points of said specific location point for two or more cameras capturing images from two or more camera angles; and

estimating a depth of each detected person of said plurality of people whose image is captured by at least said front camera by triangulation of said specific location point from said two or more cameras at two or more camera angles.

13. A whiteboard video conference collaboration system comprising:

a front camera placed in front of a non-electronically enhanced writing surface, referred to herein as a whiteboard, in order to capture video of said whiteboard and a presenter using said whiteboard;

a video processing-communication unit that is operatively connected to said front camera, that sets a depth of field and a focus for said front camera such that said front camera captures an image of said whiteboard and a predefined distance in front of said whiteboard as a presenter'"'"'s area, that sets said front camera to capture an overall front camera video stream as at least two video streams, a whiteboard video stream and a presenter video stream, such that said whiteboard video stream and said presenter video stream are portions of said overall front camera video stream, that sets whiteboard video capture settings of said whiteboard video stream on said front camera to optimize capture of said whiteboard, that sets presenter video capture settings of said presenter video stream on said front camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard, that captures said whiteboard video stream and said presenter video stream using said front camera according to said depth of field, said focus, said whiteboard video capture settings, and said presenter video capture settings, and delivers a final conference whiteboard video stream based on said whiteboard video stream and a final conference presenter video stream based on said presenter video stream to member devices of said video conference.

14. The whiteboard video conference collaboration system of claim 13 wherein said video processing-communication unit further designates a whiteboard video stream portion of said overall front camera video stream as said whiteboard video stream and a presenter video stream portion of said overall front camera video stream as said presenter video stream such that when said front camera is capturing said whiteboard video stream portion of said overall front camera video stream, said front camera operates according to said whiteboard video capture settings, and when said front camera is capturing said presenter video stream portion of said overall front camera video stream, said front camera operates according to said presenter video capture settings.

15. The whiteboard video conference collaboration system of claim 13 further comprising:

at least one additional camera is operatively connected to said video processing-communication unit, said at least one additional camera being placed such that said at least one additional camera captures video of said whiteboard and said presenter from a different angle than said front camera; and

wherein said video processing-communication unit further sets an additional camera depth of field and an additional camera focus for said at least additional camera such that said at least one additional camera captures an image of said whiteboard and said predefined distance in front of said whiteboard as said presenter'"'"'s area, sets said at least one additional camera to capture an overall additional camera video stream as at least two video streams such that said additional camera whiteboard video stream and said additional camera presenter video stream are portions of said overall additional camera video stream, an additional camera whiteboard video stream and an additional camera presenter video stream, sets additional camera whiteboard video capture settings of said additional camera whiteboard video stream on said at least one additional camera to optimize capture of said whiteboard, sets additional camera presenter video capture settings of said additional camera presenter video stream on said at least one additional camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard, captures said additional camera whiteboard video stream and said additional camera presenter video stream using said at least one additional camera according to said additional camera depth of field, said additional camera focus, said additional camera whiteboard video capture settings, and said additional camera presenter video capture settings, aggregates said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one additional camera in order to produce an optimized whiteboard video stream, aggregates said additional camera presenter video stream with said presenter video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one additional camera in order to produce an optimized presenter video stream, and delivers said optimized whiteboard video stream as said final conference whiteboard video stream and said optimized presenter video stream as said final conference presenter video stream.

16. The whiteboard video conference collaboration system of claim 13 wherein said video processing-communication unit further optimizes said whiteboard video stream prior to delivery of said whiteboard video stream and said presenter video stream as said final conference whiteboard video stream and said final conference presenter video stream to member devices of said video conference such that said optimization is at least one of a group consisting of:

image processing to transform captured images of said whiteboard to a rectangular shape, enhancing said captured images of said whiteboard to improve lines and shapes contrast, and balancing colors of said captured images of said whiteboard to obtain an authentic whiteboard color.

17. The whiteboard video conference collaboration system of claim 13 wherein said video processing-communication unit further detects that portions of said whiteboard in said whiteboard video stream are occluded, and replaces said detected occluded portions of said whiteboard in said whiteboard video stream with previously composed frames of said whiteboard video stream not having said detected occluded portions of said whiteboard in order to complete missing portions of said whiteboard in said whiteboard video stream.

18. The whiteboard video conference collaboration system of claim 15 wherein said video processing-communication unit further detects that portions of said whiteboard in said whiteboard video stream are occluded, and said aggregation of said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one side video camera in order to produce an optimized whiteboard video stream further replaces said detected occluded portions of said whiteboard in said whiteboard video stream with frames of said additional camera whiteboard video stream that show said portions of said whiteboard detected as being occluded in order to complete missing portions of said whiteboard in said whiteboard video stream.

19. The whiteboard video conference collaboration system of claim 15 wherein said video processing-communication unit further detects that portions of said whiteboard in said whiteboard video stream are distorted by specular light and reflections, and said aggregation of said additional camera whiteboard video stream with said whiteboard video stream in order to optimize images presented based on at least two camera angles of said front camera and said at least one side video camera in order to produce an optimized whiteboard video stream further replaces said detected specular light and reflections distorted portions of said whiteboard in said whiteboard video stream with frames of said additional camera whiteboard video stream that show said specular light and reflections distorted portions of said whiteboard without distortions in order to compensate for said y said detected specular light and reflections in said whiteboard video stream.

20. The whiteboard video conference collaboration system of claim 13 wherein said video processing-communication unit further automatically selects said presenter from a plurality of people whose image is captured by at least said front camera, and said presenter video capture settings of said presenter video stream on at least said front camera to optimize capture of said presenter in said presenter'"'"'s area in front of said whiteboard is optimized for said automatically selected presenter.