Video-teleconferencing system with eye-gaze correction

• US 6,771,303 B2
• Filed: 04/23/2002
• Issued: 08/03/2004
• Est. Priority Date: 04/23/2002
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Yang, Ruigang, Zhang, Zhengyou
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US10/128,888
• Publication Number:
  US 20030197779A1
• Time in Patent Office: 833 Days
• Field of Search: H04/N.714, 348/14-1-, 348/14.1, 348/14.11, 348/14.12, 348/14.13, 348/14.16, 382/103, 382/117, 382/118
• US Class Current: 348/14.16
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact

Abstract

Correcting for eye-gaze in video communication devices is accomplished by blending information captured from a stereoscopic view of the conferee and generating a virtual image of the conferee. A personalized face model of the conferee is captured to track head position of the conferee. First and second video images representative of a first conferee taken from different views are concurrently captured. A head position of the first conferee is tracked from the first and second video images. Matching features and contours from the first and second video images are ascertained. The head position as well as the matching features and contours from the first and second video images are synthesized to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.

Claims

1. A method, comprising:

concurrently capturing first and second video images representative of a first conferee taken from different views;

tracking a head position of the first conferee from the first and second video images;

ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images approximated by assigning polygonal lines; and

synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.

2. The method as recited in claim 1, further comprising storing a personalized face model of the first conferee.

3. The method as recited in claim 1, further comprising storing a personalized face model of the first conferee and evaluating the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.

4. The method as recited in claim 1, wherein ascertaining features from the first video image that matches features from the second video image comprises rectifying the first and second video images and locating features from the first and second video images that reside on epipolar lines that match.

5. The method as recited in claim 1, wherein ascertaining contours from the first video image that matches contours from the second video image comprises rectifying the first and second video images and locating contours from the first and second video images that reside on epipolar lines that match.

6. The method as recited in claim 1, wherein the synthesizing of the head position the features and the contours from the first and second video images comprises morphing the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee.

7. The method as recited in claim 1, wherein synthesizing the head position, the features, and the contours from the first and second video images comprises blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

8. One or more computer-readable media comprising computer- executable instructions that, when executed, perform the method as recited in claim 1.

9. A method, comprising:

storing a personalized face model of a first conferee;

concurrently capturing first and second video images representative of the first conferee taken from different views;

evaluating the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and

synthesizing the three dimensional information to generate a virtual image video stream of the first conferee.

10. The method as recited in claim 9, wherein evaluating the first and second video images with respect to the personalized face model of the first conferee further comprises tracking feature points from the first and second video images to monitor a head position of the first conferee.

11. The method as recited in claim 9, further comprising ascertaining features and contours from the first video image that match features and contours from the second video image.

12. The method as recited in claim 9, further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information wit features and contours to generate the virtual image video stream of the first conferee.

13. The method as recited in claim 9, further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee, and wherein synthesizing the three dimensional information, the features, and the contours from the first and second video images comprises morphing the head, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

14. The method as recited in claim 9, further comprising ascertaining features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee, and wherein synthesizing the three dimensional information, the features, and the contours from the first and second video images comprises blending multi-texture features associated with the head positions as the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

15. One or more computer-readable media comprising computer- executable instructions that, when executed, perform the method as recited in claim 9.

16. A system, comprising:

means for concurrently capturing first and second video images representative of a first conferee taken from different views;

means for tracking a head position of the first conferee from the first and second video images;

means for ascertaining features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and

means for synthesizing the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.

17. The system as recited in claim 16, further comprising means for storing a personalized face model of the first conferee in a memory device.

18. The system as recited in claim 16, further comprising means for storing a personalized face model of the first conferee in a memory device and means for evaluating the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.

19. The system as recited in claim 16, wherein the means for ascertaining features from the first video image that matches features from the second video image comprises means for rectifying the first and second video images and means for locating features from the first and second video images that reside an epipolar lines that match.

20. The system as recited in claim 16, wherein the means for ascertaining contours from the first video image that matches contours from the second video image comprises means for rectifying the first and second video images and means for locating contours from the first and second video images that reside on epipolar lines that match.

21. The system as recited in claim 16, wherein the means for synthesizing the head position, the features, and the contours from the first and second video images comprises means for morphing the head positions, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

22. The system as recited in claim 16, wherein the means for synthesizing the head position, the features, and the contours from the first and second video images comprises means for blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

23. A video-teleconferencing system, comprising:

a head pose tracking module, configured to receive first and second video images representative of a first conferee concurrently taken from different views and track head position of the first conferee;

a stereo module, configured to receive the first and second video images representative of the first conferee concurrently taken from different views and match features and contours of the first conferee observed from the first and second video images, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and

a view synthesis module, configured to synthesize the head position the matching features and the matching contours of the first conferee observed from the first and second video images to generate a virtual image video stream of the first conferee.

24. The system as recited in claim 23, wherein the stereo module is further configured to rectify the first and second video images.

25. The system as recited in claim 23, wherein the stereo module is further configured to perform contour matching of the first and second video images by subtracting the background from the first and second images to extract the contours of the first conferee.

26. The system as recited in claim 23, wherein the view system module is configured to synthesize the head position, the features and the contours from the first and second video images by is morphing the head position, the features and the contours from the first and second images to generate the virtual image video stream of the first conferee.

27. The system as recited in claim 23, wherein the view system module is configured to synthesize the head position, the features, and the contours from the first and second video images by blending multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee.

28. One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to:

concurrently capture first and second video images representative of a first conferee taken from different views;

track a head position of the first conferee from the first and second video images;

ascertain features and contours from the first video image that match features and contours from the second video image, wherein the contours from the first and second video images are approximated by assigning polygonal lines; and

synthesize the head position, the features, and the contours from the first and second video images that match to generate a virtual image video stream of the first conferee.

29. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to store a personalized face model of the first conferee.

30. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to store a personalized face model of the first conferee and evaluate the first and second video images with respect to the personalized face model of the first conferee to monitor feature points from the first and second video images to track the head position.

31. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to, rectify the first and second video images and locate features from the first and second video images that reside on epipolar lines that match when ascertaining features from the first video image that matches features from the second video image.

32. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to, rectify the first and second video images and locate contours from the first and second video images that reside on epipolar lines that match when ascertaining contours from the first video image that matches contours from the second video image.

33. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to morph the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee when synthesizing the head position as well as the features and contours from the first and second video images.

34. One or more computer-readable media as recited in claim 28, further comprising computer executable instructions that, when executed, direct the computer system to blend multi-texture features associated with the head position, the features, and the contours from the first and second images to generate the virtual image video stream of the first conferee, when synthesizing the head position, the features, and the contours from the first and second video images.

35. One or more computer-readable media having stored thereon computer executable instructions that, when executed by one or more processors, causes the one or more processors of a computer system to:

store a personalized face model of a first conferee;

concurrently, capture first and second video images representative of the first conferee taken from different views;

evaluate the first and second video images with respect to the personalized face model of the first conferee to ascertain three dimensional information; and

synthesize the three dimensional information to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.

36. One or more computer-readable media as recited in claim 35, further comprising computer executable instructions that, when executed, direct the computer system to, track feature points from the first and second video images to monitor a head position of the first conferee when evaluating the first and second video images with respect to the personalized face model of the first conferee.

37. One or more computer-readable media as recited in claim 35, further comprising computer executable instructions that, when executed, direct the computer system to, ascertain features and contours from the first video image that match features and contours from the second video image.

38. One or more computer-readable media as recited in claim 35, further comprising computer executable instructions that, when executed, direct the computer system to ascertain features and contours from the first video image that match features and contours from the second video image and synthesizing the three dimensional information with features and contours to generate the virtual image video stream of the first conferee.

39. One or more computer-readable media as recited in claim 35, further comprising computer executable instructions that, when executed, direct the computer system to ascertain features and contours from the first video image that match features and contours from the second video image and synthesize the three dimensional information with features and contours, and morph the head position as well as the features and contours from the first and second images to generate the virtual image video stream of the first conferee when synthesizing the three dimensional information as well as the features and contours from the first and second video images.

40. One or more computer-readable media as recited in claim 35, further comprising computer executable instructions that, when executed, direct the computer system to, ascertain features and contours from the first video image that match features and contours from the second video image and synthesize the three dimensional information with the features and contours to generate the virtual image video steam of the first conferee.