Cyclopean virtual imaging via generalized probabilistic smoothing

• US 7,428,330 B2
• Filed: 05/02/2003
• Issued: 09/23/2008
• Est. Priority Date: 05/02/2003
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Cox, Ingemar J., Blake, Andrew, Torr, Philip H. S.
• Primary Examiner(s): LE, BRIAN Q
• Application Number:
  US10/428,388
• Publication Number:
  US 20040218809A1
• Time in Patent Office: 1,971 Days
• Field of Search: 382/154, 382/285, 382/294
• US Class Current: 382/154
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/10 Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A cyclopean virtual imaging system provides a view from a virtual camera placed symmetrically or midway between the calibrated cameras of a physical stereo pair. Such a system may be used in video conferencing applications as well as other stereo image contexts. By combining the left and right stereo images captured by the stereo pair, a virtual image is produced such that the subject appears to be looking into a cyclopean virtual camera, rather than to the left or right of a single physical camera. The cyclopean virtual image is generated by a forward-backward algorithm using a probabilistic distribution of possible disparity families.

Claims

1. A method of determining a cyclopean image associated with a left image and a right image, the method comprising:

computing a joint posterior density ξ

(d_k, δ

_k) of a stereo disparity d_k in conjunction with a disparity gradient δ

_k, wherein the joint posterior density is based on a forward probability distribution of the stereo disparity and a backward probability distribution of the stereo disparity; and

estimating the cyclopean image based on the joint posterior density of the stereo disparity with the disparity gradient.

2. The method of claim 1 wherein the disparity gradient represents a difference between stereo disparity values at adjacent cyclopean coordinates of the cyclopean image.

3. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and further comprising:

computing the forward probability distribution of the stereo disparity based on a distribution of disparity gradients for the current cyclopean coordinate given a previous cyclopean coordinate.

4. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and further comprising:

computing the forward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a previous cyclopean coordinate.

5. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and further comprising:

computing the backward probability distribution of the stereo disparity based on a distribution of disparity gradients for a next cyclopean coordinate given the current cyclopean coordinate.

6. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and further comprising:

computing the backward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a next cyclopean coordinate.

7. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and estimating the cyclopean image comprises:

computing an intensity expectation at the current cyclopean coordinate over observation noise.

8. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and estimating the cyclopean image comprises:

optimizing a loss function to estimate the cyclopean intensity at the current cyclopean coordinate.

9. The method of claim 1 wherein the cyclopean image is associated with a

current cyclopean coordinate and estimating the cyclopean image comprises:

computing an intensity expectation at the current cyclopean coordinate over observation noise; and

computing a posterior mean based on the intensity expectation and the joint posterior density of the stereo disparity with a disparity gradient.

10. A computer-readable storage medium

product encoding a computer program for executing on a computer system a computer

process for determining a cyclopean image associated with a left image and a right image, the computer process comprising:

computing a joint posterior density ξ

(d_k, δ

_k) of a stereo disparity d_k in conjunction with a disparity gradient δ

_k, wherein the joint posterior density is based on a forward probability distribution of the stereo disparity and a backward probability distribution of the stereo disparity; and

estimating the cyclopean image based on the joint posterior density of the stereo disparity with the disparity gradient.

11. The computer-readable storage medium product of claim 10 wherein the disparity gradient represents a difference between stereo disparity values at adjacent cyclopean coordinates of the cyclopean image.

12. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and the computer process further comprises:

computing the forward probability distribution of the stereo disparity based on a distribution of disparity gradients for the current cyclopean coordinate given a previous cyclopean coordinate.

13. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and the computer process further comprises:

computing the forward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a previous cyclopean coordinate.

14. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and the computer process further comprises:

computing the backward probability distribution of the stereo disparity based on a distribution of disparity gradients for a next cyclopean coordinate given the current cyclopean coordinate.

15. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and the computer process further comprises:

computing the backward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a next cyclopean coordinate.

16. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and estimating the cyclopean image comprises:

computing an intensity expectation at the current cyclopean coordinate over observation noise.

17. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and estimating the cyclopean image comprises:

optimizing a loss function to estimate the cyclopean intensity at the current cyclopean coordinate.

18. The computer-readable storage medium product of claim 10 wherein the cyclopean image is associated with a current cyclopean coordinate and estimating the cyclopean image comprises:

computing an intensity expectation at the current cyclopean coordinate over observation noise; and

computing a posterior mean based on the intensity expectation and the joint posterior density of the stereo disparity with a disparity gradient.

19. A system for determining a cyclopean image associated with a left image and a right image, the system comprising:

a forward-backward module computing a joint posterior density ξ

(d_k, δ

_k) of a stereo disparity d_k in conjunction with a disparity gradient δ

_k, wherein the joint posterior density is based on a forward probability distribution of the stereo disparity and a backward probability distribution of the stereo disparity; and

a cyclopean image generator estimating the cyclopean image based on the joint posterior density of the stereo disparity with the disparity gradient.

20. The system of claim 19 wherein the disparity gradient represents a difference between stereo disparity values at adjacent cyclopean coordinates of the cyclopean image.

21. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and further comprising:

a forward probability module computing the forward probability distribution of the stereo disparity based on a distribution of disparity gradients for the current cyclopean coordinate given a previous cyclopean coordinate.

22. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and further comprising:

a forward probability module computing the forward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a previous cyclopean coordinate.

23. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and further comprising:

a backward probability module computing the backward probability distribution of the stereo disparity based on a distribution of disparity gradients for a next cyclopean coordinate given the current cyclopean coordinate.

24. The system of claim 19 wherein the cyclopean image is associated with a

current cyclopean coordinate and further comprising:

a backward probability module computing the backward probability distribution of the stereo disparity based on an observation given the current cyclopean coordinate and a next cyclopean coordinate.

25. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and the cyclopean image generator computes an intensity expectation at the current cyclopean coordinate over observation noise.

26. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and the cyclopean image generator optimizes a loss function to estimate the cyclopean intensity at the current cyclopean coordinate.

27. The system of claim 19 wherein the cyclopean image is associated with a current cyclopean coordinate and the cyclopean image generator computes an intensity expectation at the current cyclopean coordinate over and a posterior mean based on the intensity expectation and the joint posterior density of the stereo disparity with a disparity gradient.