Synthesizing virtual two dimensional images of three dimensional space from a collection of real two dimensional images

• US 6,219,444 B1
• Filed: 02/02/1998
• Issued: 04/17/2001
• Est. Priority Date: 02/03/1997
• Status: Expired due to Term

• Current Assignee: MobilEye Vision Technologies Ltd. (Intel Corporation)
• Original Assignee: Yissum Research Development Company of the Hebrew University of Jerusalem Ltd. (Hebrew University of Jerusalem)
• Inventors: Avidan, Shai, Shashua, Amnon
• Primary Examiner(s): Patel, Jayanti K.
• Application Number:
  US09/016,925
• Time in Patent Office: 1,170 Days
• Field of Search: 382/100, 382/154, 382/282, 382/284, 382/286, 382/285, 382/103, 382/291, 382/294, 382/296, 382/302, 382/305, 382/307, 382/289, 345/121, 345/127, 345/131, 345/139, 345/437, 345/438, 345/424, 356/12, 348/42, 348/47, 348/49, 348/50, 348/51
• US Class Current: 382/154
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

Method and apparatus are provided for generating two dimensional images representing scenes of three dimensional spaces as viewed for arbitrary observing point. In a specific embodiment of the method and apparatus, consecutive virtual images, computed from four dimensional reference images are consecutively displayed on a multimedia screen in order to convey motion of a body.

Claims

1. A method of synthesizing a virtual image of a scene comprising the steps of:

A. constructing a virtual trilinear tensor from a trilinear tensor associated with at least one real image and a positioning parameter defining a position of said virtual image relative to said at least one real image; and

B. using the virtual trilinear tensor and said at least one real image to warp said at least one real image thereby to generate said virtual image.

2. A method as defined in claim 1 further comprising the step of generating said trilinear tensor from said at least one real image and two further real images of said scene.

3. A method as defined in claim 2 in which two of said real images are acquired at a narrow angle between lines of sight between said real images and said scene.

4. A method as defined in claim 1 in which said position parameter includes a translation parameter relating a translation of the virtual image relative to said one of said real images.

5. A method as defined in claim 1 in which said position parameter includes a rotation parameter relating a rotation angle of the virtual image relative to said one of said real images.

6. A method as defined in claim 5 in which the rotation angle is greater than an angle defining an imaginary cone defined by the lines of sight connecting between the three camera positions used to acquire the real images and the scene.

7. A method as defined in claim 1 in which said one of said real images is an image other than the two real images that are acquired at the narrow angle.

8. A method as defined in claim 1 in which said one of said real images is has at least seven points that correspond to points in said two real images that are acquired at the narrow angle.

9. A method as defined in claim 1, further comprising the steps of:

A. constructing a second virtual trilinear tensor from the trilinear tensor and a positioning parameter defining a position of a second virtual image relative to one of said real images; and

B. using the second virtual trilinear tensor and one of said real images to warp a real image thereby to generate a second virtual image.

10. A method as defined in claim 1, further comprising the steps of:

A. constructing a second virtual trilinear tensor from the virtual trilinear tensor and a positioning parameter defining a position of a second virtual image relative to said virtual image; and

C. using the second virtual trilinear tensor and said virtual image to warp a real image thereby to generate a second virtual image.

11. A method as defined in claim 1 further comprising the steps of:

A. constructing a fundamental matrix from the at least one real image and a second real image; and

B. constructing said trilinear tensor from the fundamental matrix.

12. A method as defined in claim 11 in which said real images are acquired at a narrow angle between lines of sight connecting the respective real images with the scene.

13. A method as defined in claim 11 in which said fundamental matrix construction step includes the step of constructing the fundamental matrix using coordinates of corresponding points in the two real images.

14. A method as defined in claim 1 further comprising the step of transferring said trilinear tensor and said at least one real image to a user over a network.

15. A system for synthesizing a virtual image of a scene comprising:

A. a virtual trilinear tensor generator configured to construct a virtual trilinear tensor from a trilinear tensor associated with at least one real image and a positioning parameter defining a position of said virtual image relative said at least one real image; and

B. a virtual image generator configured to use the virtual trilinear tensor and said at least one real image to warp said at least one real image thereby to generate said virtual image.

16. A system as defined in claim 15 in which the virtual trilinear tensor generator is configured to generate said trilinear tensor from said at least one real image and two further real images of said scene.

17. A system as defined in claim 16 in which two of said real images are acquired at a narrow angle between lines of sight between said real images and said scene.

18. A system as defined in claim 15 in which said position parameter includes a translation parameter relating a translation of the virtual image relative to said one of said real images.

19. A system as defined in claim 15 in which said position parameter includes a rotation parameter relating a rotation angle of the virtual image relative to said one of said real images.

20. A system as defined in claim 19 in which the rotation angle is greater than an angle defining an imaginary cone defined by the lines of sight connecting between the three camera positions used to acquire the real images and the scene.

21. A system as defined in claim 15 in which said one of said real images is an image other than the two real images that are acquired at the narrow angle.

22. A system as defined in claim 15 in which said one of said real images is has at least seven points that correspond to points in said two real images that are acquired at the narrow angle.

23. A system as defined in claim 15, in which said virtual trilinear generator is further configured to construct a second virtual trilinear tensor from the trilinear tensor and a positioning parameter defining a position of a second virtual image relative to one of said real images, the virtual image generator being further configured to use the second virtual trilinear tensor and one of said real images to warp a real image thereby to generate a second virtual image.

24. A system as defined in claim 15, in which said virtual trilinear generator is further configured to constrict a second virtual trilinear tensor from the virtual trilinear tensor and a positioning parameter defining a position of a second virtual image relative to one of said real images, the virtual image generator being further configured to use the second virtual trilinear tensor and one of said real images to warp a real image thereby to generate a second virtual image.

25. A system as defined in claim 15 further comprising:

A. a fundamental matrix generator configured to construct a fundamental matrix from the at least one real image and a second real image; and

B. a trilinear tensor generator module configured to construct said trilinear tensor from the fundamental matrix.

26. A system as defined in claim 25 in which said real images are acquired at a narrow angle between lines of sight connecting the respective real images with the scene.

27. A system as defined in claim 25 in which said fundamental matrix generator is further configured to construct the fundamental matrix using coordinates of corresponding points in the two real images.

28. A system as defined in claim 15 further comprising a module configured to transfer said trilinear tensor and said at least one real image to a user over a network.

29. A computer program product for use in connection with a computer to synthesize a virtual image of a scene, the computer program product comprising a machine readable medium having encoded thereon:

A. a virtual trilinear tensor generator module configured to enable the computer to construct a virtual trilinear tensor from a trilinear tensor associated with at least one real image and a positioning parameter defining a position of said virtual image relative said at least one real image; and

B. a virtual image generator module configured to enable the computer to use the virtual trilinear tensor and said at least one real image to warp said at least one real image thereby to generate said virtual image.

30. A computer program product as defined in claim 29 in which the virtual trilinear tensor generator module is configured to enable the computer to generate said trilinear tensor from said at least one real image and two further real images of said scene.

31. A computer program product as defined in claim 30 in which two of said real images are acquired at a narrow angle between lines of sight between said real images and said scene.

32. A computer program product as defined in claim 29 in which said position parameter includes a translation parameter relating a translation of the virtual image relative to said one of said real images.

33. A computer program product as defined in claim 29 in which said position parameter includes a rotation parameter relating a rotation angle of the virtual image relative to said one of said real images.

34. A computer program product as defined in claim 33 in which the rotation angle is greater than an angle defining an imaginary cone defined by the lines of sight connecting between the three camera positions used to acquire the real images and the scene.

35. A computer program product as defined in claim 29 in which said one of said real images is an image other than the two real images that are acquired at the narrow angle.

36. A computer program product as defined in claim 29 in which said one of said real images is has at least seven points that correspond to points in said two real images that are acquired at the narrow angle.

37. A computer program product as defined in claim 29, in which said virtual trilinear generator module is further configured to enable the computer to construct a second virtual trilinear tensor from the trilinear tensor and a positioning parameter defining a position of a second virtual image relative to one of said real images, the virtual image generator module being further configured to enable the computer to use the second virtual trilinear tensor and one of said real images to warp a real image thereby to generate a second virtual image.

38. A computer program product as defined in claim 29, in which said virtual trilinear generator module is further configured to enable the computer to construct a second virtual trilinear tensor from the virtual trilinear tensor and a positioning parameter defining a position of a second virtual image relative to one of said real images, the virtual image generator module being further configured to enable the computer to use the second virtual trilinear tensor and one of said real images to warp a real image thereby to generate a second virtual image.

39. A computer program product as defined in claim 29 further comprising:

A. a fundamental matrix generator module configured to enable the computer to construct a fundamental matrix from the at least one real image and a second real image; and

B. a trilinear tensor generator module configured to enable the computer to construct said trilinear tensor from the fundamental matrix.

40. A computer program product as defined in claim 39 in which said real images are acquired at a narrow angle between lines of sight connecting the respective real images with the scene.

41. A computer program product as defined in claim 39 in which said fundamental matrix generator module is further configured to enable the computer to construct the fundamental matrix using coordinates of corresponding points in the two real images.

42. A computer program product as defined in claim 29 further comprising a module configured to enable the computer to transfer said trilinear tensor and said at least one real image to a user over a network.