System and process for image-based relativistic rendering

• US 6,757,446 B1
• Filed: 11/27/2000
• Issued: 06/29/2004
• Est. Priority Date: 11/27/2000
• Status: Expired due to Term

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Jiang, Shum, Heung-Yeung
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US09/724,207
• Time in Patent Office: 1,310 Days
• Field of Search: 382/276, 382/100, 382/293, 345/418, 345/646, 345/647
• US Class Current: 382/293
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

The present invention involves a new system and process for image-based relativistic rendering of one or more single or sequential images of a scene. Such images may be produced by any conventional method, and include, for example, photographs, panoramas, motion video or film, or any other type of image captured by a camera or other image capture device. Relativistic rendering in accordance with the present invention transforms one or more initial images to produce photo-realistic images that accurately simulate how the scene would be viewed by an observer that is traveling at any desired velocity up to the speed of light. These rendered images are equal in resolution and quality to the initial images. Such relativistic simulation is based on Einstein'"'"'s Special Theory of Relativity. Further, unlike previous techniques, relativistic rendering in accordance with the present invention does not depend upon artificial or synthetic models, or upon ray-tracing methods. Consequently, the computer power required for relativistic rendering in accordance with the present invention is considerably less than that required for the previous techniques that rely on such methods.

Claims

1. A computer-implemented process for simulating how at least one pixel-based image of a scene would appear when the scene is viewed at a different viewing velocity relative to the velocity at which each pixel-based image was acquired, comprising:

representing each image pixel using light rays by defining each light ray as a unit three-dimensional vector with respect to a position of each pixel within each image, a field of view of each image, a width and height of each image, and a point from which each image is captured;

transforming each light ray from an initial inertial reference frame representing a reference frame in which each pixel-based image was acquired, to a new inertial reference frame representing a reference frame in which the scene is viewed; and

transforming each image by mapping the pixels to locations in the new inertial reference frame where the transformed light rays intersect a plane representing a view port.

2. The computer-implemented process of claim 1, further comprising cropping each transformed image to display a subsection of each transformed image.

3. The computer-implemented process of claim 1, further comprising changing an effective field of view for each transformed image by zooming each transformed image.

4. The computer-implemented process of claim 1, wherein each pixel-based image is a panorama.

5. The computer-implemented process of claim 1, further comprising shifting an observed frequency for each pixel of each transformed image by using a relative velocity between the initial reference frame and the new reference frame to determine a Doppler shift for each image pixel.

6. The computer-implemented process of claim 1, wherein a user provided viewing velocity is specified via a user interface.

7. The computer-implemented process of claim 1, wherein a speed represented by the viewing velocity has a user provided direction specified via a user interface.

8. The computer-implemented process of claim 1, wherein a user provided angular shift for viewing the scene is specified via a user interface.

9. The computer-implemented process of claim 1, wherein a user provided field of view for viewing the scene is specified via a user interface.

10. The computer-implemented process of claim 1, wherein the viewing velocity is any velocity up to the speed of light.

11. The computer-implemented process of claim 1, wherein the viewing velocity relative to the velocity at which each pixel based image is acquired represents a relative velocity between the initial inertial reference frame and the new inertial reference frame, and wherein a Lorentz transform uses the relative velocity for transforming each light ray from the initial inertial reference frame to the new inertial reference frame.

12. The computer-implemented process of claim 1, wherein a motion video represents each pixel-based image.

13. The computer-implemented process of claim 12, wherein the motion video is captured by an accelerating image capture device.

14. The computer implemented process of claim 13 wherein the viewing velocity relative to the velocity at which each pixel based image is acquired represents a relative velocity between the initial inertial reference frame and the new inertial reference frame, and wherein the relative velocity represents an accelerating viewing velocity proportional to the acceleration of the accelerating image capture device.

15. A system for simulating how at least one sequential pixel-based image of a scene would be viewed by an observer traveling at any desired viewing velocity, comprising:

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to;

(a) represent each pixel with a corresponding light ray by defining each light ray as a unit three-dimensional vector with respect to a position of each pixel within each image, a field of view of each image, a width and height of each image, and a point from which each image is captured,

(b) transform each light ray from an initial inertial reference frame representing each image to a new inertial reference frame representing the observer, and

(c) map each pixel to a location in the new inertial reference frame where each corresponding light ray intersects a view port of the observer.

16. The system of claim 15 wherein the at least one sequential pixel-based image of a scene is represented by a motion film acquired by a stationary image capture device.

17. The system of claim 15 wherein the at least one sequential pixel-based image of a scene is represented by a motion film extracted from a set of sequential images acquired by a plurality image capture devices positioned along a path.

18. The system of claim 15 wherein the at least one sequential pixel-based image of a scene is represented by a motion film acquired by an image capture device having a constant velocity.

19. The system of claim 18 wherein the image capture device moves along a curved path.

20. The system of claim 15 wherein the at least one sequential pixel-based image of a scene is represented by a motion film acquired by an accelerating image capture device.

21. The system of claim 20 wherein the viewing velocity has an acceleration proportional to the acceleration of the image capture device.

22. The system of claim 21 wherein the image capture device accelerates along a curved path.

23. The system of claim 15 wherein the viewing velocity has a range from zero to the speed of light.

24. The system of claim 15, further comprising a program module for Doppler shifting each pixel mapped to the new inertial reference frame with respect to the viewing velocity relative to a velocity at which each pixel-based image was acquired.

25. A computer-readable medium having computer executable instructions for transforming at least one sequential pixel-based image, said computer executable instructions comprising:

representing each image pixel with a light ray by defining each light ray as a unit three-dimensional vector with respect to a position of each pixel within each image, a field of view of each image, a width and height of each image, and a point from which each image is captured;

transforming each light ray via a Lorentz Transformation between an initial inertial reference frame and a new inertial reference frame based on a relative velocity between the initial inertial reference frame and the new inertial reference frame;

mapping each pixel to a location in the new inertial reference frame where each corresponding transformed light ray intersects a viewing plane; and

Doppler shifting each image pixel based on the relative velocity between the initial inertial reference frame and the new inertial reference frame.

26. The computer-readable medium of claim 25 wherein transforming each light ray comprises transforming each light ray with respect to the relative velocity between the initial inertial reference frame and the new inertial reference frame.

27. The computer-readable medium of claim 25 wherein mapping each pixel to a location in the new inertial reference frame comprises locating each pixel at a point where each corresponding transformed light ray intersects a viewing plane, and wherein a space-time point from which each image was acquired is coincident with a point in the new inertial reference frame from which each image is viewed.

28. The computer-readable medium of claim 25 wherein the relative velocity between the initial inertial reference frame and the new initial inertial reference frame is a user specified relative velocity ranging from zero to the speed of light.

29. The computer-readable medium of claim 28 wherein a direction of the relative velocity between the inertial reference frames is provided via a user interface.

30. The computer-readable medium of claim 25 wherein the axes of a global coordinate system for the initial inertial reference frame are parallel to the axes of a global coordinate system for the new inertial reference frame.

31. The computer-readable medium of claim 30 wherein an origin for the global coordinate system for the initial inertial reference frame is coincident with an origin for the global coordinate system for the new inertial reference frame at an instant when each image is acquired.

32. The computer-readable medium of claim 31 wherein:

a local coordinate system for the initial inertial reference frame is used to define the orientation of an image capture device for acquiring each image relative to the global coordinate system for the initial inertial reference frame; and

a local coordinate system for the new inertial reference frame is used to define the orientation of an observer for viewing each image relative to the global coordinate system for the new inertial reference frame.

33. The computer-readable medium of claim 32 wherein transforming each light ray comprises:

transforming each light ray from the local coordinate system for the new inertial reference frame to the global coordinate system for the new inertial reference frame;

transforming each light ray from the global coordinate system for the new inertial reference frame to the global coordinate system for the initial inertial reference frame; and

transforming each light ray from the global coordinate system for the initial inertial reference frame to the local coordinate system for the initial inertial reference system.

34. The computer-readable medium of claim 32 wherein each image is a panorama.

35. The computer-readable medium of claim 34 wherein the global coordinate system for the new inertial reference frame coincides with the global coordinate system for the initial inertial reference frame.

36. The computer-readable medium of claim 35 wherein transforming each light ray comprises:

transforming each light ray from the local coordinate system for the new inertial reference frame to the global coordinate system for the new inertial reference frame; and

transforming each light ray from the global coordinate system for the new inertial reference frame to the local coordinate system for the initial inertial reference system.

37. The computer-readable medium of claim 25, wherein the relative velocity between the inertial reference frames represents a proportional acceleration between the inertial reference frames.

38. The computer-readable medium of claim 34 where the proportional acceleration represents acceleration along a curved path.

39. The computer-readable medium of claim 35 where the acceleration along the curved path is approximated via a series of straight-line intervals.