Computerized video imaging system for creating a realistic depiction of a simulated object in an actual environment

• US 4,970,666 A
• Filed: 03/30/1988
• Issued: 11/13/1990
• Est. Priority Date: 03/30/1988
• Status: Expired due to Fees

• Current Assignee: Land Development Laboratory Incorporated
• Original Assignee: Land Development Laboratory Incorporated
• Inventors: Welsh, William T., Pittman, Kenneth B.
• Primary Examiner(s): Harkcom, Gary V.
• Assistant Examiner(s): Zimmerman, Mark K.
• Application Number:
  US07/175,158
• Time in Patent Office: 958 Days
• Field of Search: 364/522, 364/521, 434/38, 434/43, 340/747
• US Class Current: 345/423
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering

Abstract

A system and method for producing highly realistic video images which depict the appearance of a simulated structure in an actual environment, and provides for accurate placement of the structure in the environment and matching of the perspective of the structure with that of the environment so that a highly realistic result is achieved. The system includes a video input means, such as a video camera and video recorder, by which a video image of the actual environment may be captured. A graphics processor unit receives the video image from the video input means and stores it in rasterized form. Field data input means is provided for receiving field location data regarding the precise location of the captured video image and field perspective data regarding the perspective of the captured image. Object data input means is also provided for receiving data, such as CAD data for example, for a three-dimensional model of a simulated object which is proposed to be included in the environment. From the three-dimensional model of the object, a two-dimensional perspective representation of the object is generated, which accurately matches the perspective of the captured video image. The thus generated two-dimensional perspective representation is then merged with the rasterized video image and accurately positioned at its proper location in the environment.

Claims

1. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving data for a three-dimensional model of a simulated object which is proposed to be included in the environment;

(e) means for generating from the three-dimensional model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment; and

(f) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the object is precisely positioned at the proper location in its environment.

2. A system as set forth in claim 1 including video output means operatively connected to said graphics processor means for producing a video signal of the simulation.

3. A system as set forth in claim 1 wherein said object data input means comprises means for receiving CAD data for a three-dimensional CAD model of the simulated object.

4. A system as set forth in claim 3 wherein the three-dimensional CAD model comprises a wire frame model.

5. A system as set forth in claim 3 wherein the three-dimensional CAD model comprises a solid model.

6. A system as set forth in claim 1 wherein said field data input means additionally includes means for receiving data relating to the ambient conditions, such as weather, date and time of day, existing at the location when the video image was captured.

7. A system according to claim 1 additionally including

a material and texture library containing a collection of maps defining available materials and surface textures for the object, and

texture mapping means operatively associated with said means for generating a two-dimensional perspective representation of the object for imparting to the two-dimensional representation of the object a surface appearance defined by a map selected from said material and texture library.

8. A system as set forth in claim 7 wherein said data input means includes means operable by the user of the system for assigning to the object, materials and textures selected from said material and texture library.

9. A system as set forth in claim 1 additionally including

an image library containing a collection of additional image elements for the environment, and

pre/post-processing image enhancement means, including means for selecting additional image elements from said image library and for adding such elements to the merged image produced by said means for merging for thereby enhancing the simulation.

10. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the defined reference points on the object are located at the proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect to its environment.

11. A system as set forth in claim 10 wherein said means for generating a two-dimensional perspective representation includes means for reorienting the three-dimensional model so as to correspond with its proper orientation and scale in the actual environment before producing said two-dimensional perspective representation of the object.

12. A system as set forth in claim 10 wherein the field location data regarding the captured video image includes lighting condition data, and said system additionally including modeling means responsive to said lighting condition data for imparting to the object lighting conditions corresponding to those present in the captured video image of the environment to thereby further enhance the realism of the simulation.

13. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data including the coordinates of a camera location which captured the video image in relation to a known coordinate system, the coordinates of at least one other known location in the field of view of the camera, and data relating to the optical characteristics of the camera which captured the video image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for rotating, translating and scaling the three-dimensional CAD model so that the reference points on the three-dimensional model input by said object reference point input means coincide with the proper coordinate locations in the coordinate system of the actual environment,

(g) means for generating from the three-dimensional model of said object, through the use of said field perspective data, a rasterized two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment;

(h) means for superimposing the thus generated rasterized two-dimensional perspective representation of the object with said rasterized video image of the environment at corresponding coordinate locations in said coordinate system, so that the object is precisely positioned at the proper location in its environment, and to thereby produce a video image simulation accurately depicting the object as it would appear in the actual environment, and

(i) video output means for producing a video display of the thus produced video image simulation.

14. A system for producing a video image realistically depicting how a simulated object would appear in an actual environment, comprising:

(a) video input means for capturing a video image of the actual environment, the image having been obtained at a particular point of observation in the actual environment and with a particular point of view;

(b) graphics processor means for receiving said video image and forming therefrom a rasterized video image;

(c) object data input means, including means for storing data representing a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(d) object data conversion means, including

(1) means for receiving perspective data indicating the point of observation and the direction of view of the captured video image of the actual environment;

(2) means for receiving reference points representing known locations on the three-dimensional CAD model and their proper coordinate positions in the coordinate system of the actual environment;

(3) means for rotating, translating and scaling said three-dimensional CAD model so that said reference points on the three-dimensional CAD model coincide with the proper coordinate position in the coordinate system of the actual environment;

(4) means for generating from said three-dimensional CAD model, a two-dimensional rasterized image providing a perspective representation of the three-dimensional image with the point of observation and direction of view represented by said perspective data;

(5) means for superimposing the thus produced two-dimensional rasterized image of said object onto the captured rasterized video image of the actual environment at the corresponding coordinate locations in said coordinate system to thereby produce a synthesized composite video image depicting the object as it would appear in the actual environment; and

(e) video output means for producing a video display of the thus produced synthesized composite video image.

15. The system according to claim 14 wherein said object data input means includes means for receiving and storing surface texture data representing the surface texture of the object, and wherein said means for generating a two-dimensional rasterized image of the object from said three-dimensional CAD model includes means for rendering a two-dimensional wire frame model with the surface texture defined by said surface texture data to form a surface textured solid model of the object.

16. A method of producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) capturing an image of the actual environment,

(b) converting the captured image into rasterized form,

(c) obtaining field location data regarding a location of the captured image and field perspective data regarding the perspective of the captured image,

(d) providing object data defining a three-dimensional model of a simulated object which is proposed to be included in the environment,

(e) generating from the three-dimensional model of the object, through the use of the field perspective data, a rasterized two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured image of the actual environment, and

(f) merging the thus generated rasterized two-dimensional perspective representation of the object with said rasterized image of the environment while positioning the object, through the use of said field location data, so that the object is precisely positioned at the proper location in the environment.

17. A method as set forth in claim 16 wherein said step of providing object data defining a three-dimensional model comprises inputting CAD data for the simulated object.

18. A method as set forth in claim 16 including obtaining data relating to ambient conditions, such as weather, date and time of day, existing at the location when the video image was captured.

19. A method as set forth in claim 16 including storing in a material and texture library a collection of maps defining available materials and surface textures for the object, and texture mapping the two-dimensional representation of the object to impart to the object a surface appearance as defined by a selected map from said material and texture library.

20. A method as set forth in claim 16 including storing in an image library a collection of additional image elements for the environment, and selecting additional image elements from said image library and adding such elements to the merged video image to thereby enhance the simulation.

21. A method of producing a video simulation image realistically depicting how a simulated object would appear in an actual environment, comprising

(a) capturing with a camera and storing in a storage medium an image of the actual environment;

(b) obtaining field location data accurately identifying a point of observation and direction of view of the captured image in relation to a known coordinate system;

(c) providing CAD data for a three-dimensional model of a simulated object which is proposed to be included in the environment;

(d) defining within the CAD model reference points representing known locations on the three-dimensional model and assigning to said reference points the coordinates in said known coordinate system where the reference points of the object are to be located;

(e) generating from the three-dimensional CAD model of said object a two-dimensional image providing a perspective representation of the three-dimensional object using the same point of observation and direction of view as used in the captured image of the actual environment; and

(f) superimposing the thus produced two-dimensional image of said object onto the captured image of the actual environment at corresponding coordinates in said coordinate system to thereby produce a video image depicting the object as it would appear in the actual environment.

22. The method of claim 21 wherein said CAD model comprises a wire frame image.

23. The method of claim 21 wherein said CAD model comprises a solid model.

24. The method of claim 21 wherein the captured image of the environment is a still image.

25. The method of claim 21 wherein the captured image of the environment is a moving image.

26. A method of producing a video image realistically depicting how a simulated object would appear in an actual environment, comprising

(a) capturing and storing an image of the actual environment;

(b) obtaining site data accurately identifying a point of observation and direction of view of the captured image in relation to a known coordinate system;

(c) selecting a landmark in the captured image of the actual environment;

(d) identifying from a map or survey, coordinates defining the location of said landmark in relation to said known coordinate system;

(e) creating a CAD verification model of the selected landmark and assigning to said model said coordinates in said known coordinate system which define the location of the landmark;

(f) superimposing the CAD verification model of said landmark onto said captured image at corresponding coordinates in said coordinate system to thereby ascertain the accuracy of the site data obtained in step (b), and correcting the site data, if necessary, to ensure that the CAD verification model of said landmark is in registration with the image of the landmark;

(g) providing a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(h) establishing within said CAD model reference points representing known locations on the model and assigning to said reference points coordinates in said known coordinate system;

(i) generating from the three-dimensional CAD model of the object a two-dimensional image providing a perspective representation of the three-dimensional object based upon said site data so that the perspective view of the object matches that of the actual environment; and

(j) superimposing the thus produced two-dimensional image of said object onto the captured image of the actual environment at corresponding coordinates in said coordinate system to thereby produce a video image depicting the object as it would appear in the actual environment.

27. A method for producing a video image realistically depicting how a simulated object would appear in an actual environment, comprising

(a) positioning a camera at a predetermined location in the actual environment and obtaining therefrom an image of the actual environment;

(b) obtaining and storing data accurately identifying the location of the camera in the actual environment and its direction of view with respect to a known coordinate system;

(c) rasterizing the image of the actual environment obtained from said camera and storing the rasterized image;

(d) providing data representing a three-dimensional CAD model of a simulated object which is proposed to be included in the environment, said model being comprised of data defining a vector representation of the object;

(e) establishing within the three-dimensional model, reference points representing known locations on the model, and assigning to said reference points coordinate positions in the coordinate system of the actual environment, and rotating, translating and scaling the model so that known locations on the model are positioned at their proper locations in the coordinate system of the actual environment;

(f) converting the vector representation of said model into a two-dimensional rasterized representation of the object in a three-dimensional perspective corresponding with the perspective of the image of the environment captured by the camera; and

(g) overlaying the thus produced rasterized representation of the object with the rasterized image of the actual environment to thereby produce a video image depicting the object as it would appear in the actual environment.

28. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the precise location of the captured video image and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) means operatively associated with said object data input means for translating the input CAD data from one CAD data format to another standard CAD data format;

(f) means for generating from the three-dimensional CAD data for said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the object is precisely positioned at the proper location in its environment.

29. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) equipment calibration means for generating calibration data relating to characteristics of particular pieces of video input equipment which may be used to capture the video image which is input to said video input means, and

(b) an equipment calibration library for storing the equipment calibration data which is generated by said equipment calibration means,

(c) video input means for receiving a captured video image of an actual environment;

(d) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(e) field data input means for receiving field location data regarding the precise location of the captured video image and field perspective data regarding the perspective of the captured image;

(f) object data input means for receiving data for a three-dimensional model of a simulated object which is proposed to be included in the environment;

(g) means for generating from the three-dimensional model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment; and

(h) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data and said equipment calibration data, so that the object is precisely positioned at the proper location in its environment.

30. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment, said means for generating a two-dimensional perspective representation including means for rotating and translating the three-dimensional CAD mode with the use of said reference points so that the reference points coincide with the proper coordinate position in the coordinate system of the environment; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object through the use of said field location data, so that the defined reference points on the object are located at the proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect for its environment.

31. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image, the field perspective data including data defining a location of the camera and a center of view of the video image seen by the camera;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment, said means for generating a two-dimensional perspective including means for transforming the perspective of the two-dimensional representation in accordance with said field perspective data defining the camera location and center of view; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the defined reference points on the object are located at the proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect to its environment.

32. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding a precise location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a spherical perspective to closely match any image distortion of the captured video image of the actual environment;

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the defined reference points on the object are located at the proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect to its environment.

33. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment, said means for generating a two-dimensional perspective representation including line preprocessor means for segmenting long lines of the three-dimensional model into shorter segments before conversion to two-dimensional perspective so that long straight lines present in the object will be distorted in a way similar to distortion present in the captured video image of the environment; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the defined reference points on the object are located the proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect to its environment.

34. A video imaging system for producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) video input means for receiving a captured video image of an actual environment, the image having been obtained at a particular location in the actual environment and the image having a particular perspective;

(b) graphics processor means for receiving the video image from said video input means and for storing the video image in rasterized form;

(c) field data input means for receiving field location data regarding the location of the captured video image in relation to a known coordinate system and field perspective data regarding the perspective of the captured image;

(d) object data input means for receiving CAD data for a three-dimensional CAD model of a simulated object which is proposed to be included in the environment;

(e) object reference point input means for defining reference points representing known locations on the three-dimensional model and for assigning to said reference points coordinates in said known coordinate system where the reference points on the object are to be located;

(f) means for generating from the CAD data for the three-dimensional CAD model of said object, through the use of said field perspective data, a two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured video image of the actual environment, said means for generating a two-dimensional perspective including means for compensating for camera lens distortion to thereby provide a more accurate simulation; and

(g) means for merging the thus generated two-dimensional perspective representation of the object with said rasterized video image of the environment and for positioning the object, through the use of said field location data, so that the defined reference points on the object are located at a proper coordinates in said known coordinate system of the actual environment so that the object is precisely positioned with respect to its environment.

35. A method of producing a video image simulation realistically depicting how a simulated object would appear in an actual environment, comprising

(a) capturing an image of the actual environment;

(b) converting the captured image into rasterized form;

(c) obtaining field location data regarding the precise location of the captured image and field perspective data regarding the perspective of the captured image;

(d) providing CAD data defining a three-dimensional model of a simulated object which is proposed to be included in the environment;

(e) translating the input CAD data from one CAD data format to another standard CAD data format;

(f) generating from the three-dimensional model of the object, through the use of the field perspective data, a rasterized two-dimensional perspective representation of the object in a perspective which accurately matches the perspective of the captured image of the actual environment; and

(g) merging the thus generated rasterized two-dimensional perspective representation of the object with said rasterized image of the environment while positioning the object, through the use of said field location data, so that the object is precisely positioned at the proper location in the environment.