Photogrammetry engine for model construction

• US 6,912,293 B1
• Filed: 06/25/1999
• Issued: 06/28/2005
• Est. Priority Date: 06/26/1998
• Status: Expired due to Term

• Current Assignee: Carl P. Korobkin
• Original Assignee: Carl P. Korobkin
• Inventors: Korobkin, Carl P.
• Primary Examiner(s): CHANG, JON CARLTON
• Application Number:
  US09/344,814
• Time in Patent Office: 2,195 Days
• Field of Search: 382/284, 382/285, 382/141, 382/154, 382/100, 345/420, 345/419, 345/619, 345/621, 345/629, 345/630, 345/632, 345/634, 705/26, 705/27
• US Class Current: 382/100
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes

Abstract

A system and methods for accessing 2D digital images and 3D geometric models over a network (preferably the Internet) and transforming and composing that media along with 2D digital image and 3D geometric model media, acquired by other means, into enhanced 2D image and 3D model representations for virtual reality visualization and simulation is disclosed. Digital images and models from a network and other sources are incorporated and manipulated through an interactive graphical user interface. A photogrammetric media processing engine automatically extracts virtual sensor (camera) and geometric models from imagery. Extracted information is used by a reconstruction processor to automatically and realistically compose images and models. A rendering system provides real-time visualization and simulation of the constructed media. A client-server processing model for deployment of the media processing engine system over a network is disclosed.

Claims

1. A method for generating remote presentation of products in situ for a user comprising the steps of:

(a) prompting the user to acquire a digital image of a space in which a product not of the space is to be viewed in context;

(b) accepting a product representation representing a product for which a three-dimensional geometric model exists or can be created;

(c) determining at least one dimension reference, wherein a dimension reference is a measurement of a distance in the space corresponding to a separation of two points on the digital image;

(d) calculating, from the image, the at least one dimension reference and reference information, camera parameters for a camera in the space from which the image was captured, wherein the camera parameters include at least a camera position and a focal length;

(e) accepting an input of a product location, the product location being a location in the space where the product is to be displayed in situ;

(f) transforming the geometric model of the product not of the space based on at least the camera position and the product location to form a transformed geometric model; and

(g) combining the transformed geometric model and the digital image to form a modified image of space where the modified image has been modified to show the identified product in situ in the image with a proper proportion and perspective.

2. The method of claim 1, wherein the step of accepting a product representation comprises the steps of:

prompting the user to identify a product of interest from a set of products;

searching a collection of product representations to locate a member of the collection that accepting the matching product representation.

3. The method of claim 2, further comprising a step of repeating steps b), e), and f) for second and subsequent selected products.

4. The method of claim 1, wherein the step of accepting a product representation includes a prior step of prompting the user to acquire a three-dimensional geometric model for a product of interest to the user.

5. The method of claim 1, further comprising a step of repeating step a) for second and subsequent images of the space and using each of the images of the space.

6. The method of claim 1, wherein the modified image of the space is a two-dimensional view of a three-dimensional geometric model, the method further comprising a step of moving a camera position of the modified image of the space to simulate moving around in the space.

7. The method of claim 1, wherein the step of prompting the user to identify the product from a set of products is performed using a commerce server that serves product models and further comprising a step of transmitting the digital image to the commerce server.

8. The method of claim 1, wherein the step of prompting the user to identify the product from a set of products is performed using a commerce application that receives product models and further comprising a step of providing the digital image to the commerce application.

9. The method of claim 1, wherein the camera parameters include camera position, camera rotation, focal length and center of projection.

10. The method of claim 1, wherein the reference information includes correspondence between two-dimensional image features and three-dimensional structures.

11. The method of claim 1, wherein the step of determining the at least one dimension reference is step of inputting the at least one dimension reference, wherein a dimension reference is a measurement of a distance in the space corresponding to a separation of two points on the digital image.

12. The method of claim 1, wherein the step of determining the at least one dimension reference is a step of assuming a default scale and using the default scale to determine the at least one dimension reference.

13. The method of claim 1, wherein the three-dimensional model for a product is a planar representation of an object and a texture map to be applied to a surface of the object.

14. A method for generating presentations of products in situ, wherein an in situ presentation includes a composite image showing one or more products in a space, the method comprising the steps of:

a) inputting a scene digital image of the space, wherein the scene digital image is an image of the space captured by an image capture device positioned in the space at a capture position and having a capture focal length;

b) calculating, from the image, capture parameters for the image capture device in the space, wherein the capture parameters include at least a representation of the capture position and a representation of the capture focal length;

c) inputting a product representation that represents a product not of the space being considered for placement in the space;

d) accepting an input of a product location, the product location being a location in the space at which the product is being considered;

e) transforming the product representation wherein a transformed image is an image transformed according to the image'"'"'s associated capture parameters; and

f) combining the scene digital image and the product representation, as transformed in the step of transforming, to form a combined image of the space where the combined image shows the product in situ in the scene in the combined image with a proper proportion and perspective.

15. The method of claim 14, wherein the product representation includes a texture map associated with the product.

16. The method of claim 14, wherein the product representation includes at least a product digital image, captured by a product image capture device positioned to capture a digital image of the product.

17. The method of claim 16, wherein the product representation includes a capture position, capture angle of rotation and focal length.

18. The method of claim 16, wherein the product representation includes product image capture parameters including at least a representation of the position of the product image capture device when the digital image of the product was captured.

19. The method of claim 18, wherein the product image capture parameters include projections of geometric elements from the product onto the product digital image.

20. The method of claim 14, wherein the capture parameters for the image of the scene include projections of geometric elements from the scene onto the scene digital image.

21. The method of claim 14, wherein the capture parameters for the image of the scene include an angle of rotation for the image capture device in the space.

22. The method of claim 14, wherein the step of inputting a product representation is preceded by the steps of:

capturing a product digital image using the product image capture device; and

calculating, from the product digital image, product image capture parameters that include at least a representation of the product image capture position and a representation of the product image capture focal length.

23. The method of claim 14, wherein the step of transforming results in the one or more transformed digital image sharing common capture parameters, the common capture parameters including at least a capture position.

24. The method of claim 14, further comprising a step of prompting the user to identify the product from a set of products for which a three-dimensional geometric model exists or can be created.

25. The method of claim 14, further comprising a step of inputting at least one dimension reference for one or both of the scene digital image and the product representation, wherein a dimension reference is a measurement of a distance in space corresponding to a separation of two points on the digital image or representation with which the dimension reference is associated.

26. The method of claim 14, further comprising the step of repeating the step of combining to form a further combined image of the space and a plurality of products, where the further combined image shows the plurality of products in situ in the scene with each product having a proper proportion and perspective.

27. The method of claim 14, wherein the step of inputting a product representation is followed by a step of calculating, from the product digital image, product image capture parameters that include at least a representation of the product image capture position and a representation of the product image capture focal length.

28. A method for generating a combined image that simulates a product being positioned at a location where a consumer is considering placing the product, without physically placing the product at the location, the method comprising the steps of:

a) prompting the consumer to acquire a digital image of the location;

b) generating a location representation of the location and geometric elements representing items at the location visible in the digital image of the location;

c) prompting the consumer to select a product from among a plurality of products;

d) retrieving a product representation for a selected product, wherein the product representation includes at least a digital image of the product that is not present in the location and that will be subsequently inserted into the digital image of the location and a representation of the position of a product image capture device when the digital image of the product was captured.

29. The method of claim 28, wherein the location representation includes location capture parameters of a location capture device, wherein the location capture parameters include at least a representation of a position of the location capture device.

30. The method of claim 28, further comprising the step of:

e) prompting the consumer to indicate where, at the location, the product is to be considered; and

f) combining the location representation and the product representation after transforming at least one of the representation to form the combined image showing the product in situ with a proper proportion and perspective.

31. The method of claim 28, wherein the product representation includes at least a three-dimensional geometric model and a texture map.

32. The method of claim 28, wherein the method is performed by a commerce server and a consumer computer, the method further comprising the steps of:

e) sending a commerce application form the commerce server to the consumer computer;

f) executing the commerce application on the consumer computer, wherein the commerce application performs steps a) and c);

g) using the commerce application to model geometric elements visible in the digital image of the location; and

h) generating capture parameters for the digital image of the location from the geometric elements using the commerce application, the capture parameters including a capture location and a focal length.

33. The method of claim 28, wherein the method is performed by a commerce server and a consumer computer and wherein the step of retrieving the product representations in a step of retrieving the product representations from the commerce server.

34. The method of claim 28, wherein the method is performed by a commerce server and a consumer computer and wherein the step of retrieving the product representations from a product representation server.