System and method for interactive room modeling from camera images

• US 6,580,424 B1
• Filed: 10/21/2000
• Issued: 06/17/2003
• Est. Priority Date: 10/21/2000
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Krumm, John
• Primary Examiner(s): Vo, Cliff N.
• Application Number:
  US09/692,959
• Time in Patent Office: 969 Days
• Field of Search: 345/418, 345/419, 345/420, 345/421, 345/422, 345/423, 345/426, 345/427
• US Class Current: 345/419
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]

Abstract

A system and method for interactively modeling a room using camera images. Images of the room are input. The floor is marked with calibration markers placed at known (x,y) locations. The user runs a program that lets him “click” on these points to establish a correspondence between their absolute (x,y) coordinates on the floor and their image coordinates in each view. These correspondences are used to compute a “homography” for each image which in turn is used to compute a warped version of each image, showing what the room would look like if viewed from above. These warped images are each rendered with respect to the same origin and are used as guides in a drawing program to drag drawings of objects to place them on top of the corresponding objects in the warped images. The warped images are then deleted, leaving behind the room'"'"'s objects in their correct locations.

Claims

1. A computer-implemented process for generating a geometric model of a space, comprising using a computer to perform the process actions of:

ascertaining the coordinates of a plurality of points on a ground plane of the space;

capturing at least one image of the space;

for each image, determining the correspondence between the ground plane coordinates of the space and the corresponding locations in said image;

creating a warped version for each image to depict the space from a top down view; and

superimposing models of items in said space at locations on the ground plane where actual items are depicted in each warped image as touching the ground plane.

2. The process of claim 1, wherein the action of ascertaining the coordinates of said ground plane points comprises the process actions of:

placing calibration targets at known locations on the ground plane of the space;

associating each calibration target with an index number; and

entering the known ground plane location and index number associated with each calibration target into a data file.

3. The process of claim 2, wherein the process action of placing calibration targets on the ground plane comprises an action of placing enough targets on the ground plane such that at least four targets are depicted in each image of the space.

4. The process of claim 3, wherein between four and approximately ten calibration targets are placed so as to appear in each image of the scene.

5. The process of claim 2, wherein the process action of placing calibration on the ground plane comprises the action of dispersing the calibration targets as widely as possible given their numbers.

6. The process of claim 2, wherein the process action of entering the known ground plane location and index number associated with each calibration target comprises an action of a user entering the known ground plane location and index number associated with each calibration target into said data file.

7. The process of claim 2, wherein the process action of entering the ground plane location and index number associated with each calibration target comprises the actions of:

marking each calibration target with the known ground plane location and index number associated with that target in a manner readable by a conventional character recognition program; and

using said conventional character recognition program to read the location and index number from each calibration target from each image of the space.

8. The process of claim 1, wherein the action of ascertaining the coordinates of said ground plane points comprises the actions of:

identifying the location of each of a set of points associated with prescribed ground plane features that appear in the at least one image of the space;

associating an index number with each point of the set of points; and

entering the identified location and index number associated with each of the set of points.

9. The process of claim 8, wherein the process action of identifying the location of each of the set of points comprises the action of identifying the location of at least four points in each image of the space.

10. The process of claim 9, wherein the location between four and approximately ten points are identified in each image of the space.

11. The process of claim 8, wherein the process action of identifying the location of each of the set of points comprises the action of identifying the location of points that are as widely dispersed as possible given their number.

12. The process of claim 8, wherein the process action of entering the known ground plane location and index number associated with each point comprises an action of a user entering the known ground plane location and index number associated with each point into a data file.

13. The process of claim 2 wherein the process action of determining the correspondence between the coordinates in the space and the corresponding locations in said image comprises:

for each captured image

displaying the image on a computer monitor screen of the computer;

having a user select a point corresponding to the center of at least four calibration targets depicted in the image;

having the user enter the index number associated with each selected point; and

computing a homography which is capable of mapping points on the ground plane of the space to pixel locations in the image from the selected points and the known location of these points on the ground plane as identified by the index number entered for that point.

14. The process of claim 13, wherein the process action of having the user select points corresponding to at least four calibration target centers comprises an action of having the user select more than four points, and wherein the process action of computing the homography comprises an action of using a least squares approach to compute the homography.

15. The process of claim 1, wherein the process action of creating a warped version of each image to depict the space from a top down view comprises, for each image, the process actions of:

creating an array of cells each cell of which represents a pixel of ground plane image of the space defined by a particular (x,y) position separated by a prescribed Δ

x and Δ

y;

computing the location in the image corresponding to the (x,y) position associated with each of the cells of the array using the previously determined correspondence;

identifying the pixel of the image that most closely corresponds to each computed location;

assigning the pixel value associated with each identified pixel to the cell of the array corresponding to the (x,y) position used to identify the pixel; and

generating a ground plane image of the portion of the space depicted in the image using the assigned pixel values.

16. The process of claim 15 further comprising the process actions of:

determining whether each computed image location falls outside the bounds of the image; and

whenever the computed image location falls outside the bounds of the image, setting the pixel value of the cell of the array associated with the computed image location to zero.

17. The process of claim 16, wherein said pixel value is the color intensity of the image pixel under consideration.

18. The process of claim 15 wherein the Δ

x and Δ

y of each cell in the array corresponds to the size of one pixel in the captured image.

19. The process of claim 15, wherein Δ

x and Δ

y are the same length.

20. The process of claim 1 wherein the process action of superimposing models comprises, for each warped image, the actions of:

displaying the warped image to a user on a computer monitor screen;

obtaining a 2-D drawing of each object that is to be modeled in the warped image as it would appear in a top-down view;

placing the drawing of each object onto a location on the warped image where the object touches the ground plane in the image.

21. The process of claim 20, further comprising the process action of deleting the warped images leaving just the placing drawings which represent a floor plan of the space.

22. The process of claim 21, further comprising the process action of adding any embellishing objects not in the original captured image to the floor plan.

23. The process of claim 20 wherein the action of superimposing models is performed after importing the warped images into a drawing program.

24. The process of claim 20 wherein the same scale is used for the drawing model of said object as was used for the warped image.

25. A system for interactively creating a model of a space, the system comprising:

at least one camera positioned so as to have a view of the space to be modeled;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

determine the locations of a plurality of points in a ground plane of the space;

input at least one image of the space from the at least one camera;

for each image,

compute a mapping between the locations of the plurality of points in the space and the corresponding points in the image;

create a top down view from the image correctly depicting the locations where objects in the space touch the ground plane of the space using the image and the mapping;

import the top down view into a drawing program; and

place models of objects in the space onto the top down view in the drawing program.

26. The system of claim 25, wherein the entire space it is desired to model cannot be imaged using just one camera, and wherein a sufficient number of cameras are used such that every part of the space is desired to model can be imaged using one of the cameras.

27. The system of claim 26, wherein each camera is used to capture an image of the space, and wherein the program module for inputting at least one image of the space comprises a sub-module for inputting an image from each camera.

28. A computer-readable medium having computer-executable instructions for generating a geometric model of a space using a camera, said computer-executable instructions comprising:

ascertaining the coordinates of a number of points on a ground plane of the space;

capturing at least one image of the space;

for each image, determining the relationship between the ground plane coordinates of the space and the corresponding locations in said image;

creating a warped version for each image to depict the space from a top down view; and

superimposing models of items in said space at locations on the ground plane where actual items are depicted in each warped image as touching the ground plane.