Image processing apparatus

• US 6,980,690 B1
• Filed: 11/24/2000
• Issued: 12/27/2005
• Est. Priority Date: 01/20/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Haslam, Jane, Taylor, Richard Ian, Wiles, Charles Stephen
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): CARTER, AARON W
• Application Number:
  US09/718,342
• Time in Patent Office: 1,859 Days
• Field of Search: 382/154, 382/294, 382/285, 345419-427, 345/700, 356 12- 14, 348 42- 60, 359462-477, 352 57- 65, 33/204, 353 7- 9, 378 41- 42, 396324-331
• US Class Current: 382/154
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

A 3-D model of an object is created by processing images taken from a series of camera positions. An initial sequence of the images is processed to define respective image co-ordinates of matching features to generate a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images. A new image is added to the sequence and processed to obtain a camera solution for a corresponding new virtual camera for use in generating further model data.

Claims

1. A method of operating an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the method comprising the steps of:

displaying a set of icons, each being associated with a respective one of the camera images of the object;

receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

determining a selected camera image from the set of camera images corresponding to the selected icon;

displaying the selected image;

determining position data representative of a selected camera position from which the selected image was taken;

generating in accordance with the model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

displaying the model image for visual comparison with the selected image by the user.

2. A method as claimed in claim 1, including the step of generating the icons in response to receiving a mode selection input.

3. A method as claimed in claim 1, wherein the icons are generated as thumbnail images of the respective camera images.

4. A method as claimed in claim 3, wherein the step of displaying the set of icons comprises displaying the icons in an array and displaying links between the icons such that each pair of icons corresponding to adjacent camera positions in a positional sequence of the camera positions is joined by a respective link.

5. A method as claimed in claim 4, wherein the icons are displayed in a linear array.

6. A method as claimed in claim 1, wherein the selected camera image and the model image are displayed in respective windows, and further including the step of providing relative movement of the windows in response to receiving window movement input signals.

7. A method as claimed in claim 6, wherein the icons are displayed in a further window, and further including the step of facilitating movement of the further window relative to the image windows in response to window movement input signals.

8. A method as claimed in claim 1, comprising generating the selection signal by operation of a pointing means for user actuation in selecting one of the displayed icons.

9. A method as claimed in claim 1, wherein displaying the set of icons comprises displaying a view of the model from a viewpoint in which the icons comprise representations of cameras and are shown at respective positions relative to the model which correspond substantially to the camera positions relative to the object.

19. A computer program comprising processor implementable instructions for carrying out a method as claimed in claim 1.

20. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 1.

21. A physically-embodied computer program product carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 1.

10. An apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the apparatus comprising;

display means for displaying a set of icons, each being associated with a respective one of the camera images of the object;

means for receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

means for determining a selected camera image from the set of camera images corresponding to the selected icon whereby the display means is operable to display the selected image;

means for determining position data representative of a selected camera position from which the selected image was taken;

means for generating, in accordance with the model, a model image representative of a view of the model from a viewpoint corresponding to the position data; and

control means for controlling the display means to display the model image for visual comparison with the selected image by the user.

11. An apparatus as claimed in claim 10, further comprising means for generating the icons in response to receiving a mode selection input.

12. An apparatus as claimed in claim 10, further comprising icon generating means operable to generate the icons as thumbnail images of the respective camera images.

13. An apparatus as claimed in claim 12, wherein the control means is operable to control the display means to display the set of icons in an array and to display links between the icons such that each pair of icons corresponding to adjacent camera positions in a positional sequence of the camera positions is joined by a respective link.

14. An apparatus as claimed in claim 13, wherein the control means is operable to control the display means to display the icons in a linear array.

15. An apparatus as claimed in claim 10, wherein the control means is operable control the display means to display the selected camera image and the model image in respective windows and to provide relative movement of the windows in response to receiving window movement input signals.

16. An apparatus as claimed in claim 15, wherein the control means is operable to control the display means to display the icons in a further window to facilitate movement of the further window relative to the camera image window and model image window in response to window movement input signals.

17. An apparatus as claimed in claim 10, wherein the means for generating the selection signal comprises a pointing means for user actuation in selecting one of the displayed icons.

18. An apparatus as claimed in claim 10, wherein the control means is operable to control the display means for displaying the set of icons by displaying a view of the model from a viewpoint in which the icons comprise representations of cameras and are shown at respective positions relative to the model which correspond substantially to the camera positions relative to the object.

22. In a method of operating an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the improvement wherein the method includes:

displaying a set of icons, each being associated with a respective one of the camera images of the object;

receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

determining a selected camera image from the set of camera images corresponding to the selected icon;

displaying the selected image;

determining position data representative of a selected camera position from which the selected image was taken;

generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

displaying the model image for visual comparison with the selected image by the user.

23. In an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the improvement comprising:

display means for displaying a set of icons, each being associated with a respective one of the camera images of the object;

means for receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

means for determining a selected camera image from the set of camera images corresponding to the selected icon whereby the display means is operable to display the selected image;

means for determining position data representative of a selected camera position from which the selected image was taken;

means for generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

control means for controlling the display means to display the model image for visual comparison with the selected image by the user.

24. A method of processing data defining a three dimensional computer model of an object, data defining a plurality of camera images of the object, and data defining the viewpoint of each camera image relative to the computer model, the method comprising the steps of:

displaying a plurality of icons, each icon being associated with a respective one of the camera images of the object;

receiving a selection signal generated in response to user-actuation of an input device defining a user-selection of a displayed icon;

identifying the camera image associated with the selected icon and the viewpoint of the identified camera image;

displaying the image data of the identified camera image;

rendering the three dimensional computer model data to generate image data showing a virtual image of the three dimensional computer model from the viewpoint of the identified camera image; and

displaying the virtual image for visual comparison with the camera image.

25. A method as claimed in claim 24, wherein the step of rendering the three dimensional computer model data comprises rendering the three dimensional computer model data using texture data to generate image data showing a virtual image of the three dimensional computer model rendered with texture data.

26. A method as claimed in claim 24, wherein:

data defining the field of view and magnification for each respective camera image is stored, data defining the field of view and magnification of the identified camera images is read, and

the step of rendering the three dimensional computer model data comprises rendering the three dimensional computer model data in accordance with the viewpoint, field of view and magnification of the identified camera image to generate virtual image data showing a view of the three dimensional computer model that is substantially the same as the view of the object in the identified camera image.

27. A method as claimed in claim 24, wherein the step of displaying the icons comprises displaying a representation of the model and displaying each icon at a respective position relative to the representation of the model which corresponds substantially to the position from which the camera image associated with the icon was recorded relative to the object.

28. A method as claimed in claim 27, wherein each icon is displayed together with a representation of the viewing direction from which the associated camera image was recorded.

29. A method as claimed in claim 27, wherein each icon is displayed as a representation of a camera.

37. A storage medium storing computer program instructions to program a programmable processing apparatus to become operable to perform a method as claimed in any one of claims 24 to 29.

38. A physically-embodied computer program product carrying computer program instructions to program a programmable apparatus to become operable to perform a method as claimed in any one of claims 24 to 29.

30. An apparatus for processing data defining a three dimensional computer model of an object, data defining a plurality of camera images of the object, and data defining the viewpoint of each camera image relative to the computer model, the apparatus comprising:

an icon data generator operable to generate icon data defining a plurality of icons for display, each icon being associated with a respective one of the camera images of the object;

a selection signal receiver operable to receive a selection signal generated in response to user-actuation of an input device defining a user-selection of an icon;

a camera data identifier operable to identify the camera image associated with the selected icon and the viewpoint of the identified camera image;

a renderer operable to render the three dimensional computer model data to generate image data defining a virtual image of the three dimensional computer model from the viewpoint of the identified camera image; and

a display data generator operable to generate image data for display displaying the virtual image and the identified camera image for visual comparison.

31. An apparatus as claimed in claim 30, wherein the renderer is operable to render the three dimensional computer model data using texture data to generate image data defining a virtual image of the three dimensional computer model rendered with texture data.

32. An apparatus as claimed in claim 30, wherein the apparatus is configured to store data defining the field of view and magnification for each respective camera image, and wherein:

the camera data identifier is operable to read data defining the field of view and magnification of the identified camera images, and

the renderer is operable to render the three dimensional computer model data in accordance with the viewpoint, field of view and magnification of the identified camera image to generate virtual image data defining a view of the three dimensional computer model that is substantially the same as the view of the object in the identified camera image.

33. An apparatus as claimed in claim 30, wherein the icon data generator is operable to generate the icon data to define a representation of the model and to define each icon at a respective position relative to the representation of the model which corresponds substantially to the position from which the camera image associated with the icon was recorded relative to the object.

34. An apparatus as claimed in claim 33, wherein the icon data generator is operable to generate the icon data to define each icon with a representation of the viewing direction from which the associated camera image was recorded.

35. An apparatus as claimed in claim 33, wherein the icon data generator is operable to generate the icon data to define each icon as a representation of a camera.

36. An apparatus for processing data defining a three-dimensional computer model of an object, data defining a plurality of camera images of the object, and data defining the viewpoint of each camera image relative to the computer model, the apparatus comprising:

means for generating icon data defining a plurality of icons for display, each icon being associated with a respective one of the camera images of the object;

means for receiving a selection signal generated in response to user-actuation of an input means defining a user-selection of an icon;

means for identifying the camera image associated with the selected icon and the viewpoint of the identified camera image;

means for rendering the three dimensional computer model data to generate image data defining a virtual image of the three dimensional computer model from the viewpoint of the identified camera image; and

means for generating image data for display displaying the virtual image and the identified camera image for visual comparison.