Image processing apparatus

• US 20040247174A1
• Filed: 03/08/2004
• Published: 12/09/2004
• Est. Priority Date: 01/20/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Haslam, Jane, Lyons, Alexander Ralph, Taylor, Richard Ian, Rowe, Simon Michael
• Granted Patent: US 7,508,977 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

A 3-D model of an object is created by processing images taken from a series of camera positions. An initial sequence of the images is processed to define respective image co-ordinates of matching features to generate a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images. A new image is added to the sequence and processed to obtain a camera solution for a corresponding new virtual camera for use in generating further model data. Processing of the new image comprises;

(a) identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

Claims

1. A method of creating a 3-D model of an object by processing images taken from a series of respective camera positions relative to the object;

the method comprising;

processing an initial sequence of the images to define respective image co-ordinates of matching features to generate therefrom a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images; and

adding a new image to the sequence and processing the new image to obtain a camera solution for a corresponding new virtual camera for use in generating further model data;

wherein the processing of the new image comprises;

(a) identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

2. A method as claimed in claim 1 wherein the solving process of step (c) comprises:

selecting a subset of image points and corresponding model points defined by the set of co-ordinates;

calculating a candidate camera solution from the geometrical relationship between the points defined by the subset;

repeating the selecting and calculating step for different subsets to obtain successive candidate camera solutions;

evaluating the candidate camera solutions; and

selecting a best candidate camera solution on the basis of the evaluating step.

3. A method as claimed in claim 2 wherein each subset comprises a selection of three model points and respective image points, the three model points defining apices of a first triangle and the three image points defining apices of a second triangle, and whereby the geometrical relationship is defined by the second triangle constituting a mapping of the first triangle onto the image plane.

4. A method as claimed in claim 3 wherein the mapping comprises a perspective mapping.

5. A method as claimed in claim 2 wherein the evaluating step comprises mapping model points defined by the set of existing model data into the image plane using the candidate solution to obtain coordinates of reference points in the image plane;

and

correlating the reference points with the image points.

6. A method as claimed in claim 5 wherein the correlating step comprises determining whether each image point lies within a predetermined number of pixel units from a respective reference point and counting the number of such image points as a measure of correlation.

7. A method as claimed in claim 6, wherein the step of selecting the best candidate camera solution comprises selection according to the highest measure of correlation.

8. A method as claimed in claim 2 comprising determining a set of inliers for the best candidate solution wherein each inlier comprises data defining co-ordinates of a model point together with co-ordinates of corresponding image points in each of at least three images of the sequence.

9. A method as claimed in claim 2, comprising the step of using the best camera solution to project the remainder of the set of existing model data into the image plane of the new virtual camera to obtain a set of further reference points in the image plane;

performing matching using the further reference points to identify matching further image points of the new image; and

adding the co-ordinates of the further image points and respective model points to the set of co-ordinates determined at step (b) to thereby obtain an enlarged set of co-ordinates.

10. A method as claimed in claim 9 comprising processing the enlarged set of co-ordinates using the solving process of step (c) to obtain a revised result for the best camera solution.

11. A method as claimed in claim 10 further comprising generating further model data in accordance with the revised camera solution using image co-ordinates of matching features in the new image and preceding images of the sequence, and adding the further model data to the set of model data to form an expanded set of model data.

12. A method as claimed in claim 11 further comprising repeating the calculation of camera solutions for the sequence of images using the expanded set of model data.

28. A computer program comprising processor implementable instructions for carrying out a method as claimed in claim 1.

29. A storage medium storing processor implementable instructions for carrying out a method as claimed in claim 1.

30. An electrical signal carrying processor implementable instructions for carrying out a method as claimed in claim 1.

32. A method as claimed in claim 30 including displaying in the model image a line representing the locus.

13. Apparatus for creating a 3-D model of an object by processing images taken from a series of respective camera positions relative to the object;

the apparatus comprising;

means for processing an initial sequence of the images to define respective image co-ordinates of matching features to generate therefrom a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images; and

means for adding a new image to the sequence and processing the new image to obtain a camera solution for a corresponding new virtual camera for use in generating further model data;

wherein the means for processing of the new image comprises;

(a) means for identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) means for determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) solving means for processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

14. Apparatus as claimed in claim 13 wherein the solving means comprises:

means for selecting a subset of image points and corresponding model points defined by the set of co-ordinates;

means for calculating a candidate camera solution from the geometrical relationship between the points defined by the subset;

means for repeating the selecting and calculating step for different subsets to obtain successive candidate camera solutions;

means for evaluating the candidate camera solutions; and

means for selecting a best candidate camera solution.

15. Apparatus as claimed in claim 14 wherein each subset comprises a selection of three model points and respective image points, the three, model points defining apices of a first triangle and the three image points defining apices of a second triangle, and whereby the geometrical relationship is defined by the second triangle constituting a mapping of the first triangle onto the image plane.

16. Apparatus as claimed in claim 15 wherein the mapping comprises a perspective mapping.

17. Apparatus as claimed in claim 14 wherein the evaluating means comprises means for mapping model points defined by the set of existing model data into the image plane using the candidate solution to obtain coordinates of reference points in the image plane;

and

means for correlating the reference points with the image points.

18. Apparatus as claimed in claim 17, wherein the correlating means comprises means for determining whether each image point lies within a predetermined number of pixel units from a respective reference point and counting the number of such image points as a measure of correlation.

19. Apparatus as claimed in claim 18, wherein the means for selecting the best candidate camera solution comprises means for selection according to the highest measure of correlation.

20. Apparatus as claimed in claim 14 comprising means for determining a set of inliers for the best candidate solution wherein each inlier comprises data defining co-ordinates of a model point together with co-ordinates of corresponding image points in each of at least three images of the sequence.

21. Apparatus as claimed in claim 14 wherein the solving means is operable to use the best camera solution to project the remainder of the set of existing model data into the image plane of the new virtual camera to obtain a set of further reference points in the image plane;

to perform matching using the further reference points to identify matching further image points of the new image; and

to add the co-ordinates of the further image points and respective model points to the set of co-ordinates determined at step (b) to thereby obtain an enlarged set of co-ordinates.

22. Apparatus as claimed in claim 20 wherein the solving means is operable to process the enlarged set of co-ordinates to obtain a revised result for the best camera solution.

23. Apparatus as claimed in claim 22 further comprising means for generating further model data in accordance with the revised camera solution using image co-ordinates of matching features in the new image and preceding images of the sequence, and adding the further model data to the set of model data to form an expanded set of model data.

24. Apparatus as claimed in claim 23 further comprising means for repeating the calculation of camera solutions for the sequence of images using the expanded set of model data.

25. In a method of creating a 3-D model of an object by processing images taken from a series of respective camera positions relative to the object;

the method comprising;

processing an initial sequence of the images to define respective image co-ordinates of matching features to generate therefrom a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images;

an improvement comprising;

adding a new image to the sequence and processing the new image to obtain a camera solution for a corresponding new virtual camera for use in generating further model data;

wherein the processing of the new image comprises;

(a) identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

26. In a apparatus for creating a 3-D model of an object by processing images taken from a series of respective camera positions relative to the object;

the apparatus comprising;

means for processing an initial sequence of the images to define respective image co-ordinates of matching features to generate therefrom a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images;

an improvement comprising;

means for adding a new image to the sequence and processing the new image to obtain a camera solution for a corresponding new virtual camera for use in generating further model data;

wherein the means for processing of the new image comprises;

(a) means for identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) means for determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) means for processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

27. In an apparatus for creating a 3-D model of an object by processing images taken from a series of respective camera positions relative to the object;

processing an initial sequence of the images to define respective image co-ordinates of matching features to generate therefrom a set of model data defining model points in a 3-D space of the model and to obtain respective camera solutions representative of positions and orientations of virtual cameras in the 3-D space defining views of the model corresponding to the images; and

adding a new image to the sequence and processing the new image to obtain a camera solution for a corresponding new virtual camera for use in generating further model data;

a method of performing the processing of the new image comprising;

(a) identifying a plurality of image points in the new image which are matched to a respective plurality of image points of at least one preceding image of the sequence for which respective 3-D model data defining corresponding model points exists;

(b) determining a set of 2-D image co-ordinates of the identified image points in the new image and co-ordinates of respective model points; and

(c) processing the set of 2-D image point co-ordinates and respective 3-D model point co-ordinates to obtain the camera solution for the new image using a solving process in which the position and orientation of an image plane representative of the new virtual camera are calculated from a geometrical relationship in the 3-D model space between model points and image points defined by the set of co-ordinates.

31. A method of operating an apparatus for generating model data representative of a model in a three dimensional space from image data representative of a set of camera images of an object;

the apparatus performing the steps of;

displaying a model image based on an existing set of model data;

displaying one of the camera images of the object for selection by a user of an additional feature to be represented by additional model data;

receiving an image point selection signal responsive to user actuation of an input means and identifying co-ordinates of an image point in the camera image defining the selected additional feature;

calculating a locus in the three dimensional space defining positions of possible model points corresponding to the image point and consistent with the geometric relationship between the object and a camera position from which the displayed camera image was taken;

displaying a position indicator in the model image at co-ordinates in the model image corresponding to one of the possible model points on the locus;

receiving positioning signals responsive to user actuation of the input means and updating the co-ordinates of the position indicator such that movement of the position indicator is constrained to follow a trajectory in the model image corresponding to the locus;

receiving a model point selecting signal responsive to user actuation of the input means and determining selected co-ordinates of the position indicator to be the position indicator co-ordinates at the time of receiving the model point selecting signal; and

determining co-ordinates of the additional model point in the three dimensional space corresponding to the selected co-ordinates of the position indicator.

33. A method as claimed in claim 31 wherein the locus is a straight line in the three dimensional space.

34. A method as claimed in claim 31 wherein the input means comprises a computer mouse and wherein said positioning signals are responsive to user actuation of the mouse by clicking the mouse during movement and releasing the mouse at a selected position to generate the model point selecting signal.

35. A method as claimed in claim 31 including the step of displaying a symbol representative of the additional model point at a model image point corresponding to the selected co-ordinates.

36. A method as claimed in claim 35 comprising the further step of editing the position of the additional model point in response to receiving further positioning signals and model point selecting signals responsive to user actuation of the input means.

37. A method as claimed in claim 31 including the further step of receiving a processing instruction signal and, responsive to said signal, implementing a model generating process to incorporate the additional model point into the model data.

38. A method as claimed in claim 37 including the step of generating surface elements of the model from the model data including the additional model point and displaying said surface elements in the model image.

47. A computer program comprising processor implementable instructions for carrying out a method as claimed in claim 31.

48. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 31.

49. An electrical signal carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 31.

39. Apparatus for generating model data representative of a model in a three dimensional space from image data representative of a set of camera images of an object;

the apparatus comprising;

an interface comprising display means operable to display images to a user and input means responsive to user actuation;

control means operable to control the display means to display a model image based on an existing set of model data and to display one of the camera images of the object for selection by a user of an additional feature to be represented by additional model data;

receiving means for receiving an image point selection signal responsive to user actuation of the input means and identifying co-ordinates of an image point in the camera image defining the selected additional feature;

calculating means for calculating a locus in the three dimensional space defining positions of possible model points corresponding to the image point and consistent with the geometric relationship between the object and a camera position from which the displayed camera image was taken;

the control means being further operable to control the display means to display a position indicator in the model image at co-ordinates in the model image corresponding to one of the possible model points on the locus;

the apparatus further comprising means for receiving positioning signals responsive to user actuation of the input means and updating the co-ordinates of the position indicator such that movement of the position indicator is constrained to follow a trajectory in the model image corresponding to the locus;

means for receiving a model point selecting signal responsive to user actuation of the input means and determining selected co-ordinates of the position indicator to be the position indicator co-ordinates at the time of receiving the model point selecting signal; and

means for determining co-ordinates of the additional model point in the three dimensional space corresponding to the selected co-ordinates of the position indicator.

40. Apparatus as claimed in claim 39 wherein the control means is operable to control the display means to display in the model image a line representing the locus.

41. Apparatus as claimed in claim 39 wherein the calculating means is operable to calculate the locus as a straight line in the three dimensional space.

42. Apparatus as claimed in claim 39 wherein the input means comprises a computer mouse and wherein said positioning signals are responsive to user actuation of the mouse by clicking the mouse during movement and releasing the mouse at a selected position to generate the model point selecting signal.

43. Apparatus as claimed in claim 39 wherein the control means is operable to control the display means to display a symbol representative of the additional model point at a model image point corresponding to the selected co-ordinates.

44. Apparatus as claimed in claim 43 comprising editing means for editing the position of the additional model point in response to receiving further positioning signals and model point selecting signals responsive to user actuation of the input means.

45. Apparatus as claimed in claim 39 including model generating means operable to receive a processing instruction signal and, responsive to said signal, to implement a model generating process to incorporate the additional model point into the model data.

46. Apparatus as claimed in claim 45 wherein the model generating means is operable to generate surface elements of the model from the model data including the additional model point and wherein the control means is operable to control the display to display said surface elements in the model image.

50. A method of operating an apparatus for generating model data defining a model in a three dimensional space, the model data comprising co-ordinates defining model points and surface elements generated with reference to the model points;

the method comprising editing an existing set of model data by the steps of;

adding a new model point to the existing set of model data;

projecting the new model point onto the model and identifying a selected one of the surface elements onto which the new model point is projected;

identifying a subset of the model points which define the generation of the selected surface element;

adding the new model point to the subset to form an edited subset of model points; and

generating one or more edited surface elements from the edited subset of model points to replace the selected surface element.

51. A method as claimed in claim 50 wherein the projecting step comprises receiving input data defining a centre of projection and projecting the new model point onto the model in a direction of projection along a ray generated through the centre of projection and the new model point.

52. A method as claimed in claim 51 wherein the existing set of model data is generated by processing image data representative of camera images of an object to be modelled.

53. A method as claimed in claim 52 wherein the step of receiving input data comprises receiving an image selection signal for selecting one of said camera images, and defining the centre of projection for projecting the model point to be co-ordinates representative of a camera position from which the selected camera image was taken.

54. A method as claimed in claim 53 including the step of displaying a set of camera images and receiving a selection signal responsive to user actuation of an input means to select the selected camera image.

55. A method as claimed in claim 52 wherein the step of receiving input data comprises receiving an image selection signal for selecting one of said camera images, and receiving an image point selection signal defining co-ordinates of an image point in said selected camera image corresponding to the new model point.

56. A method as claimed in claim 55 including the step of calculating in the three dimensional space co-ordinates of the centre of projection to correspond to the position of the image point in an image plane of the camera.

57. A method as claimed in claim 51 including the step of determining whether a plurality of surface elements are intersected by the ray and, if so, determining the selected surface to be whichever of the intersected surface elements is closest to the new model point.

58. A method as claimed in claims 50 wherein the surface elements comprise triangular facets and wherein each subset of the model points defining the generation of the selected surface element comprises three model points constituting apices of the triangular facets.

68. A computer program comprising processor implementable instructions for carrying out a method as claimed in claim 50.

69. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 52.

70. An electrical signal carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 50.

59. Apparatus for generating model data defining a model in a three dimensional space, the model data comprising co-ordinates defining model points and surface elements generated with reference to the model points, the apparatus being operable to edit an existing set of model data and comprising;

means for adding a new model point to the existing set of model data;

means for projecting the new model point onto the model and identifying a selected one of the surface elements onto which the new model point is projected;

means for identifying a subset of the model points which define the generation of the selected surface element;

means for adding the new model point to the subset to form an edited subset of model points; and

means for generating one or more edited surface elements from the edited subset of model points to replace the selected surface element.

60. Apparatus as claimed in claim 59 wherein the projecting means comprises receiving means for receiving input data defining a centre of projection, the projecting means being operable to project the new model point onto the model in a direction of projection along a ray generated through the centre of projection and the new model point.

61. Apparatus as claimed in claim 60 wherein the existing set of model data is generated by processing image data representative of camera images of an object to be modelled.

62. Apparatus as claimed in claim 61 wherein the receiving means is operable to receive an image selection signal for selecting one of said camera images, and to define the centre of projection for projecting the model point to be co-ordinates representative of a camera position from which the selected camera image was taken.

63. Apparatus as claimed in claim 62 comprising interface means for displaying a set of camera images and receiving a selection signal responsive to user actuation of an input means to select the selected camera image.

64. Apparatus as claimed in claim 61 wherein the receiving means is operable to receive an image selection signal for selecting one of said camera images, and to receive an image point selection signal defining co-ordinates of an image point in said selected camera image corresponding to the new model point.

65. Apparatus as claimed in claim 64 including calculating means for calculating in the three dimensional space co-ordinates of the centre of projection to correspond to the position of the image point in an image plane of the camera.

66. Apparatus as claimed in claim 60 including means for determining whether a plurality of surface elements are intersected by the ray and, if so, determining the selected surface to be whichever of the intersected surface elements is closest to the new model point.

67. Apparatus as claimed in claim 60 wherein the surface elements comprise triangular facets and wherein each subset of the model points defining the generation of the selected surface element comprises three model points constituting apices of the triangular facets.

71. A method of operating an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the method comprising;

displaying a set of icons, each being associated with a respective one of the camera images of the object;

receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

determining a selected camera image from the set of camera images corresponding to the selected icon;

displaying the selected image;

determining position data representative of a selected camera position from which the selected image was taken;

generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

displaying the model image for visual comparison with the selected image by the user.

72. A method as claimed in claim 71 including the step of generating the icons in response to receiving a mode selection input.

73. A method as claimed in claim 71 wherein the icons are generated as thumbnail images of the respective camera images.

74. A method as claimed in claim 73 wherein the step of displaying the set of icons comprises displaying the icons in an array and displaying links between the icons such that each pair of icons corresponding to adjacent camera positions in a positional sequence of the camera positions is joined by a respective link.

75. A method as claimed in claim 74 wherein the icons are displayed in a linear array.

76. A method as claimed in claim 71 wherein the selected camera image and the model image are displayed in respective windows and including the step of providing relative movement of the windows in response to receiving window movement input signals.

77. A method as claimed in claim 76 wherein the icons are displayed in a further window and including the step of facilitating movement of the further window relative to the image windows in response to window movement input signals.

78. A method as claimed in claim 71 comprising generating the selection signal by operation of a pointing means for user actuation in selecting one of the displayed icons.

79. A method as claimed in claim 71 wherein displaying the set of icons comprises displaying a view of the model from a viewpoint in which the icons comprise representations of cameras and are shown at respective positions relative to the model which correspond substantially to the camera positions relative to the object.

89. A computer program comprising processor implementable instructions for carrying out a method as claimed in claim 71.

90. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 71.

91. An electrical signal carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 71.

80. Apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, the apparatus comprising;

display means for displaying a set of icons, each being associated with a respective one of the camera images of the object;

means for receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

means for determining a selected camera image from the set of camera images corresponding to the selected icon whereby the display means is operable to display the selected image;

means for determining position data representative of a selected camera position from which the selected image was taken;

means for generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

control means for controlling the display means to display the model image for visual comparison with the selected image by the user.

81. Apparatus as claimed in claim 80 further comprising means for generating the icons in response to receiving a mode selection input.

82. Apparatus as claimed in claims 80 wherein icon generating means is operable to generate the icons as thumbnail images of the respective camera images.

83. Apparatus as claimed in claim 82 wherein the control means is operable to control the display means to display the set of icons in an array and to display links between the icons such that each pair of icons corresponding to adjacent camera positions in a positional sequence of the camera positions is joined by a respective link.

84. Apparatus as claimed in claim 83 wherein the control means is operable to control the display means to display the icons in a linear array.

85. Apparatus as claimed in claim 80 wherein control means is operable control the display means to display the selected camera image and the model image in respective windows and to provide relative movement of the windows in response to receiving window movement input signals.

86. Apparatus as claimed in claim 85 wherein the control means is operable to control the display means to display the icons in a further window to facilitate movement of the further window relative to the camera image window and model image window in response to window movement input signals.

87. Apparatus as claimed in claim 80 wherein the means for generating the selection signal comprises a pointing means for user actuation in selecting one of the displayed icons

88. Apparatus as claimed in claim 80 wherein the control means is operable to control the display means for displaying the set of icons by displaying a view of the model from a viewpoint in which the icons comprise representations of cameras and are shown at respective positions relative to the model which correspond substantially to the camera positions relative to the object.

92. In a method of operating an apparatus for generating model data representative of a model in a three dimensional space from image data representative of a set of camera images of an object, an improvement comprising;

the apparatus performing the steps of;

displaying a model image based on an existing set of model data;

displaying one of the camera images of the object for selection by a user of an additional feature to be represented by additional model data;

receiving an image point selection signal responsive to user actuation of an input means and identifying co-ordinates of an image point in the camera image defining the selected additional feature;

calculating a locus in the three dimensional space defining positions of possible model points corresponding to the image point and consistent with the geometric relationship between the object and a camera position from which the displayed camera image was taken;

displaying a position indicator in the model image at co-ordinates in the model image corresponding to one of the possible model points on the locus;

receiving positioning signals responsive to user actuation of the input means and updating the co-ordinates of the position indicator such that movement of the position indicator is constrained to follow a trajectory in the model image corresponding to the locus;

receiving a model point selecting signal responsive to user actuation of the input means and determining selected co-ordinates of the position indicator to be the position indicator co-ordinates at the time of receiving the model point selecting signal; and

determining co-ordinates of the additional model point in the three dimensional space corresponding to the selected co-ordinates of the position indicator.

93. In an apparatus for generating model data representative of a model in a three dimensional space from image data representative of a set of camera images of an object;

an improvement wherein the apparatus comprises;

an interface comprising display means operable to display images to a user and input means responsive to user actuation;

control means operable to control the display means to display a model image based on an existing set of model data and to display one of the camera images of the object for selection by a user of an additional feature to be represented by additional model data;

receiving means for receiving an image point selection signal responsive to user actuation of the input means and identifying co-ordinates of an image point in the camera image defining the selected additional feature;

calculating means for calculating a locus in the three dimensional space defining positions of possible model points corresponding to the image point and consistent with the geometric relationship between the object and a camera position from which the displayed camera image was taken;

the control means being further operable to control the display means to display a position indicator in the model image at co-ordinates in the model image corresponding to one of the possible model points on the locus;

the apparatus further comprising means for receiving positioning signals responsive to user actuation of the input means and updating the co-ordinates of the position indicator such that movement of the position indicator is constrained to follow a trajectory in the model image corresponding to the locus;

means for receiving a model point selecting signal responsive to user actuation of the input means and determining selected co-ordinates of the position indicator to be the position indicator co-ordinates at the time of receiving the model point selecting signal; and

means for determining co-ordinates of the additional model point in the three dimensional space corresponding to the selected co-ordinates of-the position indicator.

94. In an apparatus for generating model data representative of a model in a three dimensional space from image data representative of a set of camera images of an object, a method wherein;

the apparatus performs the steps of;

displaying a model image based on an existing set of model data;

displaying one of the camera images of the object for selection by a user of an additional feature to be represented by additional model data;

receiving an image point selection signal responsive to user actuation of an input means and identifying co-ordinates of an image point in the camera image defining the selected additional feature;

calculating a locus in the three dimensional space defining positions of possible model points corresponding to the image point and consistent with the geometric relationship between the object and a camera position from which the displayed camera image was taken;

displaying a position indicator in the model image at co-ordinates in the model image corresponding to one of the possible model points on the locus;

receiving positioning signals responsive to user actuation of the input means and updating the co-ordinates of the position indicator such that movement of the position indicator is constrained to follow a trajectory in the model image corresponding to the locus;

receiving a model point selecting signal responsive to user actuation of the input means and determining selected co-ordinates of the position indicator to be the position indicator co-ordinates at the time of receiving the model point selecting signal; and

determining co-ordinates of the additional model point in the three dimensional space corresponding to the selected co-ordinates of the position indicator.

95. In a method of operating an apparatus for generating model data defining a model in a three dimensional space, the model data comprising co-ordinates defining model points and surface elements generated with reference to the model points;

an improvement wherein the method comprises editing an existing set of model data by the steps of;

adding a new model point to the existing set of model data;

projecting the new model point onto the model and identifying a selected one of the surface elements onto which the new model point is projected;

identifying a subset of the model points which define the generation of the selected surface element;

adding the new model point to the subset to form an edited subset of model points; and

generating one or more edited surface elements from the edited subset of model points to replace the selected surface element.

96. In an apparatus for generating model data defining a model in a three dimensional space, the model data comprising co-ordinates defining model points and surface elements generated with reference to the model points, an improvement wherein the apparatus is operable to edit an existing set of model data and comprises;

means for adding a new model point to the existing set of model data;

means for projecting the new model point onto the model and identifying a selected one of the surface elements onto which the new model point is projected;

means for identifying a subset of the model points which define the generation of the selected surface element;

means for adding the new model point to the subset to form an edited subset of model points; and

means for generating one or more edited surface elements from the edited subset of model points to replace the selected surface element.

97. In an apparatus for generating model data defining a model in a three dimensional space, the model data comprising co-ordinates defining model points and surface elements generated with reference to the model points;

a method comprising editing an existing set of model data by the steps of;

adding a new model point to the existing set of model data;

projecting the new model point onto the model and identifying a selected one of the surface elements onto which the new model point is projected;

identifying a subset of the model points which define the generation of the selected surface element;

adding the new model point to the subset to form an edited subset of model points; and

generating one or more edited surface elements from the edited subset of model points to replace the selected surface element.

98. In a method of operating an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, an improvement wherein the method comprises;

displaying a set of icons, each being associated with a respective one of the camera images of the object;

receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

determining a selected camera image from the set of camera images corresponding to the selected icon;

displaying the selected image;

determining position data representative of a selected camera position from which the selected image was taken;

generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

displaying the model image for visual comparison with the selected image by the user.

99. In an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, an improvement wherein the apparatus comprises;

display means for displaying a set of icons, each being associated with a respective one of the camera images of the object;

means for receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

means for determining a selected camera image from the set of camera images corresponding to the selected icon whereby the display means is operable to display the selected image;

means for determining position data representative of a selected camera position from which the selected image was taken;

means for generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

control means for controlling the display means to display the model image for visual comparison with the selected image by the user.

100. In an apparatus for generating model data representative of a three dimensional model of an object from input signals representative of a set of camera images of the object taken from a plurality of camera positions, a method comprising;

displaying a set of icons, each being associated with a respective one of the camera images of the object;

receiving a selection signal responsive to user actuation of an input means whereby the selection signal identifies a selected one of the icons;

determining a selected camera image from the set of camera images corresponding to the selected icon;

displaying the selected image;

determining position data representative of a selected camera position from which the selected image was taken;

generating in accordance with said model a model image representative of a view of the model from a viewpoint corresponding to the position data; and

displaying the model image for visual comparison with the selected image by the user.

102. A method as claimed in claim 100 wherein the step of displaying the first magnified image comprises displaying the first magnified image in a first window which overlays a fixed portion of the first image.

103. A method as claimed in claim 100 wherein the step of displaying of the fiducial means comprises displaying a graticule.

104. A method as claimed in claim 100 including the step of sampling the magnified image data at the time of receiving the selection signal, storing the sampled data and continuing to display the first magnified image as a static image corresponding to the stored image data.

105. A method as claimed in claim 100 including the step of displaying a second image representative of a second frame of said image data;

receiving pointing signals responsive to user actuation of the pointing device and displaying the cursor in the second image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating magnified image data representative of a second magnified image of a portion of the second image local to the cursor position and in fixed relationship thereto, and continuously updating the magnified image data in response to changes in the cursor position;

displaying the second magnified image simultaneously with the second image with second fiducial means indicating an image point in the second magnified image corresponding to the image point indicated in the second image at the cursor position; and

receiving a selection signal responsive to user actuation of said pointing device and representative of co-ordinates of a second selected point in the second image indicated by the current cursor position.

106. A method as claimed in claim 105 wherein the second magnified image is displayed in a second window which overlays a fixed portion of the second image.

107. A method as claimed in claim 105 including the step of storing co-ordinates of the first and second selected points constituting matching points in the first and second images respectively.

108. A method as claimed in claim 107 including the step of processing the co-ordinates of the matching points to generate model data for a model in a three dimensional space of an object represented in camera images from which said image data is derived.

101. A method of operating an apparatus for processing image data in accordance with user selected co-ordinates of displayed images representative of said image data;

the apparatus performing the steps of;

displaying a first image representative of a first frame selected from said image data;

receiving pointing signals responsive to user actuation of a pointing device and displaying a cursor in the first image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating magnified image data representative of a first magnified image of a portion of the first image local to the cursor position and in fixed relationship thereto, and continuously updating the magnified image data in response to changes in the cursor position;

displaying the first magnified image simultaneously with the first image together with fiducial means indicating an image point in the first magnified image corresponding to the image point indicated in the first image at the cursor position; and

receiving a selection signal responsive to user actuation of said pointing device and representative of co-ordinates of a first selected point in the first image indicated by the current cursor position.

117. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 101.

118. An electrical signal carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 101.

119. A computer program comprising processor implementable instructions for controlling a processor to carry out a method as claimed in claim 101.

109. Apparatus for processing image data in accordance with user selected co-ordinates of displayed images representative of said image data;

the apparatus comprising;

display means operable to display a first image representative of a first frame selected from said image data;

.

pointing signal receiving means for receiving pointing signals responsive to user actuation of a pointing device and causing the display means to display a cursor in the first image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating means for generating magnified image data representative of a first magnified image of a portion of the first image local to the cursor position and in fixed relationship thereto, and for continuously updating the magnified image data in response to changes in the cursor position;

the display means being further operable to display the first magnified image simultaneously with the first image together with fiducial means indicating an image point in the first magnified image corresponding to the image point indicated in the first image at the cursor position; and

selection signal receiving means for receiving a selection signal responsive to user actuation of said pointing device in use and representative of co-ordinates of a first selected point in the first image indicated by the current cursor position.

110. Apparatus as claimed in claim 109 wherein the display means is operable to display the first magnified image in a first window which overlays a fixed portion of the first image.

111. Apparatus as claimed in claim 109 wherein the fiducial means comprises a graticule.

112. Apparatus as claimed in claim 109 including means for sampling the magnified image data at the time of receiving the selection signal, storing the sampled data and continuing to display the first magnified image as a static image corresponding to the stored image data.

113. Apparatus as claimed in claim 109 wherein the display means is operable to display a second image representative of a second frame of said image data;

the pointing signal receiving means being operable to receive pointing signals responsive to further user actuation of the pointing device and causing the display means to display the cursor in the second image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

the generating means being further operable to generate magnified image data representative of a second magnified image of a portion of the second image local to the cursor position and in fixed relationship thereto, and to continuously update the magnified image data in response to changes in the cursor position;

the display means being operable to display the second magnified image simultaneously with the second image with second fiducial means indicating an image point in the second magnified image corresponding to the image point indicated in the second image at the cursor position; and

the selection signal receiving means being operable to receive a selection signal responsive to user actuation of said pointing device and representative of co-ordinates of a second selected point in the second image indicated by the current cursor position.

114. Apparatus as claimed in claim 113 wherein the second magnified image is displayed in a second window which overlays a fixed portion of the second image.

115. Apparatus as claimed in claim 113 including means for storing co-ordinates of the first and second selected points constituting matching points in the first and second images respectively

116. Apparatus as claimed in claim 115 including means for processing the co-ordinates of the matching points to generate model data for a model in a three dimensional space of an object represented in camera images from which said image data is derived.

120. A method of operating an apparatus for generating model data representative of a model in a three dimensional space of an object from input signals representative of a set of images of the object taken from a plurality of respective camera positions, the apparatus performing the steps of;

displaying a model image derived from the model data and comprising a plurality of primitives for viewing by a user;

receiving at least one primitive selection signal responsive to user actuation of an input means whereby each primitive selection signal identifies a respective selected primitive of the model;

defining a plurality of virtual cameras in the three dimensional space having positions and look directions relative to the model which correspond substantially to those of the respective actual cameras relative to the object;

evaluating which of the virtual cameras is an optimum virtual camera for generating a view of the selected primitives;

identifying from the camera images a first camera image of the plurality of camera images taken from a camera position corresponding to that of the optimum virtual camera.

121. A method as claimed in claim 120 including the step of determining from the camera images a second camera image as being suitable for matching features in the first camera image and displaying the second camera image for comparison by the user with the first camera image.

122. A method as claimed in claim 121 wherein the second camera image is taken from a camera position proximate to the optimum camera position.

123. A method as claimed in claim 121 including the step of receiving feature matching selection signals representative of user matched points in the first and second camera images.

124. A method as claimed in claim 123 including the step of generating updated model data to include additional detail corresponding to the received feature matching selection signals rendering the updated model data to generate an updated model image and displaying the updated model image.

125. A method as claimed in claim 120 wherein the evaluating step comprises;

calculating for a selected primitive an aspect measurement representative of the visibility of the primitive when viewed in projection in the look direction of one of the virtual cameras;

repeating the calculating step to obtain a respective aspect measurement for each of the virtual cameras;

comparing the aspect measurements for the selected primitive and determining a candidate virtual camera to be the virtual camera for which the corresponding aspect measurement is a maximum;

repeating the calculating, comparing and determining steps for each of the selected primitive whereby candidate virtual cameras are determined for each selected primitive; and

choosing the optimum virtual camera on the basis of the frequency with which virtual cameras are determined to be candidate virtual cameras.

126. A method as claimed in claim 125 wherein the primitives comprise facets.

127. A method as claimed in claim 126 wherein the calculation of the aspect measurement comprises, for a given facet and a given virtual camera, calculating a scalar product of a unit vector normal to the facet and a unit vector parallel to the look direction of the virtual camera.

128. A method as claimed in claim 126 wherein the calculation of aspect measurement comprises calculating, for a given facet and for a given virtual camera, an area of the facet when viewed in projection in the look direction of the virtual camera.

129. A method as claimed in claim 120 wherein the input means is a pointing means co-operable with a display means to provide input signals in the form of image co-ordinates of the displayed image.

130. A method as claimed in claim 120 including generating the displayed model image by rendering the image data.

142. A storage medium storing processor implementable instructions for controlling a processor to carry out a method as claimed in claim 120.

143. An electrical signal carrying processor implementable instructions for controlling a processor to carry out a method as claimed in claim 120.

144. A computer program comprising processor implementable instructions for controlling a processor to carry out a method as claimed in claim 120.

131. Apparatus for generating model data representative of a model in a three dimensional space of an object from input signals representative of a set of images of the object taken from a plurality of respective camera positions, the apparatus comprising;

display means and control means operable to control the display means to display a model image derived from the model data and comprising a plurality of primitives for viewing by a user;

means for receiving at least one primitive selection signal responsive to user actuation of an input means whereby each primitive selection signal identifies a respective selected primitive of the model;

means for defining a plurality of virtual cameras in the three dimensional space having positions and look directions relative to the model which correspond substantially to those of the respective actual cameras relative to the object;

evaluating means for evaluating which of the virtual cameras is an optimum virtual camera for generating a view of the selected primitives; and

identifying means for identifying from the camera images a first camera image of the plurality of camera images taken from a camera position corresponding to that of the optimum virtual camera.

132. Apparatus as claimed in claim 131 comprising means for determining from the camera images a second camera image as being suitable for matching features in the first camera image, the control means being operable to control the display means to display the second camera image for comparison by the user with the first camera image.

133. Apparatus as claimed in claim 132 wherein the second camera image is taken from a camera position proximate to the optimum camera position.

134. Apparatus as claimed in claim 132 comprising means for receiving feature matching selection signals representative of user matched points in the first and second camera images.

135. Apparatus as claimed in claim 134 comprising means for generating updated model data to include additional detail corresponding to the received feature matching selection signals, means for rendering the updated model data to generate an updated model image and means for controlling the display means to display the updated model image.

136. Apparatus as claimed in claim 131 wherein the evaluating means comprises;

means for calculating for a selected primitive an aspect measurement representative of the visibility of the primitive when viewed in projection in the look direction of one of the virtual cameras;

means for repeating the calculating step to obtain a respective aspect measurement for each of the virtual cameras;

means for comparing the aspect measurements for the selected primitive and for determining a candidate virtual camera to be the virtual camera for which the corresponding aspect measurement is a maximum;

means for repeating the calculating, comparing and determining steps for each of the selected primitive whereby candidate virtual cameras are determined for each selected primitive; and

means for choosing the optimum virtual camera on the basis of the frequency with which virtual cameras are determined to be candidate virtual cameras.

137. Apparatus as claimed in claim 136 wherein the primitives comprise facets.

138. Apparatus as claimed in claim 137 wherein the means for calculation of the aspect measurement comprises, for a given facet and a given virtual camera, means for calculating a scalar product of a unit vector normal to the facet and a unit vector parallel to the look direction of the virtual camera.

139. Apparatus as claimed in claim 137 wherein the means for calculation of aspect measurement comprises means for calculating, for a given facet and for a given virtual camera, an area of the facet when viewed in projection in the look direction of the virtual camera.

140. Apparatus as claimed in claim 131 wherein the input means is a pointing means co-operable with the display means to provide input signals in the form of image co-ordinates of the displayed image.

141. Apparatus as claimed in claim 131 comprising means for generating the displayed model image by rendering the image data.

145. In a method of operating an apparatus for processing image data in accordance with user selected co-ordinates of displayed images representative of said image data;

an improvement wherein the apparatus performs the steps of;

displaying a first image representative of a first frame selected from said image data;

receiving pointing signals responsive to user actuation of a pointing device and displaying a cursor in the first image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating magnified image data representative of a first magnified image of a portion of the first image local to the cursor position and in fixed relationship thereto, and continuously updating the magnified image data in response to changes in the cursor position;

displaying the first magnified image simultaneously with the first image together with fiducial means indicating an image point in the first magnified image corresponding to the image point indicated in the first image at the cursor position; and

receiving a selection signal responsive to user actuation of said pointing device and representative of co-ordinates of a first selected point in the first image indicated by the current cursor position.

146. In an apparatus for processing image data in accordance with user selected co-ordinates of displayed images representative of said image data;

an improvement wherein the apparatus comprises;

display means operable to display a first image representative of a first frame selected from said image data;

pointing signal receiving means for receiving pointing signals responsive to user actuation of a pointing device and causing the display means to display a cursor in the first image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating means for generating magnified image data representative of a first magnified image of a portion of the first image local to the cursor position and in fixed relationship thereto, and for continuously updating the magnified image data in response to changes in the cursor position;

the display means being further operable to display the first magnified image simultaneously with the first image together with fiducial means indicating an image point in the first magnified image corresponding to the image point indicated in the first image at the cursor position; and

selection signal receiving means for receiving a selection signal responsive to user actuation of said pointing device in use and representative of co-ordinates of a first selected point in the first image indicated by the current cursor position.

147. In an apparatus for processing image data in accordance with user selected co-ordinates of displayed images representative of said image data;

a method wherein the apparatus performs the steps of;

displaying a first image representative of a first frame selected from said image data;

receiving pointing signals responsive to user actuation of a pointing device and displaying a cursor in the first image indicating an image point at a cursor position controlled by the pointing signals such that the cursor position is updated to track movement of the pointing device;

generating magnified image data representative of a first magnified image of a portion of the first image local to the cursor position and in fixed relationship thereto, and continuously updating the magnified image data in response to changes in the cursor position;

displaying the first magnified image simultaneously with the first image together with fiducial means indicating an image point in the first magnified image corresponding to the image point indicated in the first image at the cursor position; and

receiving a selection signal responsive to user actuation of said pointing device and representative of co-ordinates of a first selected point in the first image indicated by the current cursor position.

148. In a method of operating an apparatus for generating model data representative of a model in a three dimensional space of an object from input signals representative of a set of images of the object taken from a plurality of respective camera positions, an improvement wherein the apparatus performs the steps of;

displaying a model image derived from the model data and comprising a plurality of primitives for viewing by a user;

receiving at least one primitive selection signal responsive to user actuation of an input means whereby each primitive selection signal identifies a respective selected primitive of the model;

defining a plurality of virtual cameras in the three dimensional space having positions and look directions relative to the model which correspond substantially to those of the respective actual cameras relative to the object;

evaluating which of the virtual cameras is an optimum virtual camera for generating a view of the selected primitives;

identifying from the camera images a first camera image of the plurality of camera images taken from a camera position corresponding to that of the optimum virtual camera.

149. In an apparatus for generating model data representative of a model in a three dimensional space of an object from input signals representative of a set of images of the object taken from a plurality of respective camera positions, an improvement whereby the apparatus comprises;

display means and control means operable to control the display means to display a model image derived from the model data and comprising a plurality of primitives for viewing by a user;

means for receiving at least one primitive selection signal responsive to user actuation of an input means whereby each primitive selection signal identifies a respective selected primitive of the model;

means for defining a plurality of virtual cameras in the three dimensional space having positions and look directions relative to the model which correspond substantially to those of the respective actual cameras relative to the object;

evaluating means for evaluating which of the virtual cameras is an optimum virtual camera for generating a view of the selected primitives; and

identifying means for identifying from the camera images a first camera image of the plurality of camera images taken from a camera position corresponding to that of the optimum virtual camera.

150. In an apparatus for generating model data representative of a model in a three dimensional space of an object from input signals representative of a set of images of the object taken from a plurality of respective camera positions, an improvement whereby the apparatus performs the steps of;

displaying a model image derived from the model data and comprising a plurality of primitives for viewing by a user;

receiving at least one primitive selection signal responsive to user actuation of an input means whereby each primitive selection signal identifies a respective selected primitive of the model;

defining a plurality of virtual cameras in the three dimensional space having positions and look directions relative to the model which correspond substantially to those of the respective actual cameras relative to the object;

evaluating which of the virtual cameras is an optimum virtual camera for generating a view of the selected primitives;

identifying from the camera images a first camera image of the plurality of camera images taken from a camera position corresponding to that of the optimum virtual camera.