Image processing apparatus

• US 6,914,599 B1
• Filed: 01/14/1999
• Issued: 07/05/2005
• Est. Priority Date: 01/14/1998
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Davison, Allan Joseph, Rowe, Simon Michael
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Sealey, Lance W.
• Application Number:
  US09/229,898
• Time in Patent Office: 2,364 Days
• Field of Search: 382/106, 382/107, 382/110, 382/141, 382/154, 382/156, 382/158, 382/170, 382/199, 382/203, 348/49, 348/86, 348/89, 348/113, 348/139, 348/159, 348/705, 348/47, 348/48, 345/419, 345/420, 345/501, 345/531, 345/538, 345/541, 345/552, 345/585, 345/503, 345/506, 345/521, 345/582, 345/427, 706/16, 706/22, 706/26, 396/315, 396/332, 355/40, 355/41, 250/558, 356/12, 356/611, 356/613, 356/638, 702/158
• US Class Current: 345/420
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

In a processing system, video images of moving objects are processed to model the objects in a 3D computer model. Video from multiple cameras is processed to separate objects from their shadows, and to test whether an object is made up of separate objects, which are then modelled separately. Each object is modelled using vertical planes whose bases approximate the object'"'"'s ground footprint, using planes based on object surface planes identified in the image data, or using a single vertical plane. Pixel data from the video images is rendered onto the planes in the models. The video for rendering is selected based on the viewer'"'"'s viewing direction, the camera viewing directions, and quality characteristics of the cameras and image data. If the viewer'"'"'s viewing direction is close to vertical or a plane of an object, a schematic of the objects'"'"' positions is displayed. To account for image data from different cameras being used, successive images are tested for visual discontinuous, and are modified if necessary. Information indicating the accuracy/reliability of the rendered image is displayed.

Claims

1. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the method comprising the steps of:

processing image data from a first of the cameras to identify image data relating to objects in the scene;

processing image data from a second of the cameras to identify image data relating to objects in the scene;

processing the identified image data from the first camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the first camera;

processing the identified image data from the second camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the second camera;

comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

generating object representations in the three-dimensional computer model in dependence upon the height comparisons,

wherein, when the heights of the corresponding representations are not within a predetermined amount of each other, the taller representation is split into a first portion having a height corresponding to the height of the smaller representation and a second portion comprising the remaining part of the taller representation, and wherein a further representation is defined in the three-dimensional model by re-positioning the second portion in the three-dimensional model.

2. A method according to claim 1, wherein the second portion is re-positioned in dependence upon a representation defined on the basis of image data from the camera which produced the smaller representation.

3. A method according to claim 2, wherein the second portion is re-positioned by:

identifying which of the representations defined on the basis of image data from the camera which produced the smaller representation overlaps the image data used to define the taller representation in the image space of the camera which produced the taller representation; and

re-positioning the second portion, in dependence upon the position of the identified representation in the three-dimensional model.

4. A method according to claim 3, wherein the second portion is re-positioned by:

mapping at least part of each representation defined on the basis of image data from the camera which produced the smaller representation from the three-dimensional model to the image space of the camera which produced the taller representation;

determining which projected representation overlaps the image data for the taller representation in the image space of the camera which produced the taller representation; and

re-positioning the second portion in dependence upon the position in the three-dimensional model of the representation which, when projected into the image space of the camera which produced the taller representation, overlapped the image data for the taller representation.

5. A method according to claim 4, wherein the second portion is re-positioned so that the centre of its base is at the same position as the centre of the base of the representation which overlapped the image data for the taller representation.

6. A method according to claim 1, wherein each object representation is defined as a planar surface with its base on a predetermined surface in the three-dimensional computer model and with a position and size in dependence upon a polygon bounding the image data for the object.

7. A method according to claim 6, wherein the polygon is a rectangle.

8. A method according to claim 7, wherein the sides of the rectangle are parallel to the sides of the image.

9. A method according to claim 6, wherein the width of the planar surface is determined by the width of the bounding polygon in the image data, and the height of the planar surface is calculated using the aspect ratio of the bounding polygon in the image data.

10. A method according to claim 6, wherein the planar surface lies within a vertical plane.

11. A method according to claim 1, further comprising the step of generating image data by rendering an image of the three-dimensional computer model in which texture data based on the processed image data is rendered onto the representation of each object.

12. A method according to claim 11, further comprising the step of generating a signal conveying the image data.

13. A method according to claim 12, further comprising the step of recording the signal.

14. A method according to claim 11, further comprising the step of displaying an image of the objects using the generated image data.

15. A method according to claim 11, further comprising the step of making a recording of the image data either directly or indirectly.

18. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 1, 16 and 17.

19. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 1, 16 and 17.

16. A method of image processing in which image data from first and second cameras is processed to identify image data relating to respective objects, the height of each object in a modelling space is determined using the identified image data, and the heights of objects determined using image data from the first camera are compared with the heights of objects determined using image data from the second camera to determine which if any identified image data relates to more than one object,

wherein each object is defined as a planar surface with its base on a predetermined surface in the modelling space and with a position in dependence upon a polygon bounding the image data for the object, and wherein the width of the planar surface is determined by the width of the bounding polygon in the image data, and the height of the planar surface is calculated using the aspect ratio of the bounding polygon in the image data.

17. An image processing method in which image data from a first camera of objects in a scene is processed to identify image data relating to respective objects, and image data from a second camera of the objects in the scene is processed to determine whether any of the identified image data from the first camera relates to more than one object by comparing a size parameter of each object determined from the image data of the first camera with the corresponding size parameter determined from the image data of the second camera.

20. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the apparatus comprising:

means for processing image data from a first of the cameras to identify image data relating to objects in the scene;

means for processing image data from a second of the cameras to identify image data relating to objects in the scene;

means for processing the identified image data from the first camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the first camera;

means for processing the identified image data from the second camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the second camera;

means for comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

means for generating object representations in the three-dimensional computer model in dependence upon the height comparisons,

wherein the apparatus is arranged to perform processing such that, when the heights of the corresponding representations are not within a predetermined amount of each other, the taller representation is split into a first portion having a height corresponding to the height of the smaller representation and a second portion comprising the remaining part of the taller representation, and wherein a further representation is defined in the three-dimensional model by re-positioning the second portion in the three-dimensional model.

21. Apparatus according to claim 20, arranged to perform processing such that the second portion is re-positioned in dependence upon a representation defined on the basis of image data from the camera which produced the smaller representation.

22. Apparatus according to claim 21, arranged to perform processing such that the second portion is re-positioned by:

identifying which of the representations defined on the basis of image data from the camera which produced the smaller representation overlaps the image data used to define the taller representation in the image space of the camera which produced the taller representation; and

re-positioning the second portion in dependence upon the position of the identified representation in the three-dimensional model.

23. Apparatus according to claim 22, arranged to perform processing such that the second portion is re-positioned by:

mapping at least part of each representation defined on the basis of image data from the camera which produced the smaller representation from the three-dimensional model to the image space of the camera which produced the taller representation;

determining which projected representation overlaps the image data for the taller representation in the image space of the camera which produced the taller representation; and

re-positioning the second portion in dependence upon the position in the three-dimensional model of the representation which, when projected into the image space of the camera which produced the taller representation, overlapped the image data for the taller representation.

24. Apparatus according to claim 23, arranged to perform processing such that the second portion is re-positioned so that the centre of its base is at the same position as the centre of the base of the representation which overlapped the image data for the taller representation.

25. Apparatus according to claim 20, arranged to perform processing such that each object representation is defined as a planar surface with its base on a predetermined surface in the three-dimensional computer model and with a position and size in dependence upon a polygon bounding the image data for the object.

26. Apparatus according to claim 25, wherein the polygon is a rectangle.

27. Apparatus according to claim 26, wherein the sides of the rectangle are parallel to the sides of the image.

28. Apparatus according to claim 25, arranged to perform processing such that the width of the planar surface is determined by the width of the bounding polygon in the image data, and the height of the planar surface is calculated using the aspect ratio of the bounding polygon in the image data.

29. Apparatus according to claim 25, arranged to perform processing such that the planar surface lies within a vertical plane.

30. Apparatus according to claim 20, further comprising means for generating image data by rendering an image of the three-dimensional computer model in which texture data based on the processed image data is rendered onto the representation of each object.

31. Apparatus according to claim 30 further comprising means for displaying an image of the objects using the generated image data.

32. An image processing apparatus operable to process image data from first and second cameras to identify image data relating to respective objects, to determine the height of each object in a modelling space using the identified image data, and to compare the heights of objects determined using image data from the first camera with the heights of objects determined using image data from the second camera to determine which if any identified image data relates to more than one object, wherein the apparatus is arranged to perform processing such that each object is defined as a planar surface with its base on a predetermined surface in the modelling space and with a position in dependence upon a polygon bounding the image data for the object, and such that the width of the planar surface is determined by the width of the bounding polygon in the image data, and the height of the planar surface is calculated using the aspect ratio of the bounding polygon in the image data.

33. An image processing apparatus operable to process image data from a first camera of objects in a scene to identify image data relating to respective objects, and to process image data from a second camera of the objects in the scene to determine whether any of the identified image data from the first camera relates to more than one object by comparing a size parameter of each object determined from the image data of the first camera with the corresponding size parameter determined from the image data of the second camera.

34. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the method comprising:

processing image data from a first of the cameras to identify image data relating to the object in the scene;

processing image data from a second of the cameras to identify image data relating to the object in the scene;

applying a transformation to the identified image data from the first camera which defines a mapping from the ground plane in the space of the image data of the first camera to a surface in a modelling space;

applying a transformation to the identified image data from the second camera which defines a mapping from the ground plane in the space of the image data of the second camera to the surface in the modelling space;

comparing the transformed image data from the first and second cameras on the surface in the modelling space;

determining which part of the image data represents shadow in dependence upon the comparison results; and

generating a representation of at least the object in the three-dimensional model.

35. A method according to claim 34, further comprising the step of generating a representation of the shadow in the three-dimensional model.

36. A method according to claim 34, wherein the surface in the modelling space is the ground plane in the three-dimensional model.

37. A method according to claim 34, wherein it is determined that aligned parts of the transformed image data represent shadow.

38. A method according to claim 34, further comprising the step of generating image data by rendering an image of the three-dimensional computer model in which texture data based on the processed image data is rendered onto the representation of the object.

39. A method according to claim 38, wherein the image data rendered onto the representation is determined in dependence upon the comparison results.

40. A method according to claim 38, further comprising the step of generating a signal conveying the image data.

41. A method according to claim 40, further comprising the step of recording the signal.

42. A method according to claim 38, further comprising the step of displaying an image of the object using the generated image data.

43. A method according to claim 38, further comprising the step of making a recording of the image data either directly or indirectly.

46. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 34, 44 and 45.

47. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 34, 44 and 45.

44. A method of generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, in which image data from a first camera is processed to identify image data relating to the object and its shadow together, and image data from a second camera is used to determine the identified image data from the first camera which relates to the shadow and the identified image data from the first camera which relates to the object.

45. A method of generating a model of an object in a three-dimensional computer model, in which:

a transformation is applied to image data from a first camera relating to the object and its shadow which maps the image data to a surface to give first transformed image data;

a transformation is applied to image data from a second camera relating to the object and its shadow which maps the image data to the surface to give second transformed image data;

the first transformed image data and the second transformed image data are compared to distinguish transformed image data relating to the object from transformed image data relating to its shadow; and

the object is modelled in dependence upon the transformed image data relating to the object.

48. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

for processing image data from a first of the cameras to identify image data relating to the object in the scene;

means for processing image data from a second of the cameras to identify image data relating to the object in the scene;

means for applying a transformation to the identified image data from the first camera which defines a mapping from the ground plane in the space of the image data of the first camera to a surface in a modelling space;

means for applying a transformation to the identified image data from the second camera which defines a mapping from the ground plane in the space of the image data of the second camera to the surface in the modelling space;

means for comparing the transformed image data from the first and second cameras on the surface in the modelling space;

means for determining which part of the image data represents shadow in dependence upon the comparison results; and

means for generating a representation of at least the object in the three-dimensional model.

49. Apparatus according to claim 48, further comprising means for generating a representation of the shadow in the three-dimensional model.

50. Apparatus according to claim 48, wherein the surface in the modelling space is the ground plane in the three-dimensional model.

51. Apparatus according to claim 48, arranged such that it is determined that aligned parts of the transformed image data represent shadow.

52. Apparatus according to claim 48, further comprising means for generating image data by rendering an image of the three-dimensional computer model in which texture data based on the processed image data is rendered onto the representation of the object.

53. Apparatus according to claim 52, wherein the image data rendered onto the representation is determined in dependence upon the comparison results.

54. Apparatus according to claim 52, further comprising means for displaying an image of the object using the generated image data.

55. Apparatus for generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, the apparatus being operable to process image data from a first camera to identify image data relating to the object and its shadow together, and operable to use image data from a second camera to determine the identified image data from the first camera which relates to the shadow and the identified image data from the first camera which relates to the object.

56. Apparatus for generating a model of an object in a three-dimensional computer model, comprising:

means for applying a transformation to image data from a first camera relating to the object and its shadow which maps the image data to a surface to give first transformed image data;

means for applying a transformation to image data from a second camera relating to the object and its shadow which maps the image data to the surface to give second transformed image data;

means for comparing the first transformed image data and the second transformed image data to distinguish transformed image data relating to the object from transformed image data relating to its shadow; and

means for modelling the object in dependence upon the transformed image data relating to the object.

57. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the method comprising:

processing image data from a first of the cameras to identify image data relating to the object in the scene;

processing image data from a second of the cameras to identify image data relating to the object in the scene;

processing the identified image data from the first camera and the identified image data from the second camera to determine a footprint of the object on the ground; and

defining a model of the object in the three-dimensional computer model in dependence upon the determined footprint.

58. A method according to claim 57, wherein the step of processing the identified image data to determine the footprint of the object on the ground comprises:

applying a transformation to the identified image data from the first camera which defines a mapping from the ground plane in the image data of the first camera to a surface in a modelling space;

applying a transformation to the identified image data from the second camera which defines a mapping from the ground plane in the image data of the second camera to the surface in the modelling space; and

comparing the transformed image data on the surface in the modelling space.

59. A method according to claim 58, wherein the surface in the modelling space is the ground plane in the three-dimensional computer model.

60. A method according to claim 58, wherein the outline of the image on the ground is determined in dependence upon the aligned and non-aligned portions of the transformed image data on the surface in the modelling space.

61. A method according to claim 57, wherein, the step of defining the model of the object comprises defining the model using a plurality of vertical planar surfaces.

62. A method according to claim 61, wherein the vertical planar surfaces are defined such that their bases approximate the outline of the object on the ground.

63. A method according to claim 61, wherein each planar surface is a rectangle.

64. A method according to claim 61, wherein each planar surface is defined with a height determined in dependence upon the image data identified from the first camera or the image data identified from the second camera.

65. A method according to claim 64, wherein the height of each planar surface is defined in dependence upon a rectangle bounding some or all of the image data relating to the object identified from the first camera or the second camera.

66. A method according to claim 64, wherein each planar surface is defined to have the same height.

67. A method according to claim 61, further comprising the step of generating a top for the model of the object in dependence upon upper edges of the vertical planar surfaces.

68. A method according to claim 57, further comprising the step of generating image data by rendering an image of the modelled object, in which texture data based on the identified image data from at least one camera is rendered onto the model.

69. A method according to claim 68, wherein each planar surface is mapped onto the image data of the first camera or the second camera, and the image data enclosed by each mapped surface is rendered onto the planar surface in the model.

70. A method according to claim 68, further comprising the step of generating a signal conveying the image data.

71. A method according to claim 70, further comprising the step of recording the signal.

72. A method according to claim 68, further comprising the step of displaying an image of the object using the generated image data.

73. A method according to claim 68, further comprising the step of making a recording of the image data either directly or indirectly.

75. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 57 and 74.

76. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 57 and 74.

74. A method of generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, in which image data from a first camera is processed to identify image data relating to the object, image data from a second camera is used to determine which parts of the identified image data from the first camera relate to parts of the object on or near the ground, and the object is represented in the computer model in dependence thereon.

77. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

means for processing image data from a first of the cameras to identify image data relating to the object in the scene;

means for processing image data from a second of the cameras to identify image data relating to the object in the scene;

means for processing the identified image data from the first camera and the identified image data from the second camera to determine a footprint of the object on the ground; and

means for defining a model of the object in the three-dimensional computer model in dependence upon the determined footprint.

78. Apparatus according to claim 77, wherein the means for processing the identified image data to determine the footprint of the object on the ground comprises:

means for applying a transformation to the identified image data from the first camera which defines a mapping from the ground plane in the image data of the first camera to a surface in a modelling space;

means for applying a transformation to the identified image data from the second camera which defines a mapping from the ground plane in the image data of the second camera to the surface in the modelling space; and

means for comparing the transformed image data on the surface in the modelling space.

79. Apparatus according to claim 78, wherein the surface in the modelling space is the ground plane in the three-dimensional computer model.

80. Apparatus according to claim 78, arranged to determine the outline of the image on the ground in dependence upon the aligned and non-aligned portions of the transformed image data on the surface in the modelling space.

81. Apparatus according to claim 77, wherein, the means for defining the model of the object comprises means for defining the model using a plurality of vertical planar surfaces.

82. Apparatus according to claim 81, arranged to perform processing such that the vertical planar surfaces are defined such that their bases approximate the outline of the object on the ground.

83. Apparatus according to claim 81, wherein each planar surface is a rectangle.

84. Apparatus according to claim 81, arranged to perform processing such that each planar surface is defined with a height determined in dependence upon the image data identified from the first camera or the image data identified from the second camera.

85. Apparatus according to claim 84, arranged to perform processing such that the height of each planar surface is defined in dependence upon a rectangle bounding some or all of the image data relating to the object identified from the first camera or the second camera.

86. Apparatus according to claim 84, wherein each planar surface is defined to have the same height.

87. Apparatus according to claim 81, further comprising means for generating a top for the model of the object in dependence upon upper edges of the vertical planar surfaces.

88. Apparatus according to claim 77, further comprising the step of generating image data by rendering an image of the modelled object, in which texture data based on the identified image data from at least one camera is rendered onto the model.

89. Apparatus according to claim 88, arranged to perform processing such that each planar surface is mapped onto the image data of the first camera or the second camera, and the image data enclosed by each mapped surface is rendered onto the planar surface in the model.

90. Apparatus according to claim 88, further comprising means for displaying an image of the object using the generated image data.

91. Apparatus for generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, the apparatus being operable to process image data from a first camera to identify image data relating to the object, to use image data from a second camera to determine which parts of the identified image data from the first camera relate to parts of the object on or near the ground, and to represent the object in the computer model in dependence thereon.

92. A method of processing image data defining a sequence of images of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the method comprising:

processing the image data to identify image data relating to respective objects in the scene;

defining a representation of each object in the three-dimensional computer model, in dependence upon the identified image data; and

generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, wherein, when the selected viewing direction is within a predetermined range of viewing directions, texture data based on the identified image data is rendered onto the object representations, and, when the selected viewing direction is not within the predetermined range of viewing directions, a schematic of the positions of the objects in the scene is rendered.

93. A method according to claim 92, wherein the representation of each object comprises a plurality of vertical planar surfaces alone.

94. A method according to claim 92, wherein the representation of each object comprises a single vertical planar surface.

95. A method according to claim 92, wherein the predetermined range of viewing directions is a range relative to a fixed direction in the computer model.

96. A method according to claim 95, wherein the predetermined range of viewing direction is a range relative to the vertical direction in the computer model.

97. A method according to claim 92, wherein the predetermined range of viewing directions is a range relative to the representation of an object.

98. A method according to claim 97, wherein the representation of an object comprises at least one vertical planar surface, and the predetermined range of viewing directions is a range relative to a planar surface.

99. A method according to claim 92, wherein the schematic of the object positions is rendered from a predetermined viewing direction.

100. A method according to claim 99, wherein the schematic is rendered from a vertical downward viewing direction.

101. A method according to claim 92, further comprising the steps of processing the image data to determine at least one colour for each object, and generating image data to indicate the determined colour on the schematic of the object positions.

102. A method according to claim 92, further comprising the step of generating a signal conveying the image data.

103. A method according to claim 102, further comprising the step of recording the signal.

104. A method according to claim 92, further comprising the step of displaying an image of the objects using the generated image data.

105. A method according to claim 92, further comprising the step of making a recording of the image data either directly or indirectly.

106. A method of rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for an object, the texture data being derived from image data recorded by at least one camera, the method comprising:

rendering the texture data onto the representation for the object in accordance with the user-selected viewing direction when the user-selected viewing direction is within a predetermined range of viewing directions; and

rendering a schematic of the positions of the object when the user-selected viewing direction is not within the predetermined range of viewing directions.

107. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 92 and 106.

108. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 92 and 106.

109. Apparatus for processing image data defining a sequence of images of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

means for processing the image data to identify image data relating to respective objects in the scene;

means for defining a representation of each object in the three-dimensional computer model, in dependence upon the identified image data; and

means for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, operable such that, when the selected viewing direction is within a predetermined range of viewing directions, texture data based on the identified image data is rendered onto the object representations, and, when the selected viewing direction is not within the predetermined range of viewing directions, a schematic of the positions of the objects in the scene is rendered.

110. Apparatus according to claim 109, operable to perform processing such that the representation of each object comprises a plurality of vertical planar surfaces alone.

111. Apparatus according to claim 109, operable to perform processing such that the representation of each object comprises a single vertical planar surface.

112. Apparatus according to claim 109, operable to perform processing such that the predetermined range of viewing directions is a range relative to a fixed direction in the computer model.

113. Apparatus according to claim 112, operable to perform processing such that the predetermined range of viewing direction is a range relative to the vertical direction in the computer model.

114. Apparatus according to claim 109, operable to perform processing such that the predetermined range of viewing directions is a range relative to the representation of an object.

115. Apparatus according to claim 114, operable to perform processing such that the representation of an object comprises at least one vertical planar surface, and the predetermined range of viewing directions is a range relative to a planar surface.

116. Apparatus according to claim 109, operable to perform processing such that the schematic of the object positions is rendered from a predetermined viewing direction.

117. A method according to claim 116, operable to perform processing such that the schematic is rendered from a vertical downward viewing direction.

118. Apparatus according to claim 109, further comprising means for processing the image data to determine at least one colour for each object, and means for generating image data to indicate the determined colour on the schematic of the object positions.

119. Apparatus according to claim 109, further comprising means for displaying an image of the objects using the generated image data.

120. Apparatus for rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for an object, the texture data being derived from image data recorded by at least one camera, the apparatus comprising:

means for rendering the texture data onto the representation for the object in accordance with the user-selected viewing direction when the user-selected viewing direction is within a predetermined range of viewing directions; and

means for rendering a schematic of the positions of the object when the user-selected viewing direction is not within the predetermined range of viewing directions.

121. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the method comprising:

processing image data from a first of the cameras to identify image data relating to the object in the scene;

processing image data from a second of the cameras to identify image data relating to the object in the scene;

processing the identified image data from the first camera and the identified image data from the second camera to identify planar surfaces on which points on the object lie by matching feature points in the identified image data from the first camera with feature points in the identified image data from the second camera, and identifying planar surfaces on which matched feature points lie; and

defining a model of the object in the three-dimensional computer model in dependence upon the identified planar surfaces.

122. A method according to claim 121, wherein corner points in the identified image data from the first camera are matched with corner points in the identified image data from the second camera.

123. A method according to claim 121, wherein the planar surfaces are identified by identifying planes on which the matched feature points lie, and determining boundaries of the planes using matched feature points which lie on more than one plane.

124. A method according to claim 123, wherein each plane is identified by identifying a plane on which at least a predetermined number of feature points in the identified image data from the first camera lie, identifying the plane on which the matched feature points in the identified image data from the second camera lie, calculating a transformation between the plane in the image data from the first camera and the plane in the image data from the second camera, and testing the transformation using a plurality of other matched pairs of feature points.

125. A method according to claim 124, wherein the predetermined number is four.

126. A method according to claim 121, wherein the step of defining the model of the object comprises forming a model of planar surfaces in the three-dimensional computer model, each planar surface in the model corresponding to a planar surface identified in the image data from at least one of the cameras.

127. A method according to claim 126, wherein the step of defining the model of the object comprises identifying a planar surface which touches the ground in the image data of a camera, defining a vertical planar surface in the three-dimensional computer model in dependence upon the identified planar surface which touches the ground, and defining a further planar surface in the three-dimensional computer model for each further planar surface in the image data of the camera such that the planar surfaces in the three-dimensional computer model and the image data have the same aspect ratio.

128. A method according to claim 127, wherein a planar surface which touches the ground in the image data of the given camera is identified by:

applying a transformation to the base corner points of planar surfaces in the image data from the first camera which defines a mapping from the ground plane in the image data of the first camera to a surface in a modelling space;

applying a transformation to the base corner points of planar surfaces in the image data from the second camera which defines a mapping from the ground plane in the image data of the second camera to the surface in the modelling space; and

comparing the transformed corner points to determine which ones lie within a predetermined distance of each other.

129. A method according to claim 128, wherein the surface in the modelling space is the ground plane in the three-dimensional computer model.

130. A method according to claim 129, wherein the defined vertical planar surface in the three-dimensional computer model is defined with a base defined by transformed corner points from the given camera which lie within the predetermined distance of the corresponding transformed corner points from the other camera, and with an aspect ratio corresponding to the aspect ratio of the planar surface in the image data of the given camera to which the transformed corner points belong.

131. A method according to claim 121, further comprising the step of generating image data by rendering an image of the modelled object, in which texture data based on the identified image data from at least one camera is rendered onto the model.

132. A method according to claim 131, wherein image data enclosed by each planar surface is rendered on the corresponding planar surface of the object model.

133. A method according to claim 131, further comprising the step of generating a signal conveying the image data.

134. A method according to claim 133, further comprising the step of recording the signal.

135. A method according to claim 131, further comprising the step of displaying an image of the object using the generated image data.

136. A method according to claim 131, further comprising the step of making a recording of the image data either directly or indirectly.

138. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 121 and 137.

139. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 121 and 137.

137. A method of generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, in which image data from a first camera and a second camera is processed to match feature points in the image data from the first camera with feature points in image data from the second camera, the resulting matches are used to determine planar surfaces making up the object, and the object is represented in the computer model in dependence thereon.

140. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

means for processing image data from a first of the cameras to identify image data relating to the object in the scene;

means for processing image data from a second of the cameras to identify image data relating to the object in the scene;

means for processing the identified image data from the first camera and the identified image data from the second camera to identify planar surfaces on which points on the object lie, comprising means for matching feature points in the identified image data from the first camera with feature points in the identified image data from the second camera, and means for identifying planar surfaces on which matched feature points lie; and

means for defining a model of the object in the three-dimensional computer model in dependence upon the identified planar surfaces.

141. Apparatus according to claim 140, operable to perform processing such that corner points in the identified image data from the first camera are matched with corner points in the identified image data from the second camera.

142. Apparatus according to claim 140, operable to perform processing such that the planar surfaces are identified by identifying planes on which the matched feature points lie, and determining boundaries of the planes using matched feature points which lie on more than one plane.

143. Apparatus according to claim 142, operable to perform processing such that each plane is identified by identifying a plane on which at least a predetermined number of feature points in the identified image data from the first camera lie, identifying the plane on which the matched feature points in the identified image data from the second camera lie, calculating a transformation between the plane in the image data from the first camera and the plane in the image data from the second camera, and testing the transformation using a plurality of other matched pairs of feature points.

144. Apparatus according to claim 143, wherein the predetermined number is four.

145. Apparatus according to claim 140, wherein the means for defining the model of the object comprises means for forming a model of planar surfaces in the three-dimensional computer model, each planar surface in the model corresponding to a planar surface identified in the image data from at least one of the cameras.

146. Apparatus according to claim 145, wherein the means for defining the model of the object comprises means for identifying a planar surface which touches the ground in the image data of a camera, means for defining a vertical planar surface in the three-dimensional computer model in dependence upon the identified planar surface which touches the ground, and means for defining a further planar surface in the three-dimensional computer model for each further planar surface in the image data of the camera such that the planar surfaces in the three-dimensional computer model and the image data have the same aspect ratio.

147. Apparatus according to claim 146, operable to perform processing such that a planar surface which touches the ground in the image data of the given camera is identified by:

applying a transformation to the base corner points of planar surfaces in the image data from the first camera which defines a mapping from the ground plane in the image data of the first camera to a surface in a modelling space;

applying a transformation to the base corner points of planar surfaces in the image data from the second camera which defines a mapping from the ground plane in the image data of the second camera to the surface in the modelling space; and

comparing the transformed corner points to determine which ones lie within a predetermined distance of each other.

148. Apparatus according to claim 147, wherein the surface in the modelling space is the ground plane in the three-dimensional computer model.

149. Apparatus according to claim 148, operable to perform processing such that the defined vertical planar surface in the three-dimensional computer model is defined with a base defined by transformed corner points from the given camera which lie within the predetermined distance of the corresponding transformed corner points from the other camera, and with an aspect ratio corresponding to the aspect ratio of the planar surface in the image data of the given camera to which the transformed corner points belong.

150. Apparatus according to claim 140, further comprising means for generating image data by rendering an image of the modelled object, in which texture data based on the identified image data from at least one camera is rendered onto the model.

151. Apparatus according to claim 150, operable to perform processing such that image data enclosed by each planar surface is rendered on the corresponding planar surface of the object model.

152. Apparatus according to claim 150, further comprising means for displaying an image of the object using the generated image data.

153. Apparatus for generating a model of an object in a three-dimensional computer model by processing images of the object from a plurality of cameras, the apparatus being operable to process image data from a first camera and a second camera to match feature points in the image data from the first camera with feature points in the image data from the second camera, to use the resulting matches to determine planar surfaces making up the object, and to represent the object in the computer model in dependence thereon.

154. A method of processing image data defining a sequence of images of at least one object moving in a scene to produce signals defining a representation of each object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the method comprising:

processing the image data to identify image data relating to respective objects in the scene;

defining a representation of each object in the three-dimensional computer model in dependence upon the identified image data;

generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction in which texture data based on the identified image data is rendered onto the object representations; and

generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

155. A method according to claim 154, wherein the information indicating the reliability is generated in dependence upon a linear relationship between reliability and the angular difference between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

156. A method according to claim 154, further comprising the step of generating information indicating how to change the viewing direction to improve the generated reliability.

157. A method according to claim 154, wherein image data for a sequence of images recorded by a first camera and a sequence of images recorded by a second camera are processed such that:

in the step of processing the image data, image data from the first camera relating to the respective objects in the scene is identified, and image data from the second camera relating to the respective objects in the scene is identified;

in the step of defining a representation of each object, a first representation of each object is defined in dependence upon the identified image data from the first camera, and a second representation of each object is defined in dependence upon the identified image data from the second camera; and

in the step of generating image data, texture data based on the identified image data from at least one camera is rendered onto the object representations.

158. A method according to claim 154, wherein, in the step of defining a representation of each object, each object is represented as a planar surface.

159. A method according to claim 154, wherein the reliability information is generated as pixel data within the generated image data.

160. A method according to claim 154, further comprising the step of generating a signal conveying the image data and the reliability information.

161. A method according to claim 160, further comprising the step of recording the signal.

162. A method according to claim 154, further comprising the step of displaying an image using the generated image data and displaying the reliability information.

163. A method according to claim 154, further comprising the step of making a recording of the image data and the reliability information either directly or indirectly.

166. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 154, 164 and 165.

167. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 154, 164, and 165.

164. A method of rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for at least one object, the texture data being derived from image data recorded by at least one camera, the method comprising:

generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, in which the texture data is rendered onto each representation; and

generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

165. An image processing method in which object data defining a three-dimensional computer model of at least one object in a scene is rendered in accordance with a user-selected viewing direction using image data recorded by a camera having a viewing direction to render each object, and an indicator of a quality of the generated image data is produced for output to the user in dependence upon the angle between the user-selected viewing direction and the viewing direction of the camera.

168. Apparatus for processing image data defining a sequence of images of at least one object moving in a scene to produce signals defining a representation of each object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

means for processing the image data to identify image data relating to respective objects in the scene;

means for defining a representation of each object in the three-dimensional computer model in dependence upon the identified image data;

means for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction in which texture data based on the identified image data is rendered onto the object representations; and

means for generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

169. Apparatus according to claim 168, operable to perform processing such that the information indicating the reliability is generated in dependence upon a linear relationship between reliability and the angular difference between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

170. Apparatus according to claim 168, further comprising means for generating information indicating how to change the viewing direction to improve the generated reliability.

171. Apparatus according to claim 168, operable to process image data for a sequence of images recorded by a first camera and a sequence of images recorded by a second camera, wherein:

the means for processing the image data is operable to identify image data from the first camera relating to the respective objects in the scene and to identify image data from the second camera relating to the respective objects in the scene;

the means for defining a representation of each object is operable to define a first representation of each object in dependence upon the identified image data from the first camera, and to define a second representation of each object in dependence upon the identified image data from the second camera; and

the means for generating image data is operable to render texture data based on the identified image data from at least one camera onto the object representations.

172. Apparatus according to claim 168, wherein the means for defining a representation of each object is arranged to represent each object as a planar surface.

173. Apparatus according to claim 168, operable to perform processing such that the reliability information is generated as pixel data within the generated image data.

174. Apparatus according to claim 168, further comprising means for displaying an image using the generated image data and displaying the reliability information.

175. Apparatus for rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for at least one object, the texture data being derived from image data recorded by at least one camera, the apparatus comprising:

means for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, in which the texture data is rendered onto each representation; and

means for generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

176. An image processing apparatus operable to render object data defining a three-dimensional computer model of at least one object in a scene in accordance with a user-selected viewing direction using image data recorded by a camera having a viewing direction to render each object, and operable to produce an indicator of a quality of the generated image data for output to the user in dependence upon the angle between the user-selected viewing direction and the viewing direction of the camera.

177. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the method comprising:

receiving data defining a user-selected viewing direction;

calculating the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

identifying the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

comparing at least one camera characteristic affecting image data quality for each identified camera to determine differences therebetween;

selecting one of the identified cameras as a selected camera in dependence upon the determined differences;

processing input image data from the selected camera to define a representation of the object in the three-dimensional computer model; and

generating image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the representation of the object.

178. A method according to claim 177, further comprising the step of generating a signal conveying the image data.

179. A method according to claim 178, further comprising the step of recording the signal.

180. A method according to claim 177, further comprising the step of displaying an image of the objects using the generated image data.

181. A method according to claim 177, further comprising the step of making a recording of the image data either directly or indirectly.

224. A method according to claim 177 or claim 223, wherein, when comparing at least one camera characteristic affecting image data quality, at least one of the following are compared:

the methods of transferring the image data from respective cameras;

the resolution of respective cameras;

the shutter speed of respective cameras;

the stability of the image data from respective cameras; and

whether the image data from respective cameras is colour or black and white.

225. A method according to claim 177 or claim 223, wherein a plurality of camera characteristics affecting image quality are compared.

226. A method according to claim 225, wherein the camera characteristics affecting quality are considered in a predetermined order and values for each respective camera characteristic are compared, with the selection of a camera being made once the tests identify a characteristic which differs by more than a predetermined amount for given cameras.

227. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 177, 182, 183 and 223.

228. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus become operable to perform a method according to any one of claims 177, 182, 183 and 223.

182. An image processing method in which image data from each of a respective sequence of images, each from a different camera, is processed to define a representation of at least one object in a three-dimensional computer model, and wherein a representation of each object is selected for rendering in dependence upon a user-selected viewing direction, the viewing direction of each camera and a plurality of camera parameters related to image data quality, wherein the plurality of camera parameters are tested in a predetermined order, with the selection of a representation for rendering being made once the tests identify a camera parameter which is sufficiently different for the cameras.

183. An image processing method in which a user-selected viewing direction in accordance with which an image of at least one object in a three-dimensional computer model is to be rendered is used to select, from among image data defining a plurality of images of the object each recorded by a respective camera, image data to be used to define the object in the three-dimensional computer model, the selection being carried out in dependence upon the user-selected viewing direction, together with the viewing direction of each camera and a plurality of camera parameters related to image data quality, wherein the plurality of camera parameters are tested in a predetermined order, with the selection of image data being made once the tests identify a camera parameter which is sufficiently different for the cameras.

184. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

means for receiving data defining a user-selected viewing direction;

means for calculating the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

means for identifying the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

means for comparing at least one camera characteristic affecting image data quality for each identified camera to determine differences therebetween;

means for selecting one of the identified cameras as a selected camera in dependence upon the determined differences;

means for processing input image data from the selected camera to define a representation of the object in the three-dimensional computer model; and

means for generating image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the representation of the object.

185. Apparatus according to claim 184, further comprising means for displaying an image of the objects using the generated image data.

230. Apparatus according to claim 184 or claim 229, wherein the means for comparing at least one camera characteristic affecting image data quality is operable to compare at least one of:

the methods of transferring the image data from respective cameras;

the resolution of respective cameras;

the shutter speed of respective cameras;

the stability of the image data from respective cameras; and

whether the image data from respective cameras is colour or black and white.

231. Apparatus according to claim 184 or claim 229, operable to perform processing to compare a plurality of respective camera characteristics affecting image quality.

232. Apparatus according to claim 231, operable to perform processing such that the camera characteristics affecting quality are compared in a predetermined order, with the selection of a camera being made once the tests identify a characteristic which differs by more than a predetermined amount for given cameras.

186. An image processing apparatus operable to process image data from each of a respective sequence of images, each from a different camera, to define a representation of at least one object in a three-dimensional computer model, and to select a representation of each object for rendering in dependence upon a user-selected viewing direction, the viewing direction of each camera and a plurality of camera parameters related to image data quality, the apparatus being operable to test the plurality of camera parameters in a predetermined order and to select a representation for rendering when the tests identify a camera parameter which is sufficiently different for the cameras.

187. An image processing apparatus operable to use a user-selected viewing direction in accordance with which an image of at least one object in a three-dimensional computer model is to be rendered to select, from among image data defining a plurality of images of the object each recorded by a respective camera, image data to be used to define the object in the three-dimensional computer model, the selection being carried out in dependence upon the user-selected viewing direction, together with the viewing direction of each camera and a plurality of camera parameters related to image data quality, wherein the apparatus is operable to test the plurality of camera parameters in a predetermined order and to select image data when the tests identify a camera parameter which is sufficiently different for the cameras.

188. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data for first and second images in a sequence of images of the object by rendering images of the three-dimensional computer model in accordance with first and second user-selected viewing directions, the method comprising:

processing the image data to define at least one representation of the object in the three-dimensional computer model;

generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first of the cameras onto a representation of the object in accordance with a first user-selected viewing direction;

generating image data for use in a second image in the sequence by rendering texture data based on image data from a second of the cameras onto a representation of the object in accordance with a second user-selected viewing direction;

testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

if the image data for the object in the second image differs by more than the predetermined amount, generating modified image data for the object in the second image.

189. A method according to claim 188, wherein:

in the step of processing the image data, image data from the first camera is processed to generate a first representation of the object in the three-dimensional computer model, and image data from the second camera is processed to generate a second representation of the object in the three-dimensional computer model;

the image data for the object in the first image is generated by rendering the first representation; and

the image data for the object in the second image is generated by rendering the second representation.

190. A method according to claim 189, wherein, in the step of testing:

further image data for the object in the second image in the sequence is generated by rendering texture data based on image data from the first camera onto the first representation of the object in accordance with the second user-selected viewing direction; and

the image data for the object in the second image generated using image data from the second camera is compared with the image data for the object in the second image generated using image data from the first camera.

191. A method according to claim 190, wherein in the step of generating modified image data, the modified image data is generated in dependence upon the image data for the object in the second image generated using image data from the second camera and the image data for the object in the second image generated using image data from the first camera.

192. A method according to claim 188, wherein:

the step of generating image data for the first image comprises rendering the three-dimensional computer model in accordance with the first user-selected viewing direction;

the step of generating image data for the second image comprises rendering the three-dimensional computer model in accordance with the second user-selected viewing direction; and

the step of testing comprises comparing the rendered image data for the second image with the predetermined image data.

193. A method according to claim 188, further comprising the step of generating a signal conveying the modified image data.

194. A method according to claim 193, further comprising the step of recording the signal.

195. A method according to claim 188, further comprising the step of displaying an image of the object using the modified image data.

196. A method according to claim 188, further comprising the step of making a recording of the modified image data either directly or indirectly.

200. A storage medium storing instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 188, 197, 198 and 199.

201. A physically-embodied computer program product including instructions in computer-readable form, including instructions for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 188, 197, 198 and 199.

197. A method of generating image data for first and second images in a sequence of images by rendering a three-dimensional computer model in accordance with respective first and second user-selected viewing directions, the three-dimensional computer model comprising a representation and associated texture data for at least one object and the texture data comprising texture data derived from image data recorded by a first camera and texture data derived from image data recorded by a second camera, the method comprising:

generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first camera onto the representation of each object in accordance with the first user-selected viewing direction;

generating image data for use in the second image in the sequence by rendering texture data based on image data from the second camera onto the representation of each object in accordance with a second user-selected viewing direction;

testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

if the image data for the object in the second image differs by more than the predetermined amount, generating modified image data for the object in the second image.

198. An image processing method in which a three-dimensional computer model including at least one representation of an object is processed a first time to generate image data for a first image in a sequence of images by rendering using image data recorded by a first camera as the basis for texture data for a representation, and a second time to generate image data for a successive image in the sequence by rendering using image data recorded by a second camera as the basis for texture data for a representation, and modified image data is generated for the object in the successive image if image data comparison tests indicate that the object in the images in the sequence will appear discontinuous.

199. A method of generating image data for successive images in a sequence by rendering a representation of an object in a three-dimensional computer model using image data from a plurality of cameras, in which a test on the image data is performed to determine whether the image of the object will appear discontinuous in the successive images, and the image data is processed to reduce the discontinuity.

202. Image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data for first and second images in a sequence of images of the object by rendering images of the three-dimensional computer model in accordance with first and second user-selected viewing directions, the apparatus comprising:

means for processing the image data to define at least one representation of the object in the three-dimensional computer model;

means for generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first of the cameras onto a representation of the object in accordance with a first user-selected viewing direction;

means for generating image data for use in a second image in the sequence by rendering texture data based on image data from a second of the cameras onto a representation of the object in accordance with a second user-selected viewing direction;

means for testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

means for generating modified image data for the object in the second image if the image data for the object in the second image differs by more than the predetermined amount.

203. Apparatus according to claim 202, wherein:

the means for processing the image data is operable to process image data from the first camera to generate a first representation of the object in the three-dimensional computer model, and to process image data from the second camera to generate a second representation of the object in the three-dimensional computer model; and

the apparatus is arranged to perform processing such that;

the image data for the object in the first image is generated by rendering the first representation; and

the image data for the object in the second image is generated by rendering the second representation.

204. Apparatus according to claim 203, wherein the means for testing is operable to perform processing such that:

further image data for the object in the second image in the sequence is generated by rendering texture data based on image data from the first camera onto the first representation of the object in accordance with the second user-selected viewing direction; and

the image data for the object in the second image generated using image data from the second camera is compared with the image data for the object in the second image generated using image data from the first camera.

205. Apparatus according to claim 204, wherein the means for generating modified image data is arranged to perform processing such that the modified image data is generated in dependence upon the image data for the object in the second image generated using image data from the second camera and the image data for the object in the second image generated using image data from the first camera.

206. Apparatus according to claim 202, wherein:

the means for generating image data for the first image comprises means for rendering the three-dimensional computer model in accordance with the first user-selected viewing direction;

the means for generating image data for the second image comprises means for rendering the three-dimensional computer model in accordance with the second user-selected viewing direction; and

the means for testing comprises means for comparing the rendered image data for the second image with the predetermined image data.

207. Apparatus according to claim 202, further comprising means for displaying an image of the object using the modified image data.

208. Image processing apparatus for generating image data for first and second images in a sequence of images by rendering a three-dimensional computer model in accordance with respective first and second user-selected viewing directions, the three-dimensional computer model comprising a representation and associated texture data for at least one object and the texture data comprising texture data derived from image data recorded by a first camera and texture data derived from image data recorded by a second camera, the apparatus comprising:

means for generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first camera onto the representation of each object in accordance with the first user-selected viewing direction;

means for generating image data for use in the second image in the sequence by rendering texture data based on image data from the second camera onto the user-selected viewing direction;

means for testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

means for generating modified image data for the object in the second image if the image data for the object in the second image differs by more than the predetermined amount.

209. An image processing apparatus operable to process a three-dimensional computer model including at least one representation of an object a first time to generate image data for a first image in a sequence of images by rendering using image data recorded by a first camera as the basis for texture data for a representation, and a second time to generate image data for a successive image in the sequence by rendering using image data recorded by a second camera as the basis for texture data for a representation, and operable to generate modified image data for the object in the successive image if image data comparison tests indicate that the object in the images in the sequence will appear discontinuous.

210. Apparatus for method of generating image data for successive images in a sequence by rendering a representation of an object in a three-dimensional computer model using image data from a plurality of cameras, the apparatus being operable to perform a test on the image data to determine whether the image of the object will appear discontinuous in the successive images, and to process the image data to reduce the discontinuity.

211. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the apparatus comprising:

an image data identifier for processing image data from a first of the cameras to identify image data relating to objects in the scene, and for processing image data from a second of the cameras to identify image data relating to objects in the scene;

an object representation definer for processing the identified image data from the first camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the first camera, and for processing the identified image data from the second camera for each object to define an object representation in the three-dimensional computer model having a height dependent upon the image data for the object from the second camera;

a height comparer for comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

an object representation generator for generating object representations in the three-dimensional computer model in dependence upon the height comparisons,

wherein the apparatus is arranged to perform processing such that, when the heights of the corresponding representations are not within a predetermined amount of each other, the taller representation is split into a first portion having a height corresponding to the height of the smaller representation and a second portion comprising the remaining part of the taller representation, and wherein a further representation is defined in the three-dimensional model by re-positioning the second portion in the three-dimensional model.

212. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

an image data identifier for processing image data from a first of the cameras to identify image data relating to the object in the scene, and for processing image data from a second of the cameras to identify image data relating to the object in the scene;

an image data transformer for applying a transformation to the identified image data from the first camera which defines a mapping from the ground plane in the space of the image data of the first camera to a surface in a modelling space, and for applying a transformation to the identified image data from the second camera which defines a mapping from the ground plane in the space of the image data of the second camera to the surface in the modelling space;

an image data comparer for comparing the transformed image data from the first and second cameras on the surface in the modelling space;

a shadow determinator for determining which part of the image data represents shadow in dependence upon the comparison results; and

an object representation generator for generating a representation of at least the object in the three-dimensional model.

213. Apparatus for generating a model of an object in a three-dimensional computer model, comprising:

an image data transformer for applying a transformation to image data from a first camera relating to the object and its shadow which maps the image data to a surface to give first transformed image data, and for applying a transformation to image data from a second camera relating to the object and its shadow which maps the image data to the surface to give second transformed image data;

a transformed image data comparer for comparing the first transformed image data and the second transformed image data to distinguish transformed image data relating to the object from transformed image data relating to its shadow; and

an object modeller for modelling the object in dependence upon the transformed image data relating to the object.

214. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

an image data identifier for processing image data from a first of the cameras to identify image data relating to the object in the scene, and for processing image data from a second of the cameras to identify image data relating to the object in the scene;

a footprint determinator for processing the identified image data from the first camera and the identified image data from the second camera to determine a footprint of the object on the ground; and

an object modeller for defining a model of the object in the three-dimensional computer model in dependence upon the determined footprint.

215. Apparatus for processing image data defining a sequence of images of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

an image data identifier for processing the image data to identify image data relating to respective objects in the scene;

an object modeller for defining a representation of each object in the three-dimensional computer model, in dependence upon the identified image data; and

a renderer for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, operable such that, when the selected viewing direction is within a predetermined range of viewing directions, texture data based on the identified image data is rendered onto the object representations, and, when the selected viewing direction is not within the predetermined range of viewing directions, a schematic of the positions of the objects in the scene is rendered.

216. Apparatus for rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for an object, the texture data being derived from image data recorded by at least one camera, the apparatus comprising:

a first renderer for rendering the texture data onto the representation for the object in accordance with the user-selected viewing direction when the user-selected viewing direction is within a predetermined range of viewing directions; and

a second renderer for rendering a schematic of the positions of the object when the user-selected viewing direction is not within the predetermined range of viewing directions.

217. Apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, the apparatus comprising:

an image data identifier for processing image data from a first of the cameras to identify image data relating to the object in the scene, and for processing image data from a second of the cameras to identify image data relating to the object in the scene;

a surface identifier for processing the identified image data from the first camera and the identified image data from the second camera to identify planar surfaces on which points on the object lie, comprising a feature matcher for matching feature points in the identified image data from the first camera with feature points in the identified image data from the second camera, and a planar surface identifier for identifying planar surfaces on which matched feature points lie; and

an object modeller for defining a model of the object in the three-dimensional computer model in dependence upon the identified planar surfaces.

218. Apparatus for processing image data defining a sequence of images of at least one object moving in a scene to produce signals defining a representation of each object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

an image data identifier for processing the image data to identify image data relating to respective objects in the scene;

an object modeller for defining a representation of each object in the three-dimensional computer model in dependence upon the identified image data;

a renderer for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction in which texture data based on the identified image data is rendered onto the object representations; and

a reliability information generator for generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

219. Apparatus for rendering an image in accordance with a user-selected viewing direction of a three-dimensional computer model comprising a representation and associated texture data for at least one object, the texture data being derived from image data recorded by at least one camera, the apparatus comprising:

a renderer for generating image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, in which the texture data is rendered onto each representation; and

a reliability information generator for generating reliability information indicating the reliability of the image data in dependence upon the angle between the user-selected viewing direction and the viewing direction from which the input image data was recorded.

220. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

a data receiver for receiving data defining a user-selected viewing direction;

an angle calculator operable to calculate the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

a camera identifier operable to identify the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

a camera characteristic comparer operable to compare at least one camera characteristic affecting image data quality for each identified camera to determine differences therebetween;

a camera selector operable to select one of the identified cameras as a selected camera in dependence upon the determined differences;

an object representation generator for processing input image data from the selected camera to define a representation of the object in the three-dimensional computer model; and

a renderer for generating image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the representation of each object.

221. Image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data for first and second images in a sequence of images of the object by rendering images of the three-dimensional computer model in accordance with first and second user-selected viewing directions, the apparatus comprising:

an object representation generator for processing the image data to define at least one representation of the object in the three-dimensional computer model;

a renderer for generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first of the cameras onto a representation of the object in accordance with a first user-selected viewing direction, and for generating image data for use in a second image in the sequence by rendering texture data based on image data from a second of the cameras onto a representation of the object in accordance with a second user-selected viewing direction;

an image data tester for testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

an image data modifier for generating modified image data for the object in the second image if the image data for the object in the second image differs by more than the predetermined amount.

222. Image processing apparatus for generating image data for first and second images in a sequence of images by rendering a three-dimensional computer model in accordance with respective first and second user-selected viewing directions, the three-dimensional computer model comprising a representation and associated texture data for at least one object and the texture data comprising texture data derived from image data recorded by a first camera and texture data derived from image data recorded by a second camera, the apparatus comprising:

a renderer for generating image data for use in a first image in the sequence by rendering texture data based on image data from at least a first camera onto the representation of each object in accordance with the first user-selected viewing direction, and for generating image data for use in the second image in the sequence by rendering texture data based on image data from the second camera onto the representation of each object in accordance with a second user-selected viewing direction;

an image data tester for testing whether first and second images of the object displayed from the generated image data will be discontinuous by testing whether the image data for the object in the second image in the sequence differs by more than a predetermined amount from predetermined image data; and

an image data modifier for generating modified image data for the object in the second image if the image data for the object in the second image differs by more than the predetermined amount.

223. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the method comprising:

processing input image data from each respective camera to define a respective representation of the object in the three-dimensional computer model;

receiving data defining a user-selected viewing direction;

calculating the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

identifying the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

comparing at least one camera characteristic affecting the image data quality for each identified camera to determine differences therebetween;

selecting one of the identified cameras as a selected camera in dependence upon the determined differences;

selecting the representation of the object generated from the selected camera as a selected representation; and

generating image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the selected representation of the object.

229. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

means for processing input image data from each respective camera to define a respective representation of the object in the three-dimensional computer model;

means for receiving data defining a user-selected viewing direction;

means for calculating the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

means for identifying the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

means for comparing at least one camera characteristic affecting the image data quality for each identified camera to determine differences therebetween;

means for selecting one of the identified cameras as a selected camera in dependence upon the determined differences;

means for selecting the representation of the object generated from the selected camera as a selected representation; and

means for generating image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the selected representation of the object.

233. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of an object moving in a scene to produce signals defining a representation of the object in a three-dimensional computer model, and to generate image data by rendering an image of the three-dimensional computer model in accordance with a user-selected viewing direction, the apparatus comprising:

an object representation generator operable to process input image data from each respective camera to define a respective representation of the object in the three-dimensional computer model;

a data receiver for receiving data defining a user-selected viewing direction;

an angle calculator operable to calculate the respective angle between the user-selected viewing direction and the respective viewing direction of each camera;

a camera identifier operable to identify the cameras having a viewing direction within a predetermined angle of the user-selected viewing direction as identified cameras;

a camera characteristic comparer operable to compare at least one camera characteristic affecting the image data quality for each identified camera to determine differences therebetween;

a camera selector operable to select one of the identified cameras as a selected camera in dependence upon the determined differences;

an object representation selector operable to select the representation of the object generated from the selected camera as a selected representation; and

a renderer operable to generate image data by rendering an image of the three-dimensional computer model in accordance with the user-selected viewing direction, in which texture data based on input image data from the selected camera is rendered onto the selected representation of the object.

234. A method of processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the method comprising the steps of:

processing image data from a first of the cameras to identify image data relating to objects in the scene;

processing image data from a second of the cameras to identify image data relating to objects in the scene;

processing the identified image data from the first camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in a modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the first camera, wherein the width of each planar surface representing an object is determined by the width of the associated bounding box in the image data, and the height of each planar surface is calculated using the aspect ratio of the bounding box in the image data;

processing the identified image data from the second camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in the modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the second camera, wherein the width of each planar surface representing an object is determined by the width of the associated bounding box in the image data, and the height of each planar surface is calculated using the aspect ratio of the bounding box in the image data;

comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

generating object representations in the three-dimensional computer model in dependence upon the height comparisons.

235. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the apparatus comprising:

means for processing image data from a first of the cameras to identify image data relating to objects in the scene;

means for processing image data from a second of the cameras to identify image data relating to objects in the scene;

means for processing the identified image data from the first camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in a modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the first camera, wherein the width of each planar surface representing an object is determined by the width of the associated bounding box in the image data, and the height of each planar surface is calculated using the aspect ratio of the bounding box in the image data;

means for processing the identified image data from the second camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in the modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the second camera, wherein the width of each planar surface representing an object is determined by the width of the associated bounding box in the image data, and the height of each planar surface is calculated using the aspect ratio of the bounding box in the image data;

means for comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

means for generating object representations in the three-dimensional computer model in dependence upon the height comparisons.

236. An image processing apparatus for processing image data defining a plurality of sequences of images, each from a respective camera, of a plurality of objects moving in a scene to produce signals defining representations of the objects in a three-dimensional computer model, the apparatus comprising:

an image data identifier for processing image data from a first of the cameras to identify image data relating to objects in the scene, and for processing image data from a second of the cameras to identify image data relating to objects in the scene;

an object representation definer for processing the identified image data from the first camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in a modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the first camera, and for processing the identified image data from the second camera for each object to define an object representation comprising a planar surface with its base on a predetermined surface in the modelling space having a position, width and height dependent upon a polygon bounding the image data for the object from the second camera,

wherein the object representation definer is arranged to perform processing such that the width of each planar surface representing an object is determined by the width of the associated bounding box in the image data, and the height of each planar surface is calculated using the aspect ratio of the bounding box in the image data;

a height comparer for comparing the height of the representation of each object generated in dependence upon image data from the first camera with the height of the representation of the corresponding object generated in dependence upon image data from the second camera; and

an object representation generator for generating object representations in the three-dimensional computer model in dependence upon the height comparisons.