Image processing system

• US 20040135788A1
• Filed: 12/17/2003
• Published: 07/15/2004
• Est. Priority Date: 12/22/2000
• Status: Abandoned Application

• Current Assignee: Anthropics Technology Limited
• Original Assignee: Anthropics Technology Limited
• Inventors: Wiles, Charles Stephen, Williams, Mark Jonathan, Davidson, Colin Bruce, Sleet, Gary Michael
• Application Number:
  US10/451,397
• Publication Number:
  US 20040135788A1
• Time in Patent Office: Days
• US Class Current: 345/530
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects

Abstract

An image processing system is provided which receives image data of a user and generates a set of appearance parameters representative of the appearance of the user in the received images. These appearance parameters may then be transformed either to change the identity of the user in the images or to change the resolution of the image of the user. Synthesised images of the user can then be generated from the transformed parameters. The system can be used as a stand alone image processing system or may form part of an image transmission and reception system. It may also be used for streaming data over limited bandwidth communication channels such as the Internet. A system is also provided for animating a single image from a set of image deviations obtained from a source video sequence. A system for changing the lighting conditions in the single image is also provided.

Claims

1. An image processing apparatus comprising:

means for storing source model data defining a function which relates a set of source parameters to image data defining an appearance of a source object;

means for storing target model data defining a function which relates a set of target parameters to image data defining an appearance of a target object;

means for storing transformation data defining a transformation which relates source parameters to target parameters;

means for receiving image data defining an appearance of the source object;

means for determining a set of source parameters for the source object in the received image using the received image data and the source model data;

means for determining a set of target parameters corresponding to the determined set of source parameters using the transformation data and the determined set of source parameters; and

means for determining image data defining an appearance of the target object using said determined set of target parameters and said target model data.

2. An apparatus according to claim 1, wherein said source model defines a linear function between said set of source parameters and said image data defining the appearance of the source object.

3. An apparatus according to claim 1 or 2, wherein said target model defines a linear function between said set of target parameters and said image data defining the appearance of the target object.

4. An apparatus according to any preceding claim, wherein said transformation data defines a linear function between said set of source parameters and said set of target parameters.

5. An apparatus according to claim 4, wherein said linear transformation is such that one or more of said target parameters corresponds to a linear weighted combination of said source parameters.

6. An apparatus according to claim 4 or 5, wherein said transformation data defines the following transformation:

p^trgt=R·

p^srce+r 

 

(19)

where p^srce is the set of source parameters;

p^trgt is the set of target parameters;

R is a transformation matrix; and

r is a vector of offsets.

7. An apparatus according to claim 6, wherein R is determined from said source model data and said target model data.

8. An apparatus according to claim 7, wherein R is determined automatically.

9. An apparatus according to claim 6, wherein R and r are determined experimentally in advance by providing a plurality of corresponding sets of source parameters and target parameters.

10. An apparatus according to claim 9, wherein said corresponding sets of source parameters and target parameters include the following sets of source parameters:

11. An apparatus according to any of claims 1 to 3, wherein said transformation data defines a non-linear transformation between said source parameters and said target parameters.

12. An apparatus according to claim 11, wherein said transformation data defines a neural network which relates said source parameters to said target parameters.

13. An apparatus according to any preceding claim, wherein said source model data and said target model data define the shape of the source object and the target object respectively.

14. An apparatus according to any preceding claim, wherein said source model data and said target model data define the shape and texture of the source object and the target object respectively.

15. An apparatus according to any preceding claim, wherein said source model data and said target model data define the shape and colour of the source object and the target object respectively.

16. An apparatus according to any preceding claim, wherein said source model data and said target model data define the appearance of the source object and the target object at different resolutions.

17. An apparatus according to claim 16, wherein said target model data defines the appearance of the target object to a greater resolution than the source model data defines the appearance of the source object.

18. An apparatus according to any preceding claim, wherein said source object and said target object are the same object.

19. An apparatus according to any of claims 1 to 17, wherein said source object comprises a first human face and said target object comprises a second human face.

20. An apparatus according to any preceding claim, wherein said source object and said target object are deformable objects.

21. An apparatus according to any preceding claim, wherein at least one of said source object and said target object comprises a human face.

22. An apparatus according to any preceding claim, wherein said receiving means is operable to receive a sequence of images of the source object and wherein said apparatus is operable to determine corresponding images of said target object using said received sequence of source images and said stored data.

23. An apparatus according to any preceding claim, wherein said source model data is determined in advance from a plurality of training images of the source object.

24. An apparatus according to claim 23, wherein said source model data is determined using a principal component analysis on the training images to identify the main modes of variation of the source object in the training images.

25. An apparatus according to claim 23 or 24, wherein said target model data is determined in advance from a plurality of training images of the target object.

26. An apparatus according to claim 25, wherein said target model data is determined using a principal component analysis on the training images to identify the main modes of variation of the target object in the training images.

27. An apparatus according to claim 25 or 26, wherein said source object and said target object are for the same person, wherein said source model data is determined from general training images of the person and wherein said target model data is determined from selected training images of the person.

99. An apparatus according to any preceding claim, wherein said predetermined sets of parameters include:

113. Apparatus according to any one of claims 104 to 112 for use in image processing apparatus in accordance with any one of claims 1 to 32 for enabling input of data to facilitate generation of source or target model data.

123. Apparatus according to any one of claims 15 to 122 for use with image processing apparatus in accordance with any one of claims 1 to 32, wherein the image sequence comprises images of the source object and the reconstructed images are images of the target object.

143. A computer readable medium storing computer executable process steps for configuring a general purpose processor to become configured as an apparatus or system according to any of claims 1 to 99 or 104 to 138.

144. Computer executable process steps for configuring a general purpose processor to become configured as an apparatus of system according to any of claims 1 to 99 or 104 to 138.

28. An image processing apparatus comprising:

means for storing source model data defining a function which relates a set of source parameters to a source set of locations which identify the relative positions of a plurality of predetermined points on a source object;

means for storing target model data defining a function which relates a set of target parameters to a target set of locations which identify the relative positions of a plurality of predetermined points on a target object;

means for storing transformation data defining a transformation which relates source parameters to target parameters;

means for receiving image data for an image of the source object;

means for determining said source set of locations for the source object in the received image;

means for determining a set of source parameters for the source object in the received image using the determined source set of locations and the source model data;

means for determining a set of target parameters corresponding to the determined set of source parameters using the transformation data and the determined set of source parameters;

means for determining a target set of locations using said determined set of target parameters and said target model data; and

means for determining image data of said target object using said determined target set of locations.

29. An apparatus according to claim 28, wherein said source model data models the two dimensional shape of the source object by identifying the relative positions of said predetermined points in a predetermined plane.

30. An apparatus according to claim 28 or 29, wherein said target model data models the two dimensional shape of the target object by identifying the relative positions of said predetermined points in a predetermined plane.

31. An apparatus according to any of claims 28 to 30, wherein said source model data models the three dimensional shape of the source object by identifying the relative positions of said predetermined points in a three dimensional space.

32. An apparatus according to any of claims 28 to 31, wherein said target model data models the three dimensional shape of the target object by identifying the relative positions of said predetermined points in a three dimensional space.

33. A camera comprising:

means for storing first model data defining a function which relates a set of first parameters to image data of said object at a first resolution;

means for storing second model data defining a function which relates a set of second parameters to image data of said object at a second resolution;

means for storing transformation data defining a transformation which relates said first parameters to said second parameters;

means for sensing light from the object and for generating image data at said first resolution therefor;

means for determining a set of first parameters for the object using the sensed image data and the first model data;

means for determining a set of second parameters corresponding to the determined set of first parameters using the transformation data and the determined set of first parameters; and

means for determining image data of the object at said second resolution using said determined set of second parameters and said second model data.

34. A camera according to claim 33, wherein said first resolution is lower than said second resolution.

35. A camera according to claim 33 or 34, wherein said first model data is determined in advance from a plurality of training images of the object at said first resolution.

36. A camera according to any of claims 33 to 35, wherein said second model data is determined in advance from a plurality of training images of the object at said second resolution.

37. An apparatus for generating an appearance model for an object, the appearance model defining a function which relates a set of parameters to pixel values which define an appearance of the object, the apparatus comprising:

means for receiving plural training pixel images of the object having different appearances;

means for sampling pixel values at predetermined points over each of the training images to generate a respective plurality of sets of corresponding pixel values for the training images; and

means for processing the sets of corresponding pixel values to determine said appearance model;

wherein said object includes one or more features of interest and wherein said sampling means is operable to take more samples of pixel values over said one or more features of interest than over other parts of the object.

38. An apparatus according to claim 37, wherein said object is a face and wherein said one or more features of interest include the eyes and/or mouth of the face.

39. An apparatus according to claim 37 or 38, further comprising means for warping the training images in order to increase the size of said one or more features relative to the rest of the object and wherein said sampling means is operable to sample the warped images of the object at a substantially constant sampling density.

40. An apparatus for determining a set of parameter values representative of the appearance of an object within an input image, the apparatus comprising:

means for storing an appearance model generated using an apparatus according to any of claims 37 to 39;

means for receiving the input image of the object;

means for sampling pixel values at predetermined points over the received image to generate a set of pixel values representative of the appearance of the object in the received image; and

means for generating a set of parameters for the object in the received image using said appearance model and said sampled pixel values;

wherein said sampling means is operable to take more samples of pixel values over said one or more features of interest in the received image than over other parts of the object.

41. An apparatus for generating an appearance model for an object, the appearance model defining a function which relates a set of parameters to pixel values which define an appearance of the object, the apparatus comprising:

means for receiving plural training pixel images of the object having different appearances;

means for identifying the location within each training image of a plurality of predetermined points on the object, the predetermined points identifying the outline of the object and the locations of one or more features of interest on the object;

means for warping each training image so that the determined locations of said predetermined points are warped to the locations of the corresponding points in a reference image of the object;

means for sampling pixel values of each of the warped training images to generate a respective plurality of sets of corresponding pixel values for the training images; and

means for processing the sets of corresponding pixel values to determine said appearance model;

wherein said one or more features of interest on the object are enlarged in said reference image of the object relative to other parts of the object.

42. An apparatus according to claim 41, wherein said sampling means is operable to sample said warped image at a substantially constant pixel density.

43. An apparatus for determining a set of parameter values representative of the appearance of an object within an input image, the apparatus comprising:

means for storing an appearance model generated using an apparatus according to claim 41 or 42;

means for receiving the input image of the object;

means for identifying the location within the input image of said plurality of predetermined points on the object;

means for warping the input image so that the determined locations of said predetermined points are warped to the locations of the corresponding points in the reference image of the object;

means for sampling pixel values of the warped input image to generate a set of pixel values for the input image; and

means for determining said set of parameter values for the input image using said sampled pixel values for the input image and said appearance model.

44. An apparatus according to claim 43, wherein said sampling means is operable to sample said warped input image at a substantially constant pixel density.

45. A video communication system comprising:

a transmitter and a receiver;

wherein the transmitter comprises;

means for storing source model data defining a function which relates a set of source parameters to image data defining an appearance of a source object;

means for storing target model data defining a function which relates a set of target parameters to image data defining an appearance of a target object;

means for storing transformation data defining a transformation which relates source parameters to target parameters;

means for receiving current image data defining a current appearance of the source object;

means for determining a set of source parameters for the source object in the current image data using the current image data and the source model data;

means for determining a set of target parameters corresponding to the determined set of source parameters using the transformation data and the determined set of source parameters; and

means for transmitting the target model data and the determined target parameters; and

wherein the receiver comprises;

means for receiving the transmitted target model data and said target parameters; and

means for determining image data defining an appearance of the target object using the determined set of target parameters and said target model data.

46. A system according to claim 45, wherein said transmitter further comprises means for encoding said target model data for transmission to said receiver by said transmitting means.

47. A system according to claim 46, wherein said encoding means is operable to encode said target model data by applying predetermined sets of target parameters to said target model data to derive corresponding image data for each of the predetermined sets of target parameters and means for compressing said determined image data generated from the predetermined sets of target parameters.

48. A system according to claim 47, wherein said receiver comprises means for decompressing said compressed image data and means for resynthesising said target model data using said decompressed images and the predetermined sets of target parameters.

49. A communication system according to claim 48, wherein said predetermined sets of target parameters include the following:

50. A communication system according to any of claims 45 to 49, wherein said receiving means of said transmitter is operable to receive a sequence of images of the source object, wherein said transmitter is operable to determine corresponding target parameters for one or more of the received images of the source object and wherein said transmission means is operable to transmit sets of target parameter values to said receiver for said one or more of said images in the sequence.

51. A communication system according to claim 50, wherein said transmitter is operable to determine corresponding target parameters for each of the received images of the source object and wherein said transmission means is operable to transmit sets of target parameter values to said receiver for each image in the sequence.

52. A system according to claim 50 or 51, wherein said transmitter further comprises means for receiving audio signals corresponding to said sequence of images and wherein said transmitter is operable to transmit audio data to said receiver together with said target parameters for the sequence of images.

53. A system according to claim 52, wherein said transmitting means is operable to transmit said sets of target parameters and said audio signals to said receiver in a time interleaved manner.

54. A system according to claim 53, wherein said transmitting means is operable to transmit said target parameters and said audio signals in packets of information.

55. A communication system according to any of claims 45 to 54, wherein said transmitter is operable to transmit said data to said receiver via the Internet.

56. A system according to any of claims 45 to 55, wherein said source model data and said target model data define the appearance of the source object and the target object at difference resolutions.

57. A communication system according to claim 56, wherein said target model data define the appearance of the target object to a greater resolution than the source model data defines the appearance of the source object.

77. A transmitter circuit comprising any of the technical transmitter circuit features of any of claims 45 to 76.

78. A receiver circuit comprising any of the technical receiver circuit features according to any of claims 45 to 46.

58. An image communication system comprising:

a transmitter and a receiver;

wherein the transmitter comprises;

means for storing transmitter model data defining a function which relates a set of parameters to image data defining an appearance of an object;

means for receiving image data defining an appearance of the object;

means for determining a set of parameters for the object in the received image using the received image data and the model data; and

means for transmitting the determined set of parameter values to said receiver;

wherein the receiver comprises;

means for storing receiver model data defining a function which relates a set of parameters to image data defining an appearance of the object;

means for receiving transmitted sets of parameters; and

means for generating image data defining an appearance of the object using said received set of parameters and said model data.

59. A communication system according to claim 58, further comprising means for transforming said set of parameters in accordance with a predetermined transformation.

60. A system according to claim 59, wherein said predetermined transformation is operable to alter the identity of the object.

61. A system according to claim 59 or 60, wherein said transmitter model data relates said set of parameters to image data having a first resolution and wherein said receiver model data is operable to define a function which relates said parameters to image data having a second different resolution and wherein said transformation means is operable to transform the parameters for the transmitter model data into parameters for use with the receiver model data.

62. A system according to claim 59, 60 or 61, wherein said transformation means is located within said transmitter.

63. A system according to claim 59, 60 or 61, wherein transformation means is located within said receiver.

64. A system according to claim 59, 60 or 61, wherein said transmitter is operable to transmit said parameters to said receiver via at least one intermediate processing node and wherein said transformation means is provided within said intermediate processing node.

65. A communication system according to any of claims 58 to 64, further comprising means for storing said receiver model data and means for transmitting said stored receiver model data to said receiver.

66. A system according to claim 65, wherein said means for storing said receiver model data and said means for transmitting said receiver model data to said receiver are located within said transmitter.

67. A system according to claim 65, wherein said transmitter is operable to transmit said set of parameters to said receiver via a computer network and wherein said means for storing said receiver model data and said means for transmitting said receiver model data to said receiver are located within said computer network.

68. A system according to any of claims 65 to 67, wherein said means for transmitting said receiver model data includes means for encoding said receiver model data.

69. A system according to claim 68, wherein said encoding means is operable to encode said receiver model data by applying predetermined sets of parameters to said receiver model data to derive corresponding image data for each of the predetermined sets of parameters and means for compressing the determined image data generated from the predetermined sets of parameters.

70. A system according to claim 69, wherein said receiver comprises means for decompressing said compressed image data and means for resynthesising said receiver model data using said decompressed images and the predetermined sets of parameters.

71. A system according to claim 69, wherein said image data determined for each of the predetermined sets of parameters is composited within a rectangular frame and wherein a template image is used to identify pixels within the composited image which relate to background pixels and those which relate to the image data generated by said target parameters;

wherein said transmitting means is operable to transmit said template image to said receiver and wherein said receiver is operable to resynthesise said receiver model data using said template image.

72. A system according to claim 71, wherein said template image comprises a two dimensional array of pixel values corresponding to a binary one or a binary zero, with a binary one representing background pixels and a binary zero representing image data generated by the target parameters.

73. A communication system according to any of claims 58 to 72, wherein said receiving means of said transmitter is operable to receive a sequence of images of the object, wherein said transmitter is operable to determine corresponding parameters for each of said received images of the object and wherein said transmitter means is operable to transmit sets of parameter values to said receiver for each image in the sequence.

74. A system according to claim 73, wherein said transmitter further comprises means for receiving audio signals corresponding to said sequence of images and wherein said transmitter is operable to transmit audio data to said receiver together with said parameters for the sequence of images.

75. A system according to claim 74, wherein said transmitting means is operable to transmit said sets of parameters and said audio signals to said receiver in a time interleaved manner.

76. A system according to claim 75, wherein said transmitting means is operable to transmit said parameters and said audio signals in packets of information.

79. A data stream for driving an appearance model which relates a set of parameters to image data, to generate a synthesised image sequence, the data stream comprising:

a sequence of data packets including information packets which include data concerning the animated video sequence to be generated; and

image data packets including parameter data for driving said appearance model to generate said synthesised image sequence.

80. A data stream according to claim 79, wherein plural information packets are provided in said data stream.

82. A data stream according to any of claim 79 to 81, wherein each of said image packets includes time stamp information.

83. A data stream according to any of claims 79 to 82, wherein said stream of data packets include audio packets which comprise audio data for the animated image sequence.

84. A data stream according to claim 83, wherein said audio packets are interleaved with said image packets in said data stream.

85. A data stream according to any of claims 79 to 84, wherein said information packet is repeated at regular intervals within said packet stream.

86. A data stream according to any of claims 79 to 85, wherein said packet stream furthers comprises one or more copyright packets including copyright information.

87. A player unit for generating a synthesised image sequence, comprising:

means for storing an appearance model which relates a set of parameters to image data;

means for receiving a data stream according to any of claims 79 to 86; and

means for generating the synthesised image sequence using the stored appearance model and the image data packets in accordance with the date included within said information packets.

88. A player unit according to claim 87 when dependent upon claim 86, further comprising means for sensing the existence of said one or more copyright packets within said packet stream and means for inhibiting the operation of said image sequence generating means if said sensing means senses that the packet stream does not include said copyright packet.

81. A data stream according to 79 or 80, wherein one or more of said image packets include parameter data for plural images of the synthesised image sequence.

89. A video camera comprising:

means for storing model data defining a function which relates a set of parameters to image data defining an appearance of an object;

a wide angled lens and a light sensor for imaging a scene including the object and generating a sequence of images of the scene including the object;

means for processing said sequence of images to extract the image data corresponding to the object within each image of the sequence;

means for scaling the extracted image data to generate scaled image data for each image in the sequence;

means for generating sets of parameters for each scaled image using said model data; and

means for outputting said sets of parameters.

90. A camera according to claim 89, further comprising means for generating a synthesised video sequence using said output sets of parameters and said model data.

91. A camera according to claim 89 or 90, further comprising means for transforming the sets of parameters for the object into sets of target parameters and target model data defining a function which relates sets of target parameters to image data defining an appearance of a target object.

92. A camera according to claim 91, wherein said transformation means is operable to alter the identity of the object.

93. A camera according to claim 91 or 92, wherein said target model data is operable to relate said parameters to image data having a higher resolution than the other model data stored in the camera.

94. An apparatus for encoding an appearance model which relates a set of parameters to image data representative the appearance of an object, for transmission over a communication link, the apparatus comprising:

means for applying predetermined sets of parameters to said appearance model to derive corresponding image data for each of the predetermined sets of parameters and means for compressing said determined image data generated from the predetermined sets of target parameters.

95. An apparatus according to claim 94, wherein said derived image data corresponding to the predetermined sets of parameters are composited within a rectangular template image which identifies pixels corresponding to the image data and pixels corresponding to background;

and further comprising means for compressing said template image.

96. An apparatus according to claim 95, wherein a common rectangular template image is used for the image data generated by each of the sets of parameters.

97. An apparatus according to claim 95 or 96, wherein said template image is compressed using a first image compression technique and wherein said image data is compressed using a second image compression technique.

98. An apparatus for decoding an appearance model encoded using an apparatus according to any of claims 94 to 97 comprising:

means for receiving the compressed image data;

means for decompressing the compressed image data; and

means for reconstructing the appearance model using the decompressed image data and the predetermined sets of parameters.

100. An image processing method comprising the steps of:

receiving image data defining an appearance of a source object;

determining a set of source parameters for the, source object in the received image using the received image data and stored source model data which defines a function which relates a set of source parameters to image data defining an appearance of the source object;

determining a set of target parameters corresponding to the determined set of source parameters using the transformed data and the determined set of source parameters; and

determining image data defining an appearance of the target object using the determined set of target parameters and stored target model data which defines a function that relates a set of target parameters to image data defining appearance of the target object.

101. An image processing method comprising the steps of:

receiving image data for an image of a source object;

determining a source set of locations which identify the relative positions of a plurality of predetermined points on the source subject;

determining a set of source parameters for the source object in the received image using the determined source set of locations and stored source model data which defines a function which relates a set of source parameters to a source set of locations which identify the relative positions of a plurality of predetermined points on the source object;

determining a set of target parameters corresponding to the determined set or source parameters using stored transformation data and a determined set of source parameters;

determining a target set of locations using said determined set of target parameters and stored target model data which defines a function which relates a set of target parameters to a target set of locations which identify the relative positions of a plurality of predetermined points on the target object; and

determining image data of the target object using the determined target set of locations.

102. A method of generating an appearance model for an object, the appearance model defining a function which relates a set of parameters to pixel values which define an appearance of the object, the method comprising the steps of:

receiving plural training images of the object having different appearances;

sampling pixel values at predetermined points over each of the training images to generate a respective plurality of sets of corresponding pixel values for the training images; and

processing the sets of corresponding pixel values to determine said appearance model;

wherein the object includes one or more features of interest and wherein the sampling step takes more samples of pixel values over said one or more features of interest than over other parts of the object.

103. A image communication method comprising:

at a transmitter;

receiving current image data defining a current appearance of a source object;

determining a set of parameters for the object in the received image using the received image data and stored model data that defines a function which relates a set of parameters to image data defining an appearance of the object; and

transmitting the determined set of parameter values to a receiver; and

at the receiver;

receiving transmitted sets of parameters; and

generating image date defining an appearance of the object using the received set of parameters and stored receiver model data that defines a function which relates a set of parameters to image data defining an appearance of the object.

104. Apparatus for providing a user interface for image processing apparatus, the apparatus comprising:

means for causing a display to display on a display screen a first image on which specific positions are marked by landmark points;

means for causing the display to display on the display screen with the first image a second image on which landmark points are provided so as to be positionable by a user;

means for determining when a landmark point on the second image is selected by a user; and

means for visually identifying to the user the landmark point in the first image corresponding to the selected landmark point to assist the user in positioning the selected landmark point.

105. Apparatus according to claim 104, further comprising means for visually identifying the selected landmark point.

106. Apparatus according to claim 104 or 105, wherein the identifying means are arranged to identify a landmark point by highlighting or changing the colour of the landmark point relative to the other landmark points.

107. Apparatus according to any one of claims 104 to 106, further comprising means for moving a selected landmark point in response to a user dragging the selected landmark point and dropping it at a new position.

108. Apparatus according to any one of claims 104 to 106, further comprising means for fixing a landmark point in position.

109. Apparatus according to claim 108, further comprising means for unfixing the position of at least one landmark point.

110. Apparatus according to any one of claims 104 to 109, further comprising means for causing the display to display a description of a landmark point selected by the user.

111. Apparatus according to claim 110, further comprising means for enabling a user to alter the description of a landmark point.

112. Apparatus according to any one of claims 104 to 111, wherein the determining means is arranged to determine that a landmark point has been selected by a user when the user places a cursor over the landmark point.

114. Apparatus according to any one of claims 104 to 113 for use in image processing apparatus in accordance with any one of claims 37 to 44, wherein the first image is a warped image generated by using apparatus in accordance with any one of claims 37 to 44.

124. Apparatus for providing a user interface for image processing apparatus, the apparatus being operable in a first mode to provide apparatus in accordance with any one of claims 104 to 114 and in a second mode to provide apparatus in accordance with any one of claims 115 to 123.

115. Apparatus for providing a user interface for image processing apparatus, the apparatus comprising:

means for causing a display to display on a display screen different images of a first image sequence with each image having specific positions therein marked by landmark points;

means for causing the display to provide on the display screen an error display area for displaying to the user a visual representation of an error value relating to a difference between that image and a reconstructed version of image so as to show to the user how the error varies for different images; and

means for causing a visual representation of an error for an image to be added to the error display area when that image is displayed.

116. Apparatus according to claim 115, wherein the visual representation causing means are arranged to cause the visual representation to be displayed as a graph of error against frame numbers of the images in the image sequence.

117. Apparatus according to claim 115 or 116, further comprising means for causing the display to display a frame number for a displayed image in a frame window.

118. Apparatus according to any one of claims 115 to 117, comprising means for causing the display to display a numerical representation of the error value for a displayed image in an error display window.

119. Apparatus according to any one of claims 115 to 118, comprising means for causing the display to display the one of the images of the image sequence associated with a particular error value when the visual representation of that error value is selected by the user.

120. Apparatus according to any one of claims 115 to 119, comprising means for enabling a user to cause the images of the image sequence to be displayed in at least one of a forward or a reverse order.

121. Apparatus according to any one of claims 115 to 120, comprising means for causing the display to display with the image of the image sequence the corresponding reconstructed image.

122. Apparatus according to any one of claims 115 to 121, comprising means for causing the display to display the image of the image sequence for which the error value is greatest.

125. A method of operating a processor to provide a user interface for image processing apparatus, the method comprising causing the processor to:

cause the display to display on a display screen a first image on which specific positions are marked by landmark points;

cause the display to display on the display screen with the first image a second image on which landmark points are provided to as to be positionable by the user;

determine when a landmark point on the second image has been selected by a user; and

visually identify to the user the landmark point in the first image corresponding to the selected landmark point to assist the user in positioning the selected landmark point.

126. A method of operating a processor, which method comprises causing the processor to:

cause a display to display in succession on a display screen different images of an image sequence with each image having specific positions therein marked by landmark points;

cause the display to provide on the display screen an error display area for displaying to the user a visual representation of an error value relating to a difference between that image and a reconstructed version of that image as to show to the user how the error varies for difference images; and

cause a visual representation of the error value for the image to be added to the error display area when that image is displayed.

127. An apparatus for generating an animated sequence of images of an object from an image of the object, the apparatus comprising:

means for storing model data defining a relationship between a set of parameter values and a set of shape and texture deviations from a set of shape and texture values;

means for receiving the image of the object to be animated;

means for processing the image of the object to obtain shape data defining the shape of the object in the image and texture data defining the texture of the object in the image;

means for receiving a plurality of sets of parameter values representative of an image sequence;

means for determining sets of shape and texture deviations from the received plurality of sets of parameter values and said stored model data;

means for applying each set of determined shape and texture deviations to said obtained shape data and texture data to generate respective modified shape data and texture data for each set; and

means for generating a modified image of the object for each set of deviations from the corresponding modified shape data and texture data to generate said animated sequence of images.

128. An apparatus according to claim 127, wherein said processing means is operable to obtain initial shape data and initial texture data defining the texture of the object in the image and is operable to modify said initial shape and texture data so that the appearance of said object in said image corresponds to a predetermined appearance.

129. An apparatus according to claim 127 or 128, wherein said processing means is operable apply a scale to said shape data and said texture data.

130. An apparatus according to claim 128, wherein said processing means is operable to modify said reference shape and texture so that the expression of the object in the image is similar to the expression of another similar object.

131. The apparatus of claim 130, wherein said model data defines a relationship between said shape and texture deviations from shape and texture values derived from said other object.

132. An apparatus according to any of claims 127 to 131, further comprising means for varying lighting conditions of said object in said image.

133. An apparatus according to any of claims 127 to 132, wherein said deviations relate to a difference in texture from said texture values.

134. An apparatus according to any of claims 127 to 132, wherein said deviations relate to a ratio of the resulting texture values to said set of texture values.

135. An apparatus according to any of claims 127 to 134, wherein said texture data corresponds to the luminance of the object in the image.

136. An apparatus for determining the lighting conditions of an object in an image, comprising:

means for receiving data defining a lighting model for an object under a plurality of different lighting conditions, the lighting model relating lighting conditions to a corresponding image of the object under those lighting conditions;

means for determining an inverse of said lighting model; and

means for applying an image of an object with unknown lighting conditions to said inverse lighting model to determine said unknown lighting conditions.

138. An apparatus for changing the lighting conditions of a first object in a first image to correspond to the lighting conditions of a second object in a second image, the apparatus comprising the steps of:

means for receiving data defining a lighting model for a third object under a plurality of different lighting conditions, the lighting model relating lighting conditions to a corresponding image of the third object under those lighting conditions;

means for determining an inverse of said lighting model;

means for applying said first image of said first object to said inverse lighting model to determine the lighting conditions of said first object in said first image;

means for applying said second image of said second object to said inverse lighting model to determine the lighting conditions of said second object in said second image;

means for applying the lighting conditions determined for said first object to said lighting model to generate an image of said third object under the lighting conditions of said first object;

means for determining a ratio image of said first image of said first object and said image of said third object under the lighting conditions of said first object;

means for applying the lighting conditions of said second image to said lighting model to generate an image of said third object under the lighting conditions of said second object; and

means for generating an image of said first object under the lighting conditions of said second object from said ratio image and said image of the third object under the lighting conditions of said second object.

139. A method of determining the lighting conditions of an object in an image, the method comprising the steps of:

receiving data defining a lighting model for an object under a plurality of different lighting conditions, the lighting model relating lighting conditions to a corresponding image of the object under those lighting conditions;

determining an inverse of said lighting model; and

applying an image of an object with unknown lighting conditions to said inverse lighting model to determine said unknown lighting conditions.

137. An apparatus according to claim 139, wherein the object in the image of unknown lighting conditions is different to the object associated with said lighting model.

140. A method of changing the lighting conditions of a first object in a first image to correspond to the lighting conditions of a second object in a second image, the method comprising the steps of:

receiving data defining a lighting model for a third object under a plurality of different lighting conditions, the lighting model relating lighting conditions to a corresponding image of the third object under those lighting conditions;

determining an inverse of said lighting model;

applying said first image of said first object to said inverse lighting model to determine the lighting conditions of said first object in said first image;

applying said second image of said second object to said inverse lighting model to determine the lighting conditions of said second object in said second image;

applying the lighting conditions determined for said first object to said lighting model to generate an image of said third object under the lighting conditions of said first object;

determining a ratio image of said first image of said first object and said image of said third object under the lighting conditions of said first object;

applying the lighting conditions of said second image to said lighting model to generate an image of said third object under the lighting conditions of said second object; and

generating an image of said first object under the lighting conditions of said second object from said ratio image and said image of the third object under the lighting conditions of said second object.

141. A method of animating a sequence of images of an object from an image of the object, the method comprising the steps of:

storing model data defining a relationship between a set of parameter values and a set of shape and texture deviations from a set of shape and texture values;

receiving the image of the object to be animated;

processing the image of the object to obtain shape data defining the shape of the object in the image and texture data defining the texture of the object in the image;

receiving a plurality of sets of parameter values representative of an image sequence;

determining sets of shape and texture deviations from the received plurality of sets of parameter values and said stored model data;

applying each set of determined shape and texture deviations to said obtained shaped data and texture data to generate respective modified shape data and texture data for each set; and

generating a modified image of the object for each set of deviations from the corresponding modified shape data and texture data to generate said animated sequence of images.

142. A method of adapting a model used for modelling the appearance of a first object to generate an adapted model for modelling the appearance of a second different object, the method comprising the steps of:

receiving an image of the second object;

means for processing the image of the second object to obtain shape data defining the shape of the object in the image and texture data defining the texture of the object in the image; and

modifying said model using said shape and texture data.