Image processing apparatus

• US 6,421,459 B1
• Filed: 09/15/1998
• Issued: 07/16/2002
• Est. Priority Date: 09/16/1997
• Status: Expired due to Term

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Rowe, Simon Michael
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/153,041
• Time in Patent Office: 1,400 Days
• Field of Search: 382/154, 382/103, 382/104, 382/105, 382/106, 382/107, 382/108, 382/122, 345/125, 345/419, 345/420, 345/421, 345/422, 345/423, 345/424, 345/425, 345/441, 348/113, 348/116, 348/169, 348/170, 348/171, 348/172
• US Class Current: 382/154
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

In an image processing system, a three-dimensional computer model is created of a real-world static background scene. Video images of objects moving on the background are recorded and processed to identify the portions of the image corresponding to the moving objects and to calculate the corresponding positions in the three-dimensional computer model. Each moving object is represented by at least one vertical plane in the three-dimensional computer model. Images, comprising the simulated background and the moving object planes onto which are rendered the video data of the moving object recorded by the video camera, are displayed to the user from any desired viewing direction(s).

Claims

1. A method of processing image data defining a sequence of an object moving in a scene to produce signals defining a representation of the object at a plurality of positions in a three-dimensional computer model corresponding to positions of the object in the scene, the method comprising:

processing the image data for a plurality of the images to define for each image a respective bounding polygon that bounds the object in the respective image;

defining at least one planar surface within the three-dimensional computer model to represent each respective position of the object, each planar surface being defined with its base on a predetermined surface in the three-dimensional computer model and with a position and size in dependence upon a respective one of said bounding polygons; and

generating texture data for the planar surfaces to represent features of the object in dependence upon the image data.

2. A method according to claim 1, wherein a bounding polygon in the image data has a width and an aspect ratio and a planar surface in the three-dimensional computer model has a width and a height, and wherein the width of a planar surface is determined by the width of the bounding polygon in the image data, and the height of the planar surface is calculated using the aspect ratio of the bounding polygon in the image data.

3. A method according to claim 1, wherein each planar surface lies within a vertical plane.

4. A method according to claim 1, wherein the bounding polygon bounding the object is a rectangle.

5. A method according to claim 4, wherein the rectangle has sides and the images has sides, and wherein the sides of the rectangle are parallel to the sides of the image.

6. A method according to claim 1, wherein a single planar surface is defined to represent each position of the object in the three-dimensional computer model.

7. A method according to claim 1, wherein a plurality of connected planar surfaces are defined to represent each position of the object in the three-dimensional computer model.

8. A method according to claim 1, wherein the image data is processed to extract a mask defining a position of the object within the bounding polygon.

9. A method according to claim 1, further comprising the step of creating the three-dimensional computer model.

10. A method according to claim 1, further comprising the step of processing the image data and data defining the three-dimensional computer model to identify a plurality of corresponding points therein, and to define a transformation between the image and the three-dimensional computer model using the corresponding points.

11. A method according to claim 1, further comprising the step of generating image data by rendering an image of the three-dimensional computer model from a chosen viewing direction, in which the texture data is rendered onto a said planar surface.

12. A method according to claim 11, further comprising the step of generating a signal conveying the image data.

13. A method according to claim 12, further comprising the step of recording the signal.

14. A method according to claim 11, further comprising the step of displaying an image of the object using the generated image data.

15. A method according to claim 11, further comprising the step of making a recording of the image data either directly or indirectly.

34. A storage device storing a computer program for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 1 to 17.

35. A signal conveying a computer program for causing a programmable processing apparatus to become operable to perform a method according to any one of claims 1 to 17.

16. A method of creating a three-dimensional computer model of an object moving on a surface in a scene, comprising:

receiving image data defining a sequence of images of the object moving on the surface in the scene; and

modelling the object using at least one planar surface having a position dependent upon a position of the object in the scene determined from an image of the scene and a transformation defining a mapping between the surface on which the object is moving and a predetermined surface in the three-dimensional computer model.

17. A method of modelling the movement of an object in a scene, comprising:

receiving image data defining a sequence of images of the object moving in the scene recorded by a recording device; and

processing an image of the object without a priori knowledge of a position from which the image data was recorded by the recording device or imaging parameters of the recording device to generate a planar surface in a three-dimensional computer model of the scene and texture data for rendering onto the planar surface.

18. An image processing apparatus for processing image data defining a sequence of images of an object moving in a scene to produce signals defining a representation of the object at a plurality of positions in a three-dimensional computer model corresponding to positions of the object in the scene, comprising one or more processing components which, in use, are operable to:

process the image data for a plurality of the images to define for each image a respective bounding polygon that bounds the object in the respective image;

define at least one planar surface within the three-dimensional computer model to represent each respective position of the object, each planar surface being defined with its base on a predetermined surface in the three-dimensional computer model and with a position and size in dependence upon a respective one of said bounding polygons; and

generate texture data for the planar surfaces to represent features of the object in dependence upon the image data.

19. Apparatus according to claim 18, wherein a bounding polygon in the image data has a width and an aspect ratio and a planar surface in the three-dimensional computer model has a width and a height, and wherein the one or more processing components define a planar surface such that the width of the planar surface is defined by the width of the bounding polygon in the image data and the height of the planar surface is determined using the aspect ratio of the bounding polygon in the image data.

20. Apparatus according to claim 18, wherein the one or more processing components define each planar surface so as to lie within a vertical plane.

21. Apparatus according to claim 18, wherein the bounding polygon bounding the object is a rectangle.

22. Apparatus according to claim 21, wherein the rectangle has sides and the images has sides, and wherein the sides of the rectangle are parallel to the sides of the image.

23. Apparatus according to claim 18, wherein the one or more processing components define a single planar surface to represent each position of the object in the three-dimensional computer model.

24. Apparatus according to claim 18, wherein the one or more processing components define a plurality of connected planar surfaces to represent each position of the object in the three-dimensional computer model.

25. Apparatus according to claim 18, wherein, in use, the one or more processing components also are operable to perform further processing to process the image data to extract a mask defining a position of the object within the bounding polygon.

26. Apparatus according to claim 18, wherein, in use, the one or more processing components also are operable to perform further processing to create the three-dimensional computer model.

27. Apparatus according to claim 18, wherein, in use, the one or more processing components also are operable to perform further processing to process the image data and data defining the three-dimensional computer model to identify a plurality of corresponding points therein, and to define a transformation between the image and the three-dimensional computer model using the corresponding points.

28. Apparatus according to claim 18, wherein, in use, the one or more processing components also are operable to perform further processing to render the texture data onto said planar surface to generate image data for an image of the three-dimensional computer model from a viewing direction.

29. Apparatus for creating a three-dimensional computer model of an object moving on a surface in a scene, comprising:

a receiver for receiving image data defining a sequence of images of the object moving on the surface in the scene; and

a processor for modelling the object using at least one planar surface having a position dependent upon a position of the object in the scene determined from an image of the scene and a transformation defining a mapping between the surface on which the object is moving and a predetermined surface in the three-dimensional computer model.

30. Apparatus for modelling the movement of an object in a scene, comprising:

a receiver for receiving image data defining a sequence of images of the object moving in the scene recorded by a recording device; and

a processor for processing an image of the object without a priori knowledge of a position from which the image data was recorded by the recording device or imaging parameters of the recording device to generate a planar surface in a three-dimensional computer model of the scene and texture data for rendering onto the planar surface.

31. An image processing apparatus for processing image data defining a sequence of images of an object moving in a scene to produce signals defining a representation of the object at a plurality of positions in a three-dimensional computer model corresponding to positions of the object in the scene, comprising processing means for:

processing the image data for a plurality of the images to define for each image a respective bounding polygon that bounds the object in the respective image;

defining at least one planar surface within the three-dimensional computer model to represent each respective position of the object, each planar surface being defined with its base on a predetermined surface in the three-dimensional computer model and with a position and size in dependence upon a respective one of said bounding polygons; and

generating texture data for the planar surfaces to represent features of the object in dependence upon the image data.

32. Apparatus for creating a three-dimensional computer model of an object moving on a surface in a scene, comprising:

means for receiving image data defining a sequence of images of the object moving on the surface in the scene; and

means for modelling the object using at least one planar surface having a position dependent upon a position of the object in the scene determined from an image of the scene and a transformation defining a mapping between the surface on which the object is moving and a predetermined surface in the three-dimensional computer model.

33. Apparatus for modelling the movement of an object in a scene, comprising:

means for receiving image data defining a sequence of images of the object moving in the scene recorded by a recording device; and

means for processing an image of the object without a priori knowledge of a position from which the image data was recorded by the recording device or imaging parameters of the recording device to generate a planar surface in a three-dimensional computer model of the scene and texture data for rendering onto the planar surface.