Image generation method and apparatus

• US 20020085748A1
• Filed: 10/19/2001
• Published: 07/04/2002
• Est. Priority Date: 10/27/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Baumberg, Adam Michael
• Granted Patent: US 7,120,289 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

An image processing apparatus (2) is disclosed in which input images are processed to generate texture map data for texture rendering a generated three-dimensional computer model of object(s) appearing in the images. In order to select the portions of the images utilised, confidence data is generated indicative of the extent portions of the surface of a model are visible in each of the images. The images are then combined utilising this confidence data, where image data representative of different spatial frequencies are blended in different ways utilising the confidence data.

Claims

1. A method of generating frequency component image data representative of image information of a plurality of different spacial frequencies of an image of at least part of a subject from a predefined view, the method comprising steps of:

receiving a plurality of images comprising image data representative of a subject from a plurality of view points;

for each of said plurality of received images, determining frequency component image data representative of image information of a plurality of different spacial frequencies within said received images and determining projections of said frequency component image data projected to said predefined view; and

for each of said plurality of different spacial frequencies, utilising said projections of frequency component image data to generate frequency component image data representative of spacial frequency information of an image of at least part of said subject from said predefined view.

2. A method in accordance with claim 1, wherein the determination of frequency component image data representative of a lower spacial frequency is determined by performing a blurring and sub-sampling operation utilizing said received image data.

3. A method in accordance with claim 2, wherein the determination of frequency component image data representative of a higher frequency is determined by performing a differencing operation utilising said lower frequency component image data and said received image data.

4. A method in accordance with claim 1, wherein said projections of said frequency component image data are determined by performing the steps of:

determining from said plurality of received images the relative positions of the viewpoints of the subject represented by said images;

determining from said plurality of images and said determined positions, model data representative of the surface of said subject; and

determining said projections utilising said position data and said model data of the surface of said subject.

5. A method in accordance with claim 4, wherein said determination of position data indicative of the relative position of said plurality of viewpoints for said received images comprises matching features within said images to determine relative position and orientation of the viewpoints.

6. A method in accordance with claims 4, wherein said generation of model data comprises for each of said plurality of received images the steps of:

determining the portions of an image corresponding to the subject; and

generating model data utilising a determination of the perimeters of said portion of said image and position data indicative of the viewpoint for said image.

7. A method in accordance with claim 1, wherein said generation of frequency component image data comprises for each spatial frequency for which frequency component image data is to be generated the step of:

utilising each portion of said projections of frequency component image data to determine frequency component image data representative of spatial frequency information of said spacial frequency for corresponding portions of said image.

8. A method in accordance with claim 7, wherein said generation of frequency component image data for different spatial frequencies for an image are generated from said corresponding portions of projections of frequency component image data in different ways for different spatial frequencies.

9. A method in accordance with claim 7, wherein said generation of frequency component image data further comprises the steps of:

for each of said plurality of images, determining visibility data indicative the relative visibility of portions of the surface of said subject from said viewpoints;

associating portions of said projections of frequency component image data, with said visibility data; and

utilising said visibility data associated with portions of said projections to determine frequency component image data representative of frequency image information of an image of said subject from said predefined views.

10. A method in accordance with claim 9, wherein said generation of frequency component image data for at least one of said different spacial frequencies is generated by calculating for each portion of said image data for said spacial frequency a weighted average of corresponding portions of projections of frequency component image data for said portion of said image, weighted by said visibility data associated with said portions of image data.

11. A method in accordance with claim 9, wherein said generation of frequency component image data for at least one of said different spacial frequencies is generated by selecting for each pixel of said image data for said spacial frequency said corresponding portions of projections of frequency component image data associated with visibility data indicative of the highest visibility.

12. A method of generating image data representative of an image of a subject from a predefined viewpoint, the method of comprising the steps of:

generating frequency component image data representative of image information of a plurality of different spacial frequencies of an image of a subject from said predefined view in accordance with a method of claim 1;

and

generating image data representative said image of said subject utilising said frequency component image data.

13. A method in accordance with claim 12, wherein said generation of image data comprises for each portion of said image determining the sum of frequency component image data for said different spacial frequencies.

14. A method in accordance with claim 12 further comprising the step of outputting said generated image data.

15. A method of generating texture map data for texture rendering a model of a subject comprising the steps of:

generating a plurality of images of a subject from a plurality of predefined views in accordance with the method of claim 12.

16. A method in accordance with claim 15, wherein said plurality of predefined views comprise views of a subject from viewpoints indicative of views from the surface of a cuboid bounding said subject.

17. A method of generating image data of an image of a subject from a selected viewpoint, the method comprising the steps of:

generating texture map data in accordance with claim 15;

determining the projection of a three dimensional model defined by said model data as perceived from said selected viewpoint; and

texture rendering said projection of said model utilising data said texture map data.

18. A method in accordance with claim 17 further comprising a step of outputting said generated image data of a subject from a selected viewpoint.

20. Apparatus in accordance with claim 19, wherein said determination unit is operable to determine frequency component image data representative of a lower spacial frequency by performing a blurring and sub-sampling operation utilizing said image data received by said receiver.

21. Apparatus in accordance with claim 20 wherein said determination unit is arranged to determine frequency component image data representative of a higher frequency by performing a differencing operation utilising a said lower frequency component image data and a said image data received by said receiver.

22. Apparatus in accordance with claim 19, wherein said determination unit comprises:

a position determination unit operable to determine from a said plurality of received images the relative positions of the viewpoints of the subject represented by said images;

a modeling unit for determining from a said plurality of images and said determined positions, model data representative of the surface of said subject; and

a projection generation unit operable to determine said projections utilising said position data and said model data of the surface of said subject.

23. Apparatus in accordance with claim 22, wherein said position determination unit is operable to determine position data indicative of the relative position of said plurality of viewpoints for said received images by matching features within said images to determine relative position and orientation of the viewpoints.

24. Apparatus in accordance with claim 22, wherein said modeling unit is arranged to generate model data by:

determining the portions of an image corresponding to the subject; and

generating model data utilising a determination of the perimeters of said portion of said image and position data indicative of the viewpoint for said image.

25. Apparatus in accordance with claim 19, wherein said image generator is arranged to generate frequency component image data for different spatial frequencies by utilising each portion of said projections of frequency component image data for a frequency to determine frequency component image data representative of spatial frequency information of said spacial frequency for corresponding portions of said image.

26. Apparatus in accordance with claim 25, wherein said image generator is arranged to generate frequency component image data for different spatial frequencies for an image from said corresponding portions of projections of frequency component image data in different ways for different spatial frequencies.

27. Apparatus in accordance with claim 25, wherein said image generator further comprises:

a visibility determination unit operable to determine visibility data indicative of the relative visibility of portions of the surface of said subject from said viewpoints;

an association unit operable to associate portions of said projections of frequency component image data, with said visibility data; and

a processor for utilising visibility data associated with portions of said projections to determine frequency component image data representative of frequency image information of an image of said subject from said predefined views.

28. Apparatus in accordance with claim 27, wherein said processor is arranged to generate frequency component image data by calculating for each portion of said image data for a said spacial frequency a weighted average of corresponding portions of projections of frequency component image data for said portion of said image, weighted by said visibility data associated with said portions of image data.

29. Apparatus in accordance with claim 27, wherein said processor is arranged to generate frequency component image data by selecting for each pixel of said image data for a said spacial frequency said corresponding portions of projections of frequency component image data associated with visibility data indicative of the highest visibility.

37. A recording medium for storing computer implementable process steps for causing a programmable computer to perform a method in accordance with any claims 1 to 18 or 35.

19. Apparatus for generating frequency component image data representative of image information of a plurality of different spacial frequencies of an image of at least part of a subject from a predefined view, the apparatus comprising:

a receiver operable to receive a plurality of images comprising image data representative of a subject from a plurality of view points;

a determination unit operable to determine for each of said plurality of received images, frequency component image data representative of image information of a plurality of different spacial frequencies within said images received by said receiver and determining projections of said frequency component image data projected to said predefined view; and

an image generator operable to utilise said projections of frequency component image data determined by said determination unit for each of said plurality of different spacial frequencies, to generate frequency component image data representative of spacial frequency information of an image of at least part of said subject from said predefined view.

30. Apparatus for generating image data representative of an image of a subject from a predefined viewpoint, the apparatus comprising:

frequency component generator operable to generate frequency component image data representative of image information of a plurality of different spacial frequencies of an image of a subject from said predefined view comprising apparatus in accordance with claim 19; and

a combining unit operable to generate image data representative of said image of said subject utilising said frequency component image data.

31. Apparatus in accordance with claim 30, wherein said combining unit is arranged to determine for each portion of said image a determined the sum of frequency component image data for said different spacial frequencies.

32. Apparatus in accordance with claims 30 and 31 further comprising an output unit operable to output said generated image data.

33. Apparatus for generating image data of a subject from selected viewpoint comprising:

a map generating unit for generating texture map data for texture rendering a model of said subject, comprising apparatus in accordance with claim 30;

a projection unit operable to determine the projection of three dimensional computer model as perceived from said selected viewpoint; and

a texture renderer operable to texture render said projection utilising texture map data generated by said map generating unit.

36. A recording medium for storing computer implementable process steps for generating within a programmable computer, apparatus in accordance with any of claims 19 to 34.

38. A recording medium in accordance with claim 37 comprising a computer disc.

39. A computer disc in accordance with claim 38 comprising an optical, magneto-optical or magnetic disc.

40. A recording medium in accordance with claim 37, comprising an electrical signal transferred via the Internet.

34. An apparatus for generating texture render data comprising:

a confidence image generator operable to generate a plurality of confidence images, said confidence images being representative of the relative visibility of parts of the surface of an object from defined viewpoints;

an association unit operable to associate each of said confidence images with a source image, at least one of the source images comprising an image other than an image of said object from the defined viewpoint represented by said confidence image; and

a texture generator operable to generate texture render data by selecting portions of said source images on the basis of corresponding portions of said associated confidence images.

35. A method of generating texture render data comprising the steps of:

generating a plurality of confidence images, said confidence images being representative of the relative visibility of parts of the surface of an object from defined viewpoints;

associating each of said confidence images with a source image, at least one of the source images comprising an image other than an image of said object from the defined viewpoint represented by said confidence image; and

generating texture render data by selecting portions of said source images on the basis of corresponding portions of said associated confidence images.