Apparatus, method, program code, and storage medium for image processing

• US 20020118217A1
• Filed: 03/29/2001
• Published: 08/29/2002
• Est. Priority Date: 02/23/2001
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Fujiki, Masakazu
• Granted Patent: US 6,717,586 B2
• Time in Patent Office: Days
• US Class Current: 345/623
• CPC Class Codes: G06T 15/10 Geometric effects

Abstract

An accumulation buffer is cleared (S501), an real image is input (S502), a color buffer is cleared (S503), and the real image is drawn in the color buffer (S504). Then the viewpoint (including parameters concerning the viewpoint) and the view volume of a virtual camera is set (S505) and a computer graphics image is drawn in the color buffer (S506). Then each pixel value of the image in the color buffer is scaled by a factor of 1/N (S507) and the scaled image is added to the accumulation buffer (S508). The process from step S503 to step S508 is repeated N times (S509) by moving the viewpoint position at step S505. An averaged image provided by averaging, on a pixel basis, N combined images stored in the accumulation buffer is output (S510).

Claims

1. An image processing apparatus for combining a real image with computer graphics image to generate a combined image, comprising:

setting means for setting the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image;

computer graphics image generation means for moving said viewpoint based on the moving distance of said viewpoint set by said setting means to generate the computer graphics image based on the moved viewpoint;

combined image generation means for combining the computer graphics image generated by said computer graphics image generation means with the real image to generate the combined image; and

average image generation means for generating a combined image containing an averaged computer graphics image of computer graphics images contained in a plurality of combined images generated by said combined image generation means corresponding to each of the viewpoints.

3. The image processing apparatus according to claim 1, further comprising display means for displaying the image generated by said average image generation means.

4. The image processing apparatus according to claim 1, further comprising control means for controlling the moving distance of the viewpoint and the position of the focal plane in the computer graphics image.

5. The image processing apparatus according to claim 1, wherein said combined image generation means stores the generated combined image in first memory and said average image generation means performs predetermined scaling for the value of pixels constituting the combined image stored in said first memory and adds the values of the pixels to the values of corresponding pixels of an image stored in second memory different from said first memory to re-store the image.

6. The image processing apparatus according to claim 5, wherein said average image generation means generates an image having the average of the pixel values of pixels in the same position in a plurality of combined images corresponding to said viewpoints.

7. The image processing apparatus according to claim 5, wherein said average image generation means generates an image having the weighted average of the pixel values of pixels in the same position in a plurality of combined images corresponding to said viewpoints.

8. The image processing apparatus according to claim 5, wherein said first memory is a color buffer and said second memory is an accumulation buffer.

16. The image processing apparatus according to claim 1, wherein said computer graphics generation means re-sets a view volume so that the cross section of the view volume cut through the focal plane position does not change when said viewpoint is moved according to the moving distance of said viewpoint.

2. An image processing apparatus for combining a real image with a computer graphics image to generate a combined image, comprising:

calculation means for calculating the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image according to the lens characteristics of image capture means for capturing a real image;

computer graphics image generation means for moving said viewpoint based on the moving distance of said viewpoint calculated by said calculation means to generate the computer graphics image based on the moved viewpoint;

combined image generation means for combining the computer graphics image generated by said computer graphics image generation means with the real image to generate the combined image; and

average image generation means for generating a combined image containing an averaged computer graphics image of computer graphics images contained in a plurality of combined images generated by said combined image generation means corresponding to each of the viewpoints.

17. The image processing apparatus according to claim 2, wherein said calculation means calculates the position of the focal plane of a computer graphics drawing system so that the focal plane position of the computer graphics drawing system is brought into coincidence with the focal plane of a image capture system and provides virtual base points in the computer graphics drawing system and the image capture system in any position different from the focal plane and common to the computer graphics drawing system and the image capture system to calculate the moving distance of the viewpoint so that the blur level at said base point in said computer graphics drawing system become equal to the blur level at said base point in said image capture system.

9. An image processing apparatus for combining a real image with computer graphics image to generate a combined image, comprising:

setting means for setting the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image;

computer graphics image generation means for moving said viewpoint based on the moving distance of said viewpoint set by said setting means to generate the computer graphics image based on the moved viewpoint,

said computer graphics image generation means generating a plurality of computer graphics images corresponding to the viewpoints;

averaged computer graphics image generation means for generating an averaged computer graphics image, said average computer graphics image being an average of said plurality of computer graphics images; and

combined image generation means for combining said averaged computer graphics image generated by said averaged computer graphics image generation means with the real image to generate a combined image.

11. The image processing apparatus according to claim 9, further comprising display means for displaying said combined image generated by said combined image generation means.

12. The image processing apparatus according to claim 9, further comprising control means for controlling the moving distance of the viewpoint and the position of the focal plane in the computer graphics image.

13. The image processing apparatus according to claim 9, wherein said averaged computer graphics image generation means stores the generated computer graphics image in first memory, performs predetermined scaling for the value of pixels constituting said computer graphics image and adds the values of the pixels to the values of corresponding pixels of an image stored in second memory different from said first memory to re-store the image.

14. The image processing apparatus according to claim 13, wherein said averaged computer graphics image generation means generates an image having the average of the pixel values of pixels in the same position in a plurality of computer graphics images corresponding to said viewpoints.

15. The image processing apparatus according to claim 13, wherein said averaged computer graphics image generation means generates an image having the weighted average of the pixel values of pixels in the same position in a plurality of computer graphics images corresponding to said viewpoints.

20. The image processing method according to claim 18, further comprising the display step of displaying the image generated at said average image generation step.

21. The image processing method according to claim 18, further comprising the control step of controlling the moving distance of the viewpoint and the position of the focal plane in the computer graphics image.

22. The image processing method according to claim 18, wherein said combined image generation step stores the generated combined image in first memory and said average image generation step performs predetermined scaling for the value of pixels constituting the combined image stored in said first memory and adds the values of the pixels to the values of corresponding pixels of an image stored in second memory different from said first memory to re-store the image.

23. The image processing method according to claim 22, wherein said average image generation step generates an image having the average of the pixel values of pixels in the same position in a plurality of combined images corresponding to said viewpoints.

24. The image processing method according to claim 22, wherein said average image generation step generates an image having the weighted average of the pixel values of pixels in the same position in a plurality of combined images corresponding to said viewpoints.

25. The image processing method according to claim 22, wherein said first memory is a color buffer and said second memory is an accumulation buffer.

28. The image processing method according to claim 26, further comprising the display step of displaying said combined image generated at said combined image generation step.

29. The image processing method according to claim 26, further comprising the control step of controlling the moving distance of the viewpoint and the position of the focal plane in the computer graphics image.

30. The image processing method according to claim 26, wherein said averaged computer graphics image generation step stores the generated computer graphics image in first memory, performs predetermined scaling for the value of pixels constituting said computer graphics image and adds the values of the pixels to the values of corresponding pixels of an image stored in second memory different from said first memory to restore the image.

31. The image processing method according to claim 30, wherein said averaged computer graphics image generation step generates an image having the average of the pixel values of pixels in the same position in a plurality of computer graphics images corresponding to said viewpoints.

32. The image processing method according to claim 30, wherein said averaged computer graphics image generation step generates an image having the weighted average of the pixel values of pixels in the same position in a plurality of computer graphics images corresponding to said viewpoints.

33. The image processing method according to claim 18, wherein said computer graphics image generation step re-sets a view volume so that the cross section of the view volume cut through the focal plane position does not change when said viewpoint is moved according to the moving distance of said viewpoint.

34. The image processing method according to claim 19, wherein said calculation step calculates the position of the focal plane of a computer graphics drawing system so that the focal plane position of the computer graphics drawing system is brought into coincidence with the focal plane of a image capture system and provides virtual base points in the computer graphics drawing system and the image capture system in any position different from the focal plane and common to the computer graphics drawing system and the image capture system to calculate the moving distance of the viewpoint so that the blur level at said base point in said computer graphics drawing system become equal to the blur level at said base point in the image capture system.

35. A program code for performing the image processing method according to claim 18.

36. A program code for performing the image processing method according to claim 19.

37. A program code for performing the image processing method according to claim 26.

38. A program code for performing the image processing method according to claim 27.

39. A computer-readable storage medium storing the program code set forth in claim 35.

40. A computer-readable storage medium storing the program code set forth in claim 36.

41. A computer-readable storage medium storing the program code set forth in claim 37.

42. A computer-readable storage medium storing the program code set forth in claim 38.

10. An image processing apparatus for combining a real image with computer graphics image to generate a combined image, comprising:

calculation means for calculating the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image according to the lens characteristics of an image capture means for capturing a real image;

computer graphics image generation means for moving said viewpoint based on the moving distance of said viewpoint calculated by said calculation means to generate the computer graphics image based on the moved viewpoint;

said computer graphics image generation means generating a plurality of computer graphics images corresponding to the viewpoints;

averaged computer graphics image generation means for generating an averaged computer graphics image, said average computer graphics image being an average of said plurality of computer graphics images; and

combined image generation means for combining said averaged computer graphics image generated by said averaged computer graphics image generation means with the real image to generate a combined image.

18. A image processing method for combining a real image with a computer graphics image to generate a combined image, comprising:

the setting step of setting the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image;

the computer graphics image generation step of moving said viewpoint based on the moving distance of said viewpoint set at said setting step to generate the computer graphics image based on the moved viewpoint;

the combined image generation step of combining the computer graphics image generated at said computer graphics image generation step with the real image to generate the combined image; and

the average image generation step of generating a combined image containing an averaged computer graphics image of computer graphics images contained in a plurality of combined images generated at said combined image generation step corresponding to each of the viewpoints.

19. An image processing apparatus for combining a real image with a computer graphics image to generate a combined image, comprising:

the calculation step of calculating the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image according to the lens characteristics of image capture means for capturing a real image;

the computer graphics image generation step of moving said viewpoint based on the moving distance of said viewpoint calculated at said calculation step to generate the computer graphics image based on the moved viewpoint;

the combined image generation step of combining the computer graphics image generated at said computer graphics image generation step with the real image to generate the combined image; and

the average image generation step of generating a combined image containing an averaged computer graphics image of computer graphics images contained in a plurality of combined images generated at said combined image generation step corresponding to each of the viewpoints.

26. An image processing method for combining a real image with computer graphics image to generate a combined image, comprising:

the setting step for setting the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image;

the computer graphics image generation step of moving said viewpoint based on the moving distance of said viewpoint set at said setting step to generate the computer graphics based on the moved viewpoint,

said computer graphics image generation step generating a plurality of computer graphics images corresponding to the viewpoints;

the averaged computer graphics image generation step of generating an averaged computer graphics image, said average computer graphics image being an average of said plurality of computer graphics images; and

the combined image generation step of combining said averaged computer graphics image generated at said averaged computer graphics image generation step with the real image to generate a combined image.

27. An image processing method for combining a real image with computer graphics image to generate a combined image, comprising:

the calculation step of calculating the moving distance of a viewpoint and the position of a focal plane for generating the computer graphics image according to the lens characteristics of an image capture step of capturing a real image;

the computer graphics image generation step of moving said viewpoint based on the moving distance of said viewpoint calculated at said calculation step to generate the computer graphics image based on the moved viewpoint,

said computer graphics image generation step generating a plurality of computer graphics images corresponding to the viewpoints;

the averaged computer graphics image generation step of generating an averaged computer graphics image, said average computer graphics image being an average of said plurality of computer graphics images; and

the combined image generation step of combining said averaged computer graphics image generated at said averaged computer graphics image generation step with the real image to generate a combined image.

43. An image processing apparatus comprising:

means for generating an image having depth information based on predetermined space coordinates; and

image processing means for controlling the generated state of said image based on depth information about said image and position information about a focal plane when said space is observed.

44. The image processing apparatus according to claim 43, wherein said image processing means performs a process for changing the sharpness of said image based on the distance from said focal plane to the depth position of said image.

45. The image processing apparatus according to claim 44, wherein said image processing means changes said sharpness by performing a process for composing an image, said image being an average of said image viewed from a plurality of viewpoints.

46. The image processing apparatus according to claim 43, wherein said image is a computer graphics image.

47. The image processing apparatus according to claim 45, wherein said image processing means changes said sharpness by performing a process for composing an image, said image being an average of images when the viewpoint for said computer graphics image is moved to a plurality of positions.

49. The image processing apparatus according to claim 48, wherein said image processing means changes the sharpness of said second image based on information about a focal plane when said first image is captured and information about the depth position of said second image with respect to said focal plane.

50. The image processing apparatus according to claim 49, wherein said first image is real image and said second image is a computer graphics image.

51. The image processing apparatus according to claim 50, wherein said image processing means changes the sharpness of said computer graphics image based on the distance from said focal plane to the depth position of said computer graphics image.

52. The image processing apparatus according to claim 51, wherein said image processing means changes said sharpness by performing a process for composing an image, said image being an average of said computer graphics image from a plurality of viewpoints.

48. An image processing apparatus comprising:

image generation means for combining a first image with a second image to generate a combined image; and

image processing means for controlling the generated state of said second image according to position information about a focal plane set for said first image.

53. An image processing method comprising:

the step of generating an image having depth information based on predetermined space coordinates; and

the image processing step for controlling the generated state of said image based on depth information about said image and position information about a focal plane when said space is observed.

55. A program code for performing the image processing method according to claim 53.

57. A computer-readable storage medium storing the program code set forth in claim 55.

54. An image processing method comprising:

the image generation step of combining a first image and a second image to generate a combined image; and

the image processing step of controlling the generated state of said second image according to position information about a focal plan e set for said first image.

56. A program code for performing the image processing method according to claim 54.

58. A computer-readable storage medium storing the program code set fort in claim 56.