Image sensing apparatus and method capable of merging function for obtaining high-precision image by synthesizing images and image stabilization function

• US 20020097324A1
• Filed: 01/22/2002
• Published: 07/25/2002
• Est. Priority Date: 12/27/1996
• Status: Active Grant

• Current Assignee: Ichiro Onuki
• Original Assignee: Ichiro Onuki
• Inventors: Onuki, Ichiro
• Granted Patent: US 7,042,509 B2
• Time in Patent Office: Days
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/663 for controlling interchangeable camera parts based on electronic image sensor signals
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/687 by shifting the lens or sensor position
• CPC Class Codes: H04N 25/48 Increasing resolution by shifting the sensor relative to the scene

Abstract

An image sensing apparatus, which performs pixel shifting operation for shifting an image formation position of an image on an image sensing device using a lens capable shifting a light path and senses images at respective image formation positions, has a vibration sensor for detecting vibration. The lens is moved so as to cancel the vibration detected by the vibration sensor when performing pixel shifting operation. Since the shifting characteristics of the lens slightly changes depending upon states of zooming and focusing lenses, the lens is moved by a shift amount which is adjusted in accordance with the states of the zooming and focusing lens.

Claims

1. An image sensing apparatus comprising:

image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals;

shifting means for shifting an image formation position of the optical image formed on said image sensing means to a plurality of different positions;

first vibration detection means for detecting vibration of the image sensing apparatus and outputting vibration information;

control means for controlling said shifting means on the basis of the vibration information outputted by said first vibration detection means; and

image signal synthesis means for synthesizing image signals of a plurality of images outputted by said image sensing means to generate a single image,

wherein said image sensing means converts the optical image into electric signals at each of the plurality of different image formation positions shifted by said shifting means.

2. The image sensing apparatus according to claim 1, wherein said control means comprises:

pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation;

vibration compensation data generation means for generating vibration compensation data for compensating the vibration on the basis of the vibration information outputted by said first vibration detection means; and

control data synthesis means for synthesizing the pixel shifting control data and the vibration compensation data to generate a control signal for controlling said shifting means.

3. The image sensing apparatus according to claim 2 further comprising optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system,

wherein said control means further comprises data correction means for correcting at least one of the pixel shifting control data and the vibration compensation data in accordance with the state detected by said optical state detection means.

4. The image sensing apparatus according to claim 3 further comprising storage means for storing first correction data for correcting the pixel shifting control data and second correction data for correcting the vibration compensation data,

wherein said data correction means reads at least one of the first and second correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and performs at least one of an addition of the read first correction data to the pixel shifting control data and an addition of the read second correction data to the vibration compensation data.

5. The image sensing apparatus according to claim 1, wherein said shifting means is a movable optical means.

6. The image sensing apparatus according to claim 1, wherein said shifting means is a plane parallel plate.

7. The image sensing apparatus according to claim 1, wherein said shifting means is a variable apical angle prism.

8. The image sensing apparatus according to claim 1, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation,

wherein said change-over means changes the pixel shifting modes on the basis of the vibration information outputted by said first vibration detection means.

9. The image sensing apparatus according to claim 8, wherein the pixel shifting modes indicate number of the image formation positions to be shifted in pixel shifting operation, and said change-over means selects a first number of image formation positions when a degree of vibration indicated by the vibration information is equal or less than a first predetermined value;

and selects a second number of image formation positions which is smaller than the first number when the degree of vibration is larger than the first predetermined value.

10. The image sensing apparatus according to claim 9, wherein said change-over means selects a third number of image formation positions which is smaller than the second number when the degree of vibration is larger than the second predetermined value which is larger than the first predetermined value

11. The image sensing apparatus according to claim 8 further comprising optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system,

wherein said control means further comprises data correction means for correcting the pixel shifting control data in accordance with the state detected by said optical state detection means.

12. The image sensing apparatus according to claim 11 further comprising storage means for storing correction data for correcting the pixel shifting control data,

wherein said data correction means reads the correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and performs an addition of the read correction data to the pixel shifting control data.

13. The image sensing apparatus according to claim 1, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation,

wherein said change-over means changes between the pixel shifting modes in accordance with a user designation.

14. The image sensing apparatus according to claim 13 further comprising optical system state detection means for detecting a state of at least one of a zooming optical system and a focusing optical system,

wherein said control means further comprises data correction means for correcting the pixel shifting control data in accordance with the state detected by said optical state detection means.

15. The image sensing apparatus according to claim 14 further comprising storage means for storing correction data for correcting the pixel shifting control data,

wherein said data correction means reads the correction data, corresponding to the state detected by said optical state detection means, stored in said storage means, and performs an addition of the read correction data to the pixel shifting control data.

16. The image sensing apparatus according to claim 1 further comprising second vibration detection means for detecting blurring in an image on the basis of the image signals outputted by said image sensing means and outputting blurring information,

wherein said control means controls said shifting means on the basis of the vibration information outputted by said first vibration detection means and the blurring information outputted by said second vibration detection means.

17. The image sensing apparatus according to claim 16, wherein said first vibration detection means is a vibration-type gyro and said second vibration detection means is movement vector detection means.

18. The image sensing apparatus according to claim 1 further comprising photometry means and luminous exposure adjustment means for determining a luminous exposure on the basis of a result of the photometry performed by said photometry means and controlling said image sensing means to use the determined luminous exposure.

19. The image sensing apparatus according to claim 18, wherein, when the image formation position is shifted by said shifting means, said luminous exposure adjustment means determines a different luminous exposure and controls said image sensing means to use the different luminous exposure.

20. The image sensing apparatus according to claim 19, wherein said photometry means performs photometry on each of a plurality of divided areas of an image sensed by said image sensing means.

21. The image sensing apparatus according to claim 20 further comprising main object determination means for determining a divided area which includes a main object among the plurality of divided areas,

wherein said luminous exposure adjustment means controls said image sensing means to use a first luminous exposure which is suitable for the divided area including the main object, which is determined by said main object determination means, and to use at least one of second and third luminous exposures where the second luminous exposure is larger than the first luminous exposure and the third exposure is smaller than the first luminous exposure.

22. The image sensing apparatus according to claim 18, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing exposure time.

23. The image sensing apparatus according to claim 18, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing aperture of an iris diaphragm.

24. The image sensing apparatus according to claim 18, wherein said luminous exposure adjustment means adjusts the luminous exposure using an electrochromic element.

25. The image sensing apparatus according to claim 21, wherein said control means comprises changeover means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation,

wherein said change-over means changes the pixel shifting modes on the basis of the vibration information outputted by said first vibration detection means, and said luminous exposure adjustment means determines the luminous exposure to be used in said image sensing device on the basis of the pixel shifting mode selected by said change-over means.

26. The image sensing apparatus according to claim 25, wherein the pixel shifting modes indicate number of the image formation positions to be shifted in pixel shifting operation, and said change-over means selects a first number of image formation positions when a degree of vibration indicated by the vibration information is equal or less than a first predetermined value;

selects a second number of image formation positions which is smaller than the first number when the degree of vibration is larger than the first predetermined value and equal or less than a second predetermined value which is larger than the first predetermined value; and

selects a third number of image formation positions which is smaller than the second number when the degree of vibration is larger than the second predetermined value.

27. The image sensing apparatus according to claim 21, wherein said control means comprises change-over means for changing between a plurality of pixel shifting modes, and pixel shifting control data generation means for generating pixel shifting control data for controlling said shifting means in pixel shifting operation,

wherein said change-over means changes between the pixel shifting modes in accordance with a user designation and said luminous exposure adjustment means determines the luminous exposure to be used in said image sensing device on the basis of the pixel shifting mode selected by said change-over means.

28. The image sensing apparatus according to claim 1 further comprising blurring detection means for detecting relative blurring amount between two images out of a plurality of images obtained at the plurality of image formation positions shifted by said shifting means by comparing image signals of the two images, and outputting the blurring amount.

29. The image sensing apparatus according to claim 28, wherein said blurring detection means detects the relative blurring amount by comparing the image signals of the two images which are obtained in series out of the plurality of images obtained at the plurality of image formation positions.

30. The image sensing apparatus according to claim 28, wherein said blurring detection means detects the relative blurring amount by comparing the image signals of the two images which are obtained at the same image formation position in series out of the plurality of images obtained at the plurality of image formation positions.

31. The image sensing apparatus according to claim 28 further comprising disabling means for disabling the image signal synthesis means in a case where the blurring amount detected by said blurring detection means is greater than a predetermined value.

32. The image sensing apparatus according to claim 31 further comprising notification means for notifying a user of image synthesis being disabled in a case where said disabling means disabled said image signal synthesis means.

33. The image sensing apparatus according to claim 31, wherein said image signal synthesis means has a plurality of image synthesis modes, and selects one of the plurality of image synthesis modes in accordance with the blurring amount.

34. The image sensing apparatus according to claim 33, wherein the image synthesis modes indicate numbers of images to be synthesized out of the plurality of images sensed by said image sensing means, and said image signal synthesis means selects a first number of images when the blurring amount is equal or less than a first predetermined value;

and selects a second number of images which is smaller than the first number when the blurring amount is larger than the first predetermined value.

35. The image sensing apparatus according to claim 34, wherein said image signal synthesis means selects a third number of images which is smaller than the second number when the blurring amount is larger than the second predetermined value which is larger than the first predetermined value.

36. The image sensing apparatus according to claim 34 further comprising notification means for notifying a user of the image sensing mode selected by said image signal synthesis means.

37. The image sensing apparatus according to claim 28, wherein said image signal synthesis means synthesizes the image signals of the plurality of images on the basis of the blurring amount detected by said blurring detection means so that blurring is compensated.

38. The image sensing apparatus according to claim 37, wherein said image signal synthesis means synthesizes the plurality of images after changing relative positions between the plurality of images on the basis of the blurring amounts between respective pairs of the plurality of images.

40. The image sensing apparatus according to claim 39, wherein said photometry means performs photometry on each of a plurality of divided areas of an image sensed by said image sensing means.

41. The image sensing apparatus according to claim 40 further comprising main object determination means for determining a divided area which includes a main object among the plurality of divided areas,

wherein said luminous exposure adjustment means controls said image sensing means to use a first luminous exposure which is suitable for the divided area including the main object, which is determined by said main object determination means, and to use at least one of second and third luminous exposures where the second luminous exposure is larger than the first luminous exposure and the third exposure is smaller than the first luminous exposure.

42. The image sensing apparatus according to claim 39, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing exposure time.

43. The image sensing apparatus according to claim 39, wherein said luminous exposure adjustment means adjusts the luminous exposure by changing aperture of an iris diaphragm.

44. The image sensing apparatus according to claim 39, wherein said luminous exposure adjustment means adjusts the luminous exposure using an electrochromic element.

45. The image sensing apparatus according to claim 39 further comprising vibration detection means for detecting vibration of the image sensing apparatus, wherein said luminous exposure adjustment means determines luminous exposures to be used on the basis of a detection result by said vibration detection means.

46. The image sensing apparatus according to claim 39 further comprising change-over means for changing between a plurality of image sensing modes which designates the number of image sensing operations to be performed for sensing an object by said image sensing means,

wherein said luminous exposure adjustment means determines luminous exposures to be used in accordance with the number of image sensing operations set by said change-over means.

48. The image sensing apparatus according to claim 47, wherein said relationship determination means is blurring detection means which detects blurring amounts between the plurality of images.

49. The image sensing apparatus according to claim 47, wherein said relationship determination means is correlation calculation means for calculating correlation between the plurality of images.

39. An image sensing apparatus comprising:

image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals;

photometry means;

luminous exposure adjustment means for determining a plurality of luminous exposures to be used in said image sensing means on the basis of a photometry result obtained by said photometry means;

control means for controlling said image sensing means so as to sense an object a plurality of times using the plurality of luminous exposures determined by said luminous exposure adjustment means; and

image signal synthesis means for synthesizing image signals of a plurality of images of the object outputted by said image sensing means to generate a single image.

47. An image synthesis apparatus comprising:

image signal synthesis means having plural kinds of image synthesis control means for synthesizing image signals of a plurality of images to generate a single image;

relationship determination means for determining relationship between the plurality of images; and

selection means for selecting one of the plural kinds of image synthesis control means on the basis of a determination result by said relationship determination means.

50. An image sensing apparatus comprising:

image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals;

optical means for forming the optical image of an object on said image sensing means;

shifting means for shifting an image formation position of the optical image formed on said image sensing means to a plurality of different positions;

pixel shifting control means for controlling said shifting means to shift the image formation position and controlling said image sensing means to sense a plurality of images at the respective image formation positions;

image signal synthesis means for synthesizing image signals of the plurality of images outputted by said image sensing means to generate a single high-resolution image;

blurring detection means for detecting partial blurring due to movement of the object between the plurality of images; and

image synthesis control means for controlling said image synthesis means on the basis of a detection result by said blurring detection means.

51. The image sensing apparatus according to claim 50, wherein said blurring detection means detects blurring between the plurality of images by each of a plurality of divided areas of an image sensed by said image sensing means.

52. The image sensing apparatus according to claim 51, wherein said blurring detection means has correlation calculation means for calculating correlation between the plurality of images, and detects an area where the partial blurring due to movement of the object occurs on the basis of the calculated correlation, further detects a movement vector in the area.

53. The image sensing apparatus according to claim 50, wherein said image synthesis means has a plurality of image synthesis modes, and said image synthesis control means selects one of the plurality of image synthesis modes on the basis of the detection result by said blurring detection means.

54. The image sensing apparatus according to claim 50, wherein, when the partial blurring detected by said blurring detection means is larger than a predetermined value, said image synthesis control means prohibits said image synthesis means to synthesize the plurality of images in an area where the partial blurring is detected.

55. The image sensing apparatus according to claim 50, wherein, when the partial blurring detected by said blurring detection means is larger than a predetermined value, said image synthesis control means performs an image synthesis processing on an area where the partial blurring is detected which is different from an image synthesis processing performed on an area where the partial blurring is not detected.

56. The image sensing apparatus according to claim 55, wherein said image signal synthesis means generates an image in the area where the partial blurring is detected by performing sweeping processing.

57. The image sensing apparatus according to claim 50 further comprising display means,

wherein said image synthesis control means controls said display means in accordance with an image synthesis operation performed by said image synthesis means.

59. The image sensing apparatus according to claim 58 further comprising vibration detection means for detecting vibration of the image sensing apparatus by detecting uniform blurring between the plurality of images,

wherein said relationship determination means is a blurring detection means for detecting movement of the object, and said image synthesis control means controls said image signal synthesis means on the basis of a detection result by said vibration detection means and a detection result by said blurring detection means.

60. The image sensing apparatus according to claim 58 further comprising:

vibration detection means for detecting vibration of the image sensing apparatus by detecting uniform blurring between the plurality of images; and

vibration compensation means for compensating the vibration on the basis of a detection result by said vibration detection means.

58. An image sensing apparatus comprising:

image sensing means for converting an optical image into electric signals and outputting the electric signals as image signals;

image sensing control means for controlling said image sensing means to sense a plurality of images within a predetermined period of time;

image signal synthesis means for synthesizing image signals of the plurality of images outputted by said image sensing means to generate a single image;

division means for dividing an image into a plurality of small areas;

relationship determination means for determining relationship between the plurality of images by each of the plurality of small areas; and

image synthesis control means for controlling image synthesis operation by said image signal synthesis means on the basis of the relationship between the plurality of images determined by said relationship determination means.

61. An image synthesis apparatus comprising:

image signal synthesis means for synthesizing image signals of a plurality of images to generate a single image;

blurring area detection means for detecting an area where a partial blurring occurs between the plurality of images; and

movement vector detection means for detecting a movement vector in the area, detected by said blurring area detection means, between the plurality of images,

wherein said image signal synthesis means synthesizes an image in the area detected by said blurring area detection means by applying a first method using the movement vector, and synthesizes an image outside of the area by applying a second method without using the movement vector.

62. An image synthesis apparatus comprising:

image signal synthesis means for synthesizing image signals of a plurality of images to generate a single image; and

area division means for dividing each of the plurality of images into a first image area where relationship between the plurality of images has a first relationship and a second image area where relationship between the plurality of images has a second relationship,

wherein said image signal synthesis means performs an image synthesis processing, on the first image area, which is different from an image synthesis processing performed on the second image synthesis area.