Method and apparatus for processing camera an image produced by a video camera to correct for undesired motion of the video camera

• US 4,959,725 A
• Filed: 07/13/1988
• Issued: 09/25/1990
• Est. Priority Date: 07/13/1988
• Status: Expired due to Term

• Current Assignee: Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Mandle, Gary
• Primary Examiner(s): Groody, James J.
• Assistant Examiner(s): NOT, DEFINED
• Application Number:
  US07/218,626
• Time in Patent Office: 804 Days
• Field of Search: 358/222, 358/217, 358/223, 358/229, 358/105, 358/125, 358/126
• US Class Current: 348/208.6
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/682 Vibration or motion blur correction

Abstract

A method and apparatus for processing a raw video camera signal to identify undesired multidimensional video image motion caused by rotational or translational camera motion in two or three dimensions, and to generate a corrected video camera signal in which such undesired video image motion is reduced. In a preferred embodiment, the apparatus includes a video camera having an oversized image pickup device, four accelerometers mounted in the video camera, and means for processing the accelerometer output signals and the raw video camera output signal to generate a corrected video camera signal. The processing means preferably employs selective clocking to generate a corrected video signal representing a stabilized version of the view of interest. Such a corrected video signal will exclude extraneous visual information (i.e., visual information other than the view of interest), and will be "stabilized" in the sense that it exhibits less image motion than does the draw video camera signal. The preferred embodiment of the invention is capable of efficiently correcting for undesired camera rotation about three orthogonal rotational axes, as well as for undesired camera translations in a plane orthogonal to the axis of symmetry of the camera lens.

Claims

1. A system for processing an image generated by a video camera, including:

(a) an array of image sensing elements having dimensions larger than the image, wherein the array is oriented for exposure to the image so that undesired motion of the camera will cause undesired image motion relative to the array;

(b) a set of camera motion sensors attached to the camera, for generating camera motion signals indicative of the camera'"'"'s motion in at least two dimensions; and

(c) a means for processing the camera motion signals to generate control signals for extracting from the array a stabilized array signal indicative of the image, wherein the stabilized array signal exhibits reduced undesired image motion due to undesired camera motion.

2. The system of claim 1, wherein the array comprises an image region to which the image is exposed, and a peripheral region, and wherein element (c) includes:

(d) means responsive to the control signals for scanning the image region of the array to generate said stabilized array signal.

3. The system of claim 1, wherein the array comprises an image region to which the image is exposed, and a peripheral region, and wherein element (c) includes:

(e) a means for scanning the array to generate a raw array signal, said raw array signal comprising a first component representing the scanned peripheral region, and a second component representing the scanned image region; and

(f) means responsive to the control signals for generating said stabilized array signal by extracting the second component from the raw array signal.

4. The system of claim 1, also including means for generating a video signal from the stabilized array signal.

5. A method for processing an image generated by a video camera, including the steps of:

(a) exposing an array of image sensing elements to the image, said array having dimensions larger than the image so that undesired motion of the camera during exposure will cause undesired image motion relative to the array;

(b) contemporaneously with step (a), generating camera motion signals indicative of the camera'"'"'s motion in at least two dimensions;

(c) generating from the camera motion signals a set of control signals for reducing undesired image motion due to undesired camera motion; and

(d) extracting a stabilized array signal from the array in response to the control signals, wherein said stabilized array signal is indicative of the image and exhibits reduced undesired image motion due to undesired camera motion.

6. The method of claim 5, wherein the array comprises an image region to which the image is exposed, and a peripheral region, and including the step of:

(e) scanning the image region of the array, in response to the control signals, to generate said stabilized array signal.

7. The method of claim 5, wherein the array comprises an image region to which the image is exposed, and a peripheral region, and including the steps of:

(f) scanning substantially the entire array to generate a raw array signal, said raw array signal comprising a first component representing the scanned peripheral region, and a second component representing the scanned image region; and

(g) generating said stabilized array signal by extracting the second component from the raw array signal in response to the control signals.

8. The method of claim 5 also including the step of:

(h) generating a video signal from the stabilized array signal.

9. A video signal processing system, including:

(a) a video camera, including an oversized image pickup device for sequentially producing a set of output signals each indicative of a primary image and a secondary image, and also including a means for generating a raw video camera signal from the output signals, wherein the raw video camera signal exhibits primary image motion corresponding to undesired motion of the camera relative to a view being imaged by the camera;

(b) a camera motion sensor array attached to the camera, for generating camera motion signals indicative of the camera'"'"'s motion in at least two dimensions; and

(c) means for processing the camera motion signals and the raw video camera signal to generate a corrected video signal which exhibits less primary image motion than does the raw vide camera signal.

10. The system of claim 9, wherein the image pickup device includes a video image region and a peripheral region.

11. The system of claim 9, wherein the image pickup device includes rectangular pixel array, and wherein said rectangular pixel array includes a video image region and a peripheral region.

12. The system of claim 11, wherein the rectangular pixel array is a 768 x 751 pixel array, and the video image region is a 510 x 492 pixel array.

13. The system of claim 9, wherein the camera motion sensor array includes an array of accelerometers.

14. The system of claim 9, wherein the accelerometers measure rotational motion of the camera in three dimensions.

15. The system of claim 9, wherein the accelerometers measure translational motion of the camera in two dimensions.

16. The system of claim 9, wherein the accelerometers measure rotational of the camera in two dimensions.

17. The system of claim 9, wherein the accelerometers measure translational motion of the camera in two dimensions.

18. The system of claim 9, wherein the camera motion sensor array includes an array of accelerometers, and also including:

(d) a movement filter for filtering the output of each accelerometer and for supplying the filtered accelerometer output signals to element (c), wherein each filtered accelerometer output signal will indicate camera motion to element (c) only if the output of the associated accelerometer has magnitude above a selected value.

19. The system of claim 9, wherein element (c) includes a memory unit, a means for writing only the primary image component of the raw video camera signal into the memory unit, and a means for reading the primary image component out of the memory unit to produce the corrected video signal.

20. The system of claim 19, wherein the memory unit includes a first charge-coupled device and a second charge-coupled device, and wherein element (c) includes:

a means for sequentially writing adjacent horizontal lines comprising the raw video camera signal'"'"'s primary image component into altering ones of the first and second charge-coupled devices; and

a means for reading one of said horizontal lines out of the second charge-coupled device while writing the next sequential one of said horizontal lines into the first charge-coupled device.

21. A video signal processing method, including the steps of:

(a) A video signal processing method, including the steps of;

(a) exposing an oversized image pickup device of a video camera to generate a set of output signals, in a manner so that each output signal includes a primary image and a secondary image;

(b) generating a raw video camera signal from the output signals in a manner so that the raw video camera signal exhibits primary image motion corresponding to undesired motion of the camera during step (a);

(c) during step (a), generating camera motion signals indicative of motion of the camera in at least two dimensions;

(d) processing the camera motion signals and the raw video camera signal to generate a corrected video signal, where the corrected video signal exhibits less primary image motion than does the raw video camera signal.

22. The method of claim 21, wherein step (d) includes the steps of selectively clocking the raw video camera signal to separate the primary image components of the raw video camera signal from the secondary image components of the raw video camera signal, and generating the corrected video signal from the primary image components.

23. The method of claim 21, wherein the corrected video signal has NTSC format.

24. The method of claim 21, wherein the corrected video signal has PAL format.

25. The method of claim 21, wherein step (b) includes the step of measuring camera acceleration due to rotational motion of the camera in three dimensions.

26. The method of claim 25, wherein step (b) also includes the step of measuring camera acceleration due to translational motion of the camera in two dimensions.

27. The method of claim 21, wherein step (b) includes the step of measuring camera acceleration due to rotational motion of the camera in two dimensions.

28. The method of claim 21, wherein step (b) includes the step of measuring camera acceleration due to translational motion of the camera in two dimensions.