Image processing scheme

• US 7,548,256 B2
• Filed: 10/07/2004
• Issued: 06/16/2009
• Est. Priority Date: 10/18/2003
• Status: Expired due to Fees

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Pilu, Maurizio
• Primary Examiner(s): Ho, Tuan V
• Assistant Examiner(s): HOGUE, DENNIS A
• Application Number:
  US10/960,157
• Publication Number:
  US 20050104971A1
• Time in Patent Office: 1,713 Days
• Field of Search: 34820899-2086, 348/159, 348/208.3, 386/117, 396 52- 55, 382/260, 600/519
• US Class Current: 348/208.3
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/6842 by controlling the scanning position, e.g. windowing
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

A method for processing an image sequence captured using an image capture device, wherein a captured sequence to be stabilized by estimating motion of the device, abstracting data from the image sequence, stabilizing the image sequence in response to the estimated motion of the device, and forming an output image sequence from the stabilized image sequence and the abstracted data.

Claims

1. A method of processing an image sequence captured by an image capture device, the method comprising:

estimating motion of the device, to determine estimated motion data;

determining intentional movement data from the estimated motion data;

stabilizing the image sequence by reducing movement in the image sequence in response to the estimated motion of the device, wherein reducing movement in the image sequence includes reducing the effects of intentional movement on the sequence; and

forming an output image sequence by combining the determined intentional movement data with the stabilized image sequence.

2. The method as claimed in claim 1, further comprising:

providing at least one model associated with the image sequence; and

determining, on the basis of the at least one model and the image sequence, the intentional movement data.

3. The method as claimed in claim 2, wherein the at least one model is adapted to model an aspect of the image sequence.

4. The method as claimed in claim 3, wherein determining intentional movement data further comprises performing a probability computation in order to resolve, on the basis of the image sequence and the at least one model, the likelihood that the aspect has occurred in the image sequence.

5. The method as claimed in claim 2, wherein the model is a hidden Markov model.

6. The method as claimed in claim 1, wherein forming the output image sequence further comprises blending the image sequence and the determined intentional movement data using a blending function.

7. The method as claimed in claim 6, wherein the blending function is a Gaussian function.

8. The method as claimed in claim 1, wherein determining the intentional movement data further comprises determining intentional movement of the image capture device.

9. The method as claimed in claim 8, wherein stabilizing the image sequence further comprises extracting the determined intentional movement from the image sequence.

10. The method as claimed in claim 9, wherein forming the output image sequence further comprises introducing the extracted intentional movement back into the output image sequence.

11. The method as claimed in claim 8, wherein determining the intentional movement further comprises comparing the image sequence against a plurality of intentional movement models.

12. The method as claimed in claim 11, wherein the plurality of intentional movement models are a plurality of Hidden Markov Models (HMMs).

13. The method as claimed in claim 11, further comprising determining the plurality of intentional movement models from training image data.

14. The method as claimed in claim 13, wherein the training image data is a pre-captured video sequence which includes a number of intentional movements associated with a scene or object of interest.

15. A method of processing an image sequence comprising a plurality of image frames, the method comprising:

estimating relative motion between the image frames to determine estimated motion data;

determining intentional movement data from the estimated motion data;

stabilizing the image sequence by reducing movement in the image sequence in response to the estimated motion, wherein reducing movement in the image sequence includes reducing the effects of intentional movement on the sequence; and

forming an output image sequence by combining the determined intentional movement data with the stabilized image sequence.

16. The method as claimed in claim 15, further comprising:

providing at least one model associated with the image sequence; and

determining, on the basis of the at least one model and the image sequence, the intentional movement data.

17. The method as claimed in claim 16, wherein the at least one model is adapted to model an aspect of the image sequence.

18. The method as claimed in claim 17, wherein determining intentional movement data further comprises performing a probability computation in order to resolve, on the basis of the image sequence and the at least one model, the likelihood that the aspect has occurred in the image sequence.

19. The method as claimed in claim 16, wherein the model is a hidden Markov model.

20. The method as claimed in claim 15, wherein forming the output image sequence further comprises blending the image sequence and the determined intentional movement data using a blending function.

21. The method as claimed in claim 20, wherein the blending function is a Gaussian function.

22. A computer program product for use with a computer, the computer program product comprising:

a computer useable medium having computer executable program code embodied thereon, wherein the product is operable, in association with the computer, to process an image sequence captured by an image capture device by;

estimating motion of the device, to determine estimated motion data;

determining intentional movement data from the estimated motion data;

stabilizing the image sequence by reducing movement in the image sequence in response to the estimated motion of the device, wherein reducing movement in the image sequence includes reducing the effects of intentional movement on the sequence; and

forming an output image sequence by combining the determined intentional movement data with the stabilized image sequence.

23. An image capture device, comprising:

an image sensor that captures image sequence data;

a motion estimation module that senses motion of the image capture device;

a central processing device (CPU); and

a memory having computer executable program code embodied thereon, wherein upon execution of the code by the CPU, the image sequence data is processed by;

estimating motion of the image capture device based upon information from the motion estimation module, to determine estimated motion data;

determining intentional movement data from the estimated motion data;

stabilizing the image sequence data by reducing movement in the image sequence data in response to the estimated motion of the device, wherein reducing movement in the image sequence data includes reducing the effects of intentional movement on the sequence; and

forming an output image sequence by combining the determined intentional movement data with the stabilized image sequence data .

24. The image capture device as claimed in claim 23, wherein the motion estimation module further comprises at least one internal sensor that senses the motion of the image capture device.

25. The image capture device as claimed in claim 23, wherein the motion of the image capture device is inferred from the image sequence data using inter-frame displacements.

26. The image capture device as claimed in claim 23, further comprising at least one Hidden Markov Model (HMM) that determines intentional movement data that is extracted from the image sequence data, and wherein the intentional movement data is introduced back into the output image sequence when the output image sequence is formed.

27. The image capture device as claimed in claim 26, further comprising training image data, wherein the training image data is used to determine the HMM.

28. The image capture device as claimed in claim 27, wherein the training image data is a pre-captured video sequence which includes a number of intentional movements associated with a scene or object of interest.

29. The image capture device as claimed in claim 23, further comprising a look-up table having data corresponding to a predetermined set of intentional movements, wherein a rule based scheme determines intentional movement data that is extracted from the image sequence data, and wherein the intentional movement data is introduced back into the output image sequence when the output image sequence is formed.