Method and device for determining a displacement of images in an image sequence

• US 7,250,966 B1
• Filed: 01/15/2000
• Issued: 07/31/2007
• Est. Priority Date: 03/05/1999
• Status: Expired due to Fees

• Current Assignee: Robert Bosch GmbH
• Original Assignee: Robert Bosch GmbH
• Inventors: Engelsberg, Andreas
• Primary Examiner(s): Villecco; John M.
• Application Number:
  US09/914,982
• Time in Patent Office: 2,754 Days
• Field of Search: 348/170, 348/208.99, 348/208.1, 348/208.3, 348/208.4, 348/208.14, 348/240.2, 358/464, 358/540, 358/3.26, 382/289, 382/291, 382/293, 382/294, 382/295, 396/52, 396/54, 396/55
• US Class Current: 348/208.3
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20

Abstract

A device and a method for determining a displacement of images in an image sequence, which are specially used to compensate for a camera movement. A plurality of image regions of the images is available for determining the displacement, each image region being provided at a preselected position of the images, each image region having preselected dimensions, in particular, a predetermined number of pixels in various directions of the image, and an image movement mainly caused by a camera movement being separated from an additional movement that is superimposed on the image movement in a subregion of the image to be corrected, by calculating the probability of the movement occurring without the additional movement is calculated at different positions of an image, specifying and fixing for each image region, the position and the dimensions as a function of the probability of the image movement occurring inside the one image region without the additional movement, and preferentially using at least one image region out of the plurality of image regions for determining the displacement.

Claims

1. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement; and

determining the displacement by preferentially using the at least one image region of the plurality of image regions.

2. The method of claim 1, wherein:

the method includes;

correcting a first image of the image sequence by determining the displacement from one of (i) first image data of the first image of the image sequence and second image data of a second image of the image, and (ii) the first image data of the first image of the image sequence and starting image data of a starting image sequence; and

determining a regional displacement of the at least one image region of the plurality of the image regions is determined from one of (i) the first image data of the first image and the second image inside the at least one image region, and (ii) the first image data of the first image and the second starting image inside the at least one image region; and

the step of determining the displacement uses the regional displacement of the at least one image region of the plurality of image regions as the displacement.

3. The method of claim 2, wherein:

two first image regions and one second image region are available for image correction; and

the displacement is determined from one of the following, in a descending priority order;

a mean of regional displacements of the two first image regions, when a reliability of the regional displacement determinations for the two first image regions is considered sufficient;

a regional displacement of one of the two first image regions, when a reliability of the regional displacement determination is considered sufficient; and

a regional displacement of the second image region.

4. The method of claim 2, wherein at least one of at least one first image region and at least one second image region is used as a function of a reliability factor for determining the regional displacement to determine the displacement.

5. The method of claim 4, wherein for each of the image regions, the regional displacement is determined using block-displacement information from a block-based encoding process by considering the block-displacement information of blocks lying inside a specific image region.

6. The method of claim 5, wherein the block-displacement information includes displacement vectors.

7. The method of claim 2, wherein the second image data of the second image of the image sequence precedes directly the first image.

8. The method of claim 2, wherein the starting image data of the starting image of the image sequence precedes directly the first image.

9. The method of claim 1, wherein the position and the dimensions of at least one first image region are selected so that the at least one first image region of images to be corrected is mainly filled in by an image background.

10. The method of claim 1, wherein the position and the dimensions of at least one second image region are selected so that the at least one second image region of images to be corrected is mainly filled in by an image foreground.

11. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, wherein the method is used for a head-shoulder recording situation, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement;

determining the displacement by preferentially using the at least one image region of the plurality of image regions;

correcting a first image of the image sequence by determining the displacement from one of (i) first image data of the first image of the image sequence and second image data of a second image of the image, and (ii) the first image data of the first image of the image sequence and starting image data of a starting image sequence;

determining a regional displacement of the at least one image region of the plurality of the image regions is determined from one of (i) the first image data of the first image and the second image inside the at least one image region, and (ii) the first image data of the first image and the second starting image inside the at least one image region, wherein the step of determining the displacement uses the regional displacement of the at least one image region of the plurality of image regions as the displacement;

selecting the two first image regions to be in a lateral region to the left and right of a vertical center line of a predetermined rectangular image;

selecting a distance of the two first image regions from a bottom image margin to be greater than another distance of the two first image regions from a top image margin; and

selecting the second image region to be near a center of the image;

wherein;

two first image regions and one second image region are available for image correction; and

the displacement is determined from one of the following, in a descending priority order;

a mean of regional displacements of the two first image regions, when a reliability of the regional displacement determinations for the two first image regions is considered sufficient;

a regional displacement of one of the two first image regions, when a reliability of the regional displacement determination is considered sufficient; and

a regional displacement of the second image region.

12. The method of claim 11, wherein the two first image regions are selected to be in the lateral region to the left and right of the vertical center line of the predetermined rectangular image, and symmetrical to the vertical center line.

13. The method of claim 12, wherein the second image region is selected to be near the center of the image, and symmetric to the vertical center line of the rectangular image.

14. The method of claim 13, wherein the distance of the second image region from the top image margin is selected to be greater than the distance of the second image region from the bottom image margin.

15. The method of claim 11, wherein the second image region is selected to be near the center of the image, and symmetric to the vertical center line of the rectangular image.

16. The method of claim 11, wherein the distance of the second image region from the top image margin is selected to be greater than the distance of the second image region from the bottom image margin.

17. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement;

determining the displacement by preferentially using the at least one image region of the plurality of image regions;

correcting a first image of the image sequence by determining the displacement from one of (i) first image data of the first image of the image sequence and second image data of a second image of the image, and (ii) the first image data of the first image of the image sequence and starting image data of a starting image sequence; and

determining a regional displacement of the at least one image region of the plurality of the image regions is determined from one of (i) the first image data of the first image and the second image inside the at least one image region, and (ii) the first image data of the first image and the second starting image inside the at least one image region;

wherein the determining of the displacement uses the regional displacement of the at least one image region of the plurality of image regions as the displacement;

wherein at least one of at least one first image region and at least one second image region is used as a function of a reliability factor for determining the regional displacement to determine the displacement; and

wherein determined regional displacements and reliability factors of the determined regional displacements for two image regions are determined by;

determining a regional displacement and a correlation quotient for each of the two image regions;

determining a threshold value function as a function of the regional displacement of each of the two image regions;

comparing a determined correlation quotient of one of the two image regions to a threshold value resulting from the threshold value function for a regional displacement of another of the two image regions; and

determining the reliability factor of the determined regional displacement of each of the two image regions to be sufficient when the determined correlation quotient corresponding to the one of the two image regions is greater than the threshold value of the threshold value function compared to the determined correlation quotient.

18. The method of claim 17, wherein the determined correlation quotient of one of the two image regions is determined by:

using block-matching to determine displacement correlation values for a plurality of possible regional displacements;

determining a regional displacement having a maximum displacement correlation value to be the regional displacement of the image regions; and

determining the correlation quotient by dividing the maximum displacement correlation value by a mean of determined displacement correlation values.

19. The method of claim 17, wherein:

the threshold value function assumes a value of a preselected second threshold value by for each arbitrary regional displacement less than a preselected first threshold value;

the threshold value function assumes a value of the preselected second threshold value minus a product for each arbitrary regional displacement greater than the preselected first threshold value;

the product includes a preselected slope parameter and a difference as factors; and

the difference being formed from the arbitrary regional displacement and the preselected first threshold value.

20. The method of claim 17, wherein the two image regions include the two first image regions.

21. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement;

determining the displacement by preferentially using the at least one image region of the plurality of image regions;

correcting a first image of the image sequence by determining the displacement from one of (i) first image data of the first image of the image sequence and second image data of a second image of the image, and (ii) the first image data of the first image of the image sequence and starting image data of a starting image sequence; and

determining a regional displacement of the at least one image region of the plurality of the image regions is determined from one of (i) the first image data of the first image and the second image inside the at least one image region, and (ii) the first image data of the first image and the second starting image inside the at least one image region, wherein the determining of the displacement uses the regional displacement of the at least one image region of the plurality of image regions as the displacement;

wherein at least one of at least one first image region and at least one second image region is used as a function of a reliability factor for determining the regional displacement to determine the displacement;

wherein for each of the image regions, the regional displacement is determined using block-displacement information from a block-based encoding process by considering the block-displacement information of blocks lying inside a specific image region; and

wherein for each of the image regions, the regional displacement, including a horizontal component and a vertical component, and the reliability factor of a determined regional displacement are determined by;

determining the horizontal component of the regional displacement by generating a first frequency distribution for frequencies of different values of the horizontal component of the block-displacement information, the horizontal component of the regional displacement corresponding to a value of the horizontal component of the block-displacement information, at which the first frequency distribution assumes its principal maximum;

determining the vertical component of the regional displacement by generating a second frequency distribution for frequencies of different values of the vertical component of the block-displacement information, the vertical component of the regional displacement corresponding to a value of the vertical component of the block-displacement information, at which the second frequency distribution assumes its principal maximum;

determining the reliability factor of the determined regional displacement to be sufficient when all of the following conditions are satisfied;

an absolute value of a difference in position of values of the horizontal component of the block-displacement information, corresponding to the principal maximum and a secondary maximum of the first frequency distribution, is less than a preselected first difference threshold;

the absolute value of a difference in position of values of the vertical component of the block-displacement information, corresponding to the principal maximum and a secondary maximum of the second frequency distribution, is less than a preselected second difference threshold;

the principal maximum of the first frequency distribution is greater than a first frequency threshold; and

the principal maximum of the second frequency distribution is greater than a second frequency threshold.

22. A device for determining a displacement of images in an image sequence, the device comprising:

a displacement-detection arrangement to perform a process for determining the displacement of images in the image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, the process including;

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement; and

determining the displacement by preferentially using the at least one image region of the plurality of image regions.

23. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, wherein the method is used for a head-shoulder recording situation, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement;

determining the displacement by preferentially using the at least one image region of the plurality of image regions;

selecting the two first image regions to be in a lateral region to the left and right of a vertical center line of a predetermined rectangular image;

selecting a distance of the two first image regions from a bottom image margin to be greater than another distance of the two first image regions from a top image margin; and

selecting the second image region to be near a center of the image;

wherein;

two first image regions and one second image region are available for image correction; and

the displacement is determined from one of the following, in a descending priority order;

a mean of regional displacements of the two first image regions, when a reliability of the regional displacement determinations for the two first image regions is considered sufficient;

a regional displacement of one of the two first image regions, when a reliability of the regional displacement determination is considered sufficient; and

a regional displacement of the second image region.

24. A method for determining a displacement of images in an image sequence for compensating for a camera movement, a plurality of image regions of the images being available for determining the displacement, each of the plurality of image regions being provided at a preselected position of the images, and each of the plurality of image regions having preselected dimensions, in which there are predetermined numbers of pixels in various directions of the image, and for separating an image movement mainly caused by a camera movement from an additional movement, which is superimposed on the image movement in subregions of an image to be corrected, the method comprising:

determining a probability of the image movement occurring without the additional movement at different positions of an image;

specifying and fixing, for each of the plurality of image regions, a position and dimensions as a function of the probability of the image movement occurring inside at least one image region of the plurality of image regions without the additional movement; and

determining the displacement by preferentially using the at least one image region of the plurality of image regions;

wherein at least one of at least one first image region and at least one second image region is used as a function of a reliability factor for determining the regional displacement to determine the displacement; and

wherein for each of the image regions, the regional displacement is determined using block-displacement information from a block-based encoding process by considering the block-displacement information of blocks lying inside a specific image region.