Fast rotation estimation of objects in sequences of acquired digital images

• US 8,587,665 B2
• Filed: 02/15/2011
• Issued: 11/19/2013
• Est. Priority Date: 02/15/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia, Inc.)
• Inventors: Albu, Felix, Murray, Larry, Stec, Piotr, Raducan, Ilariu
• Primary Examiner(s): CHEN, CHIA WEI A
• Application Number:
  US13/028,203
• Publication Number:
  US 20120206617A1
• Time in Patent Office: 1,008 Days
• Field of Search: 348 36- 39, 348169-172, 348135-142, 382190-208, 382/107, 382/276, 382286-299
• US Class Current: 348/169
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 10/243 by compensating for image skew or non-uniform image deformations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A measure of frame-to-frame rotation is determined. A global XY alignment of a pair of frames is performed. Local XY alignments in at least two matching corner regions of the pair of images are determined after the global XY alignment. Based on differences between the local XY alignments, a global rotation is determined between the pair of frames.

Claims

1. Within an image acquisition device, a method of detecting rotation from matching corner regions of acquired images, comprising:

acquiring a sequence of image frames; and

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame;

performing a further action based on the determined global rotation;

determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

2. The method of claim 1, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

3. The method of claim 1, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

4. Within an image acquisition device, a method of detecting rotation from matching corner regions of acquired images, comprising:

acquiring a sequence of image frames; and

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame;

performing a further action based on the determined global rotation;

determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

5. The method of claim 4, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

6. The method of claim 4, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

7. Within an image acquisition device, a method of detecting rotation from matching regions of acquired images, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences;

performing a further action based on the calculated global frame to frame rotation; and

the method further comprising

determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

8. The method of claim 7, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

9. Within an image acquisition device, a method of detecting rotation from matching regions of acquired images, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences;

performing a further action based on the calculated global frame to frame rotation;

determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

10. The method of claim 9, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

11. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of detecting rotation from matching corner regions of acquired images, wherein the method comprises;

acquiring a sequence of image frames; and

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame; and

performing a further action based on the determined global rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

12. The device of claim 11, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

13. The device of claim 11, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

14. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of detecting rotation from matching corner regions of acquired images, wherein the method comprises;

acquiring a sequence of image frames; and

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame;

performing a further action based on the determined global rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

15. The device of claim 14, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

16. The device of claim 14, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

17. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of detecting rotation from matching regions of acquired images, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences; and

performing a further action based on the calculated global frame to frame rotation;

wherein the method further comprises determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

18. The device of claim 17, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

19. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of detecting rotation from matching regions of acquired images, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences; and

performing a further action based on the calculated global frame to frame rotation

wherein the method further comprises determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

20. The device of claim 19, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

21. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of detecting rotation from matching corner regions of acquired images of a sequence of image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame; and

performing a further action based on the determined global rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

22. The one or more non-transitory processor-readable media of claim 21, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

23. The one or more non-transitory processor-readable media of claim 21, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

24. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of detecting rotation from matching corner regions of acquired images of a sequence of image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and a second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the first and second frames after the global XY alignment; and

based on the differences between said local XY alignments, determining a global rotation of the second frame relative to the first frame;

performing a further action based on the determined global rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

25. The one or more non-transitory processor-readable media of claim 24, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

26. The one or more non-transitory processor-readable media of claim 24, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and wherein the global rotation determining is based further on said delta-X and delta-Y differences.

27. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of detecting rotation from matching regions of acquired images of a sequence of image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences;

performing a further action based on the calculated global frame to frame rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vector gradients for each of the first frame and the second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

wherein the global rotation determining is further based on normalized distances between the primary maximum and minimum peaks.

28. The one or more non-transitory processor-readable media of claim 27, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

29. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of detecting rotation from matching regions of acquired images of a sequence of image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining local XY alignments in at least two matching regions of the aligned image pair;

determining delta-X and delta-Y differences between the at least two matching regions; and

calculating a global frame-to-frame rotation from said delta-X and delta-Y differences;

performing a further action based on the calculated global frame to frame rotation; and

wherein the method further comprises determining horizontal and vertical integral projection vectors for each of the first frame and the second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

wherein the global rotation determining is further based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences.

30. The one or more non-transitory processor-readable media of claim 29, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.