Object detection from image profiles within sequences of acquired digital images

• US 8,587,666 B2
• Filed: 02/15/2011
• Issued: 11/19/2013
• Est. Priority Date: 02/15/2011
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia, Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia, Inc.)
• Inventors: Albu, Felix, Murray, Larry, Stec, Piotr, Raducan, Ilariu
• Primary Examiner(s): CHEN, CHIA WEI A
• Application Number:
  US13/028,206
• Publication Number:
  US 20120206618A1
• Time in Patent Office: 1,008 Days
• Field of Search: 348 36- 39, 348169-172, 348135-142, 382190-208, 382/107, 382/276, 382286-299
• US Class Current: 348/169
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A measure of frame-to-frame rotation is determined. A global XY alignment of a pair of image frames is performed. At least one section of each of the X and Y integral projection vectors is determined, where aligned global vectors demonstrate a significant localized difference. Based on X and Y locations of the at least one section of the X and Y integral projection vectors, location, relative velocity and/or approximate area of at least one moving object within the sequence of image frames is/are determined.

Claims

1. Within an image acquisition device, a method of determining moving objects in integral projection vectors, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

2. The method of claim 1, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

3. The method of claim 1, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

4. The method of claim 1, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

5. Within an image acquisition device, a method of determining moving objects in integral projection vectors, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

6. The method of claim 5, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

7. The method of claim 5, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

8. The method of claim 5, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

9. Within an image acquisition device, a method of determining moving objects in integral projection vectors, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

10. The method of claim 9, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

11. The method of claim 9, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

12. The method of claim 9, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

13. Within an image acquisition device, a method of determining moving objects in integral projection vectors, comprising:

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

14. The method of claim 13, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

15. The method of claim 13, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

16. The method of claim 13, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

17. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

18. The device of claim 17, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

19. The device of claim 17, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

20. The device of claim 17, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

21. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

22. The device of claim 21, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

23. The device of claim 21, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

24. The device of claim 21, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

25. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

26. The device of claim 25, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

27. The device of claim 25, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

28. The device of claim 25, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

29. An image acquisition device, comprising:

a lens, shutter and image sensor for acquiring digital images;

a processor;

a memory having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors, wherein the method comprises;

acquiring a sequence of image frames;

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

30. The device of claim 29, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

31. The device of claim 29, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

32. The device of claim 29, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

33. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors for a sequence of acquired image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

34. The one or more non-transitory processor-readable media of claim 33, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

35. The one or more non-transitory processor-readable media of claim 33, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

36. The one or more non-transitory processor-readable media of claim 33, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

37. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors for a sequence of acquired image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global XY alignment of a first frame and second, following frame of said image sequence;

determining at least one section of each of the X and Y integral projection vectors where aligned global vectors demonstrate a significant localized difference; and

based on X and Y locations of said at least one section of the X and Y integral projection vectors, determining location, relative velocity or approximate area, or combinations thereof, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined location, relative velocity or approximate area, or combinations thereof, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

38. The one or more non-transitory processor-readable media of claim 37, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

39. The one or more non-transitory processor-readable media of claim 37, wherein the method further comprises determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is further based also on the determined global rotation.

40. The one or more non-transitory processor-readable media of claim 37, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

41. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors for a sequence of acquired image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

the method further comprises;

determining horizontal and vertical integral projection vector gradients for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

normalizing said integral projection vector gradients; and

determining locations of primary maximum and minimum peaks of said integral projection vector gradients; and

determining a global rotation based on normalized distances between the primary maximum and minimum peaks, and

wherein the further action is also based on said determined global rotation.

42. The one or more non-transitory processor-readable media of claim 41, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

43. The one or more non-transitory processor-readable media of claim 41, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

44. The one or more non-transitory processor-readable media of claim 41, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.

45. One or more non-transitory processor-readable media having code embedded therein for programming the processor to perform a method of determining moving objects in integral projection vectors for a sequence of acquired image frames, wherein the method comprises:

determining a measure of frame-to-frame rotation, including;

performing a global alignment of a first and second, following frame of said image sequence;

determining at least one section of an integral projection vector where the aligned global vectors demonstrate a significant localized difference; and

based on a location of each said at least one section of the integral projection vector, determining an approximate size or relative velocity, or both, of at least one moving object within the sequence of image frames; and

performing a further action based on the determined approximate size or relative velocity, or both, of the at least one moving object; and

wherein the method comprises;

determining horizontal and vertical integral projection vectors for each of the same or a different first frame and the same or a different second, following, image frame in said sequence;

successively convoluting a first integral projection vector with a variable length vector;

convoluting a second integral projection vector with a fixed length vector; and

determining an absolute sum of differences between filtered vectors based on the convoluting of the first and second integral projection vectors; and

determining a global rotation based on a length of a variable vector N that leads to a minimum value of the absolute sum of the differences, and

wherein the further action is also based on said determined global rotation.

46. The one or more non-transitory processor-readable media of claim 45, wherein the further action comprises joining adjacent frames of a panorama image;

predicting a location of a tracked object in a subsequent frame;

providing an image with reduced blur than at least one of the first and second frames;

or providing a more stable video sequence by removing one or more undesired motion artifacts, or combinations thereof.

47. The one or more non-transitory processor-readable media of claim 45, further comprising determining delta-X and delta-Y differences between the at least two matching corner regions;

and determining a global rotation based on said delta-X and delta-Y differences, and wherein the further action is also based on the determined global rotation.

48. The one or more non-transitory processor-readable media of claim 45, further comprising performing a global XY alignment of the same or a different first frame and the same or a different second, following frame of said sequence of image frames;

determining local XY alignments in at least two matching corner regions of the aligned image pair; and

based on the differences between said local XY alignments, determining a global rotation of the second image frame relative to the first; and

wherein the further action is also based on the determined global rotation.