Method, device and computer program product for image stabilization using color matching

• US 20040061786A1
• Filed: 09/24/2003
• Published: 04/01/2004
• Est. Priority Date: 09/07/2001
• Status: Active Grant

• Current Assignee: Intergraph Software Technologies Company
• Original Assignee: Intergraph Software Technologies Company
• Inventors: Grindstaff, Gene Arthur, McKay, Therman Ward III, Fletcher, Susan Heath Calvin
• Granted Patent: US 7,436,437 B2
• Time in Patent Office: Days
• US Class Current: 348/208.6
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/20 Image preprocessing
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

A method for image stabilization of at least two digital images is disclosed. In such an embodiment, a first digital image having a plurality of pixels and at least a second image having a plurality of pixels are provided. Each pixel has an associated address for display and is representative of a color. Either a user of the system provides a color match range or the system provides a predetermined color matched range. A pixel is selected within the first digital image. Preferably the pixel is one that represents an item within the image that is either blurry due to movement of the item or appears jittery due to camera movement. Due to the vast number of available colors a pixel selected in a first image can be matched to a pixel in a second image within a range. The range allows for compensation due to lighting changes. Once the pixel in the first image is selected it can be compared to all pixels within the second image. Each pixel within the second image that is within the color match range is saved and the closest pixel color to the pixel from the first image is selected. The address of the pixels in the second image are then readdressed such that the address of the pixel located in the second image that has the closest color to the pixel in the first image now has the same display address as that of the pixel in the first image. The repositioned second digital image is then stored to memory.

Claims

1. A method of processing a motion picture having first and second frames, each frame having a plurality of pixels, the method comprising:

selecting a first pixel in the first frame having a given color;

locating a second pixel in the second frame having the given color;

aligning the first frame and the second frame based upon the locations of the first and second pixels.

2. The method as defined by claim 1 wherein the first and second frames are aligned in memory.

3. The method as defined by claim 1 wherein aligning comprises aligning the first frame and second frame about the first and second pixels.

4. The method as defined by claim 1 further comprising determining the color of the first pixel, wherein the color is determined to be the given color.

5. The method as defined by claim 1 wherein locating comprises determining if the color of the second pixel is within a given color tolerance.

6. The method as defined by claim 1 wherein the first pixel has a first address, further wherein aligning includes rearranging the second pixel to have an address that equals the first address.

7. The method as defined by claim 1 wherein the first frame and the second frame are linearly averaged to provide a third frame.

8. The method as defined by claim 7, wherein the third frame has a higher resolution than the first or second frame.

9. The method as defined by claim 7, wherein the third frame is displayed on a display device.

10. The method according to claim 7 wherein the first and second frames are aligned to a sub-pixel level.

11. The method according to claim 1 further comprising:

selecting a region of the second frame for locating a second pixel having the given color within that region.

12. The method according to claim 11, further comprising:

if a second pixel is not found within the second frame having the given color, aligning the first frame and the second frame based upon a predetermined formula.

13. The method according to claim 12 wherein the predetermined formula is based upon past alignments of frames.

14. A method of processing a motion picture having a plurality of frames, each frame having a plurality of pixels, the method comprising:

selecting an initial frame and a final frame, at least one frame being between the initial and final frames;

selecting an initial pixel in the initial frame and a final pixel in the final frame, the initial pixel having an initial color, the final pixel having a final color;

determining an interpolated pixel for each frame between the initial and final frame based on the initial color and the final color; and

aligning the initial frame, final frame, and the at least one frame between the initial and final frames, the frames being aligned based upon the locations of the initial pixel, final pixel, and each interpolated pixel.

15. The method as defined by claim 14 wherein the interpolation is based upon the initial color, the final color, and the total number of frames between the initial and final frames.

16. The method as defined by claim 14 further comprising locating the interpolated pixel in a given frame between the initial and final frames.

17. The method as defined by claim 16 wherein each interpolated pixel has an interpolated color, further wherein the interpolated pixel in the given frame is located based upon the interpolated color of the interpolated pixel.

18. The method as defined by claim 17 wherein locating comprises determining if the interpolated color is within a given color tolerance.

19. The method as defined by claim 14 wherein the initial frame, final frame, and the at least one frame are aligned in memory.

20. The method as defined by claim 14 wherein aligning comprises setting the address of the initial pixel, final pixel and interpolated pixel to the same value.

21. The method according to claim 14 further comprising displaying at least the initial frame, final frame, and the at least one frame between the initial and final frames on a display.

22. A method for image stabilization of a digital sequence of images, wherein each image has a plurality of pixels and wherein each pixel has an associated address for display and is representative of a color, the method comprising:

selecting a pixel in the beginning image and a pixel in the ending image;

interpolating between the colors associated with the selected pixels from the beginning image and the ending image;

in the step of interpolating determining a color value for each image between the beginning image and the ending image;

locating within each image a pixel that is within the range of the determined color; and

readdressing each image so that the located pixels are aligned at the same display address.

23. A method for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the method comprising:

receiving a signal representative of a color of a pixel in a first image;

identifying a pixel in a second image having the same color within a predetermined range of colors wherein the range may include a single color; and

readdressing the second image so that the pixel in the second image resides at the same address as the pixel in the first image.

24. A method for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the method comprising:

interpolating between a color of a pixel in a first image and a color of a pixel in a third image to determine a searchable color;

identifying a pixel having a color in a second image within a range of the searchable color; and

repositioning the images such that the pixels from the first second and third images are aligned.

25. A system for image stabilization of a video sequence having a plurality of video frames, the system comprising:

a repositioning module for readdressing a second video image so that a pixel of the second video image having a color within a range of colors of a color of a pixel in the first image is readdressed to the address of the pixel in the first image.

26. The system according to claim 25, wherein all of the pixels in the second video image are readdressed.

27. The system according to claim 26, wherein the pixels are readdressed in relation to other pixels in the video frame.

28. A computer program product for use with a computer system, the computer program product containing computer code for processing a motion picture having first and second frames, each frame having a plurality of pixels, the computer code comprising:

computer code for selecting a first pixel in the first frame having a given color;

computer code for locating a second pixel in the second frame having the given color;

computer code for aligning the first frame and the second frame based upon the locations of the first and second pixels.

29. The computer program product according to claim 28 wherein the first and second frames are aligned in memory.

30. The computer program product as defined by claim 28 wherein the computer code for aligning comprises aligning the first frame and second frame about the first and second pixels.

31. The computer program product as defined by claim 28 further comprising computer code for determining the color of the first pixel, wherein the color is determined to be the given color.

32. The computer program product as defined by claim 28 wherein the computer code for locating includes computer code for determining if the color of the second pixel is within a given color tolerance.

33. The computer program product as defined by claim 28 wherein the first pixel has a first address, further wherein the computer code for aligning includes computer code for rearranging the second pixel to have an address that equals the first address.

34. The computer program product as defined by claim 28 further including computer code to linearly average the first frame and the second frame to provide a third frame.

35. The computer program product as defined by claim 34, wherein the third frame has a higher resolution than the first or second frame.

36. The computer program product as defined by claim 35, further comprising computer code for displaying the third frame on a display device.

37. The computer program product according to claim 28 further comprising:

computer code for selecting a region of the second frame for locating a second pixel having the given color within that region.

38. The computer program product according to claim 37, further comprising:

computer code for aligning the first frame and the second frame based upon a predetermined formula if a second pixel is not found within the second frame having the given color.

39. A computer program product for use on a computer system, the computer program product having computer readable code thereon for processing a motion picture having a plurality of frames, each frame having a plurality of pixels, the computer code comprising:

computer code for selecting an initial frame and a final frame, at least one frame being between the initial and final frames;

computer code for selecting an initial pixel in the initial frame and a final pixel in the final frame, the initial pixel having an initial color, the final pixel having a final color;

computer code for determining an interpolated pixel for each frame between the initial and final frame based on the initial color and the final color; and

computer code for aligning the initial frame, final frame, and the at least one frame between the initial and final frames, the frames being aligned based upon the locations of the initial pixel, final pixel, and each interpolated pixel.

40. The computer program product as defined by claim 39 wherein the interpolation is based upon the initial color, the final color, and the total number of frames between the initial and final frames.

41. The computer program product as defined by claim 39 further comprising computer code for locating the interpolated pixel in a given frame between the initial and final frames.

42. The computer program product as defined by claim 41 wherein each interpolated pixel has an interpolated color, further wherein the interpolated pixel in the given frame is located based upon the interpolated color of the interpolated pixel.

43. The computer program product as defined by claim 42 wherein the computer code for locating includes computer code for determining if the interpolated color is within a given color tolerance.

44. The computer program product as defined by claim 39 including computer code for aligning the initial frame, final frame, and the at least one frame in memory.

45. The computer program product as defined by claim 39 wherein the computer code for aligning includes computer code for setting the address of the initial pixel, final pixel and interpolated pixel to the same value.

46. The computer program product according to claim 39 further comprising computer code for displaying at least the initial frame, final frame, and the at least one frame between the initial and final frames on a display.

47. A computer program product for use with a computer system, the computer program product including computer readable code for image stabilization of a digital sequence of images, wherein each image has a plurality of pixels and wherein each pixel has an associated address for display and is representative of a color, the computer readable code comprising:

computer code for selecting a pixel in the beginning image and a pixel in the ending image;

computer code for interpolating between the colors associated with the selected pixels from the beginning image and the ending image;

the computer code for interpolating including computer code for determining a color value for each image between the beginning image and the ending image;

computer code for locating within each image a pixel that is within the range of the determined color; and

computer code for readdressing each image so that the located pixels are aligned at the same display address.

48. A computer program product for use with a computer system, the computer program product including computer readable code for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the computer readable code comprising:

computer code for receiving a signal representative of a color of a pixel in a first image;

computer code for identifying a pixel in a second image having the same color within a predetermined range of colors wherein the range may include a single color; and

computer code for readdressing the second image so that the pixel in the second image resides at the same address as the pixel in the first image.

49. A computer program product for use with a computer system, the computer program product including computer readable code for image stabilization of a sequence of temporally displayed digital video images, wherein each image contains a number of pixels and each pixel has an associated display position, the computer readable code comprising:

computer code for interpolating between a color of a pixel in a first image and a color of a pixel in a third image to determine a searchable color;

computer code for identifying a pixel having a color in a second image within a range of the searchable color; and

computer code for repositioning the images such that the pixels from the first second and third images are aligned.

50. A computer program product for use with a computer system, the computer program product including computer readable code for image stabilization of a video sequence having a plurality of video frames, the computer readable code comprising:

computer code for readdressing a second video image so that a pixel of the second video image having a color within a range of colors of a color of a pixel in the first image is readdressed to the address of the pixel in the first image.

51. The computer program product according to claim 50, wherein in the computer code for readdressing, all of the pixels in the second video image are readdressed.

52. The computer program product according to claim 51, wherein in the computer code for readdressing, the pixels are readdressed in relation to other pixels in the video frame.