SYSTEM, METHOD, DEVICE, AND COMPUTER PROGRAM PRODUCT FOR PROVIDING IMAGE CORRECTION

• US 20080144964A1
• Filed: 12/18/2006
• Published: 06/19/2008
• Est. Priority Date: 12/18/2006
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Nokia Corporation
• Inventors: Friman, Jonni, Hyyppa, Henrick, Soinio, Asmo, Vermola, Larri
• Granted Patent: US 7,936,945 B2
• Time in Patent Office: Days
• US Class Current: 382/275
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

A system, method, device, and computer program product are provided for correcting motion skew in a frame of video or in other images captured using a rolling shutter image sensor. A method for removing motion skew may create a mathematical model of the motion skew, compute the motion skew using the mathematical model of the motion skew, remove the distorted image of the object from view in the image using the computed motion skew, and render a corrected image of the object in the image using the computed motion skew.

Claims

1. A method for removing motion skew from an image that otherwise causes a distorted image of the object, the method comprising:

computing a representation of the distorted object image using a mathematical model of the motion skew, the representation of the distorted object image providing an approximate shape and location within the image of the distorted image of the object;

removing the distorted image of the object from view in the image using the computed representation of the distorted object image; and

rendering a corrected image of the object in the image using the computed representation of the distorted object image.

2. The method of claim 1, further comprising:

creating the mathematical model of the motion skew.

3. The method of claim 2, wherein creating the mathematical model of the motion skew comprises:

creating a mathematical model of the approximate motion between the object and the rolling shutter image sensor at the time that the image was captured; and

using the mathematical model of the approximate motion of the object to create the mathematical model of the motion skew.

4. The method of claim 3, wherein the mathematical model of the motion skew is a function of velocity of the object relative to the rolling shutter image sensor during exposure of the object, and rate of a rolling shutter action of the rolling shutter image sensor.

5. The method of claim 1 further comprising:

detecting a distorted image of the object in the image.

6. The method of claim 5, wherein detecting the distorted image of the object further comprises:

analyzing the detected distorted image; and

gathering information from the distorted image; and

wherein the computing of the representation of the distorted object image using the mathematical model comprises;

using the gathered information as values for parameters in the mathematical model.

7. The method of claim 5, wherein the image comprises a frame in a video, and wherein detecting the distorted image of the object further comprises:

detecting at least a portion of the distorted object in the frame in the video;

detecting at least a portion of the distorted object in a different frame in the video; and

computing the velocity of the moving object by comparing the detected object portions of the two frames;

and wherein the computing of the representation of the distorted object image using the mathematical model comprises;

using the computed velocity of the moving object as a value of a parameter in the mathematical model.

8. The method of claim 1, wherein the removing of the distorted image of the object further comprises:

generating a mask for removing from view the distorted image of the object.

9. The method of claim 8, wherein generating a mask for removing from view the distorted image of the object comprises using the computed representation of the distorted object image to create a mask for removing the distorted image of the object from view in the image.

10. The method of claim 1, wherein the image comprises a frame in a video, the method further comprising:

replacing pixels in an area of the image where distorted image of the object has been removed from view with pixels gathered from a corresponding area of a different frame in the video.

11. The method of claim 1, wherein the rendering of a corrected image of the object further comprises:

using the mathematical model of the motion skew to translate pixels of the distorted image of the object to locations in the image that form the corrected image of the object.

12. The method of claim 1, wherein the rendering of a corrected image of the object comprises rendering the corrected image of the object to form an image of the object as the object generally appeared at a time proximate when the object was first exposed by a rolling shutter image sensor when capturing the image.

13. The method of claim 1, wherein the image of the object is captured by a rolling shutter image sensor embodied in a device configured to capture video, and wherein the image comprises a frame in a video.

14. The method of claim 1, wherein the object comprises a generally straight elongate object having a first end and a second end, wherein the object is generally moving about the first end relative to a rolling shutter image sensor during exposure of the object by the image sensor, wherein the object generally moves from a first angle (α

) to a second angle (β

) during the exposure of the object by the image sensor, wherein the creating of the mathematical model comprises;

providing x- and y-coordinate axes to the image so that the y-axis is perpendicular to rows of pixels sequentially exposed by the rolling shutter image sensor;

standardizing the x- and y-coordinates so that a Euclidean distance between the first end and the second end of the object has a value of 1;

selecting a point in the image proximate the first end of the object; and

assuming that this point is the origin of the coordinate axes;

wherein the mathematical model of the motion skew is configured to be used to generate a representation of the distorted elongate object image, the x-coordinate of which (x_image) can be represented as a mathematical function of the y-coordinate (y) as y advances from 0 to y_max, where y_max is the y-coordinate of the second end of the object when the object is at angle β

, the mathematical function comprising;

x_image=y tan(α

(y_max−

y)/y_max+β

y/y_max), where the angle α

can be determined by the function α

=β

−

(θ

_club t_shutter height_clubβ

), where θ

_club is the angular velocity of the object during the exposure of the object, where t_shutter is the time taken by the rolling shutter image sensor to expose each row of pixels before exposing a next row of pixels, where height_clubβ

is the height of the object in pixels in the y-direction when the object is at angle β

, and where the coordinates are divided by the length.

15. A computer program product for removing motion skew from an image that otherwise causes a distorted image of the object, the computer program product comprising at least one computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

a first executable portion for computing a representation of the distorted object image based on a mathematical model of the motion skew, the representation of the distorted object image providing an approximate shape and location within the image of the distorted image of the object;

a second executable portion for removing the distorted image of the object from view in the image using the computed representation of the distorted object image; and

a third executable portion for rendering a corrected image of the object in the image using the computed representation of the distorted object image.

16. The computer program product of claim 15, the computer-readable program code portions comprising:

a fourth executable portion for generating a mathematical model of the motion skew.

17. The computer program product of claim 15, wherein the mathematical model of the motion skew is a function of velocity of the object relative to the rolling shutter image sensor during exposure of the object, and rate of a rolling shutter action of the rolling shutter image sensor.

18. The computer program product of claim 15, the computer-readable program code portions comprising:

a fourth executable portion for detecting the distorted image of the object in the image.

19. The computer program product of claim 18, wherein the fourth executable portion further comprises:

a fifth executable portion for analyzing the detected distorted image and gathering information from the distorted image;

and wherein the first executable portion comprises;

a sixth executable portion for computing the representation of the distorted object image from the mathematical model of the motion skew using the gathered information.

20. The computer program product of claim 18, wherein the image comprises a frame in a video, and wherein the fourth executable portion further comprises:

a fifth executable portion for detecting at least a portion of the distorted object in the frame in the video;

detecting at least a portion of the distorted object in a different frame in the video; and

computing the velocity of the moving object by comparing the detected object portions of the two frames;

and wherein the first executable portion further comprises;

a sixth executable portion for computing the representation of the distorted object image from the mathematical model of the motion skew using the computed velocity of the moving object.

21. The computer program product of claim 15, wherein the second executable portion further comprises:

a fourth executable portion for creating a mask to remove the distorted image of the object from view in the image based at least partially upon the computed representation of the distorted object image.

22. The computer program product of claim 21, wherein the image comprises a frame in a video having a plurality of frames, and wherein the second executable portion further comprises:

a fourth executable portion for replacing pixels in an area of the image comprising the removed distorted image of the object with pixels gathered from a corresponding area of a different frame in the video.

23. The computer program product of claim 15, wherein the third executable portion further comprises:

a fourth executable portion for translating pixels of the distorted image of the object to locations in the image that form the corrected image of the object using the representation of the distorted object image computed by the first executable portion.

24. A device configured to correct motion skew in an image that includes a skewed object image, the device comprising:

a processor configured to;

compute a representation of the skewed object image based on a mathematical model of the motion skew, the representation of the skewed object image providing an approximate shape and location within the image of the skewed object image;

remove the skewed object image from view in the image using the computed representation of the skewed object image; and

render a corrected object image in the image using the representation of the skewed object image.

25. The device of claim 24, further comprising:

a rolling shutter image sensor for capturing the image; and

a display for displaying the captured image.

26. The device of claim 24, wherein the processor is further configured to geometrically translate pixels that comprise the skewed object image based on the computed representation of the skewed object image.

27. The device of claim 25, wherein the mathematical model comprises:

a mathematical function of the representation of the skewed object image as a function of the object'"'"'s velocity during the exposure of the object and the rolling shutter image sensor'"'"'s rate of rolling shutter action.

28. The device of claim 27, wherein the image comprises one frame in a video having a plurality of frames, and wherein the processor is further configured to calculate at least an approximation of the velocity of the object by comparing the current frame with a different frame in the video.

29. The device of claim 24, wherein the image comprises one frame in a video having a plurality of frames, and wherein the processor is further configured to copy pixels from a different frame of the video in order to replace the pixels in the current frame of the distorted object image.

30. A method for removing distortion from a video of a swinging golf club, wherein the distortion comprises a distorted golf club image, the method comprising:

generating a mathematical model of the distortion in the golf club;

removing the distorted golf club image from a frame of the video; and

rendering a corrected golf club image in the frame based on the mathematical model of the distortion.

31. The method of claim 30, wherein the rendering of a corrected golf club comprises:

geometrically translating the pixels that comprise the distorted golf club image based on the mathematical model of the distortion in the golf club.

32. The method of claim 30, wherein generating a mathematical model comprises:

generating a mathematical function of a representation of the distorted golf club image as a function of the golf club'"'"'s angular velocity while rotating about a point during the exposure of the golf club and the rolling shutter image sensor'"'"'s rate of rolling shutter action.

33. The method of claim 32, wherein the mathematical function of the distorted golf club image comprises:

x_image=y tan(α

(y_max−

y)/y_max+β

y/y_max), where the x- and y-coordinates are based on x- and y-axes, where the y-axis is perpendicular to rows of pixels sequentially exposed by the rolling shutter image sensor, where the x-axis is perpendicular to the y-axis, where the coordinates are standardized so that the Euclidean distance between a first end and a second end of the golf club has a value of 1, where the origin of the axis is located at the first end of the golf club which comprises the point about which the golf club rotates, where x_image is the x-coordinate of the representation of the skewed golf club image, where y_max is the y-coordinate of the second end of the golf club when the object is at angle β

, where the angle α

can be determined by the function α

=β

−

(θ

club t_shutter height_clubβ

), where θ

_club is the angular velocity of the golf club during the exposure of the golf club, where t_shutter is the time taken by the rolling shutter image sensor to expose each row of pixels before exposing a next row of pixels, and where the height_clubβ

is the height of the golf club image in pixels in the y-direction when the object is at angle β

.

34. A device configured to remove distortion from an object image in a video of a moving object, the device comprising:

means for generating a mathematical model of the distortion in the object image;

means for removing the distorted object image from a frame of the video; and

means for rendering a corrected object image in the frame based on the mathematical model of the distortion.

35. The device of claim 34, further comprising:

means for determining an approximation of a velocity of the moving object in the video; and

means for generating the mathematical model of the distortion in the object image based at least partially on the approximation of the velocity .