IMAGE DISTORTION CORRECTION METHOD AND IMAGE DISTORTION CORRECTION DEVICE USING THE SAME

• US 20150063685A1
• Filed: 02/07/2014
• Published: 03/05/2015
• Est. Priority Date: 08/30/2013
• Status: Active Grant

• Current Assignee: National Central University
• Original Assignee: National Central University
• Inventors: CHEN, CHING-HAN, YAO, TUN-KAI
• Granted Patent: US 9,165,219 B2
• Time in Patent Office: Days
• US Class Current: 382/156
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/80 Geometric correction

Abstract

An image distortion correction method and an image distortion correction device are provided. The image distortion correction method uses a neural network model to perform a correcting operation on an original image so as to obtain a correction image with plural correction points. Firstly, a position coordinate of the correction point is inputted into the neural network model, so that a first direction coordinate correction amount is outputted from the neural network model. Then, the position coordinate of the correction point is inputted into the neural network model, so that a second direction coordinate correction amount is outputted from the neural network model. Afterwards, a pixel value of the original image corresponding to the first direction coordinate correction amount and the second direction coordinate correction amount is used as a pixel value of the correction point.

Claims

1. An image distortion correction method, comprising steps of:

(1) providing a standard picture and a target image, wherein the standard picture has plural standard points, and the target image has plural target points corresponding to the plural standard points;

(2) providing a position coordinate of each standard point and a first direction coordinate change amount which is between each standard point and the corresponding target point to the neural network model so as to train and obtain a first direction parameter group of the neural network model, and providing the position coordinate of each standard point and a second direction coordinate change amount which is between each standard point and the corresponding target point to the neural network model so as to train and obtain a second direction parameter group of the neural network model; and

(3) performing a correcting process to correct distortion of an original image by the neural network model according to the first direction parameter group and the second direction parameter group of the neural network model, so that a correction image is obtained.

2. The image distortion correction method according to claim 1, wherein in the step (1), the target image is obtained by using an image pickup device to shoot the standard picture, wherein in the step (3), the original image is captured by the image pickup device.

3. The image distortion correction method according to claim 1, wherein the correction image comprises at least one correction point, and the step (3) comprises sub-steps of:

(3-1) inputting a position coordinate of the at least one correction point into the neural network model, so that a first direction coordinate correction amount is outputted from the neural network model according to the first direction parameter group, wherein after the first direction coordinate correction amount and the position coordinate of the at least one correction point are added together, a first direction sum coordinate is obtained;

(3-2) inputting the position coordinate of the at least one correction point into the neural network model, so that a second direction coordinate correction amount is outputted from the neural network model according to the second direction parameter group, wherein after the second direction coordinate correction amount and the position coordinate of the at least one correction point are added together, a second direction sum coordinate is obtained; and

(3-3) using a pixel value of the original image corresponding to the first direction sum coordinate and the second direction sum coordinate as a pixel value of the at least one correction point.

4. The image distortion correction method according to claim 3, wherein in the step (3-3), the pixel value of the original image corresponding to the first direction sum coordinate and the second direction sum coordinate is obtained by interpolating pixel values of plural neighboring points around the first direction sum coordinate and the second direction sum coordinate of the original image.

5. The image distortion correction method according to claim 3,

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are a horizontal direction coordinate correction amount and a vertical direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are a horizontal direction sum coordinate and a vertical direction sum coordinate, respectively;

or

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are the vertical direction coordinate correction amount and the horizontal direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are the vertical direction sum coordinate and the horizontal direction sum coordinate, respectively.

6. The image distortion correction method according to claim 1, wherein the neural network model comprises an input layer, a hidden layer and an output layer, and the hidden layer comprises plural neurons, wherein a transfer function of each neuron of the hidden layer is a non-linear function, the first direction parameter group comprises a first horizontal direction coordinate weight, a first vertical direction coordinate weight and a first direction bias value corresponding to at least one neuron of the hidden layer, and the second direction parameter group comprises a second horizontal direction coordinate weight, a second vertical direction coordinate weight and a second direction bias value corresponding to the at least one neuron of the hidden layer.

7. The image distortion correction method according to claim 6, wherein the hidden layer comprises four neurons.

8. An image distortion correction method for using a neural network model to perform a correcting operation on an original image so as to obtain a correction image with at least one correction point, the image distortion correction method comprising steps of:

(1) inputting a position coordinate of the at least one correction point into the neural network model so as to allow the neural network model to output a first direction correction point parameter, and inputting the position coordinate of the at least one correction point into the neural network model so as to allow the neural network model to output a second direction correction point parameter; and

(2) using a pixel value of the original image corresponding to the first direction correction point parameter and the second direction correction point parameter as a pixel value of the at least one correction point.

9. The image distortion correction method according to claim 8, wherein the first direction correction point parameter is a first direction coordinate correction amount, and the second direction correction point parameter is a second direction coordinate correction amount.

10. The image distortion correction method according to claim 9, wherein a first direction sum coordinate is obtained after the first direction coordinate correction amount and the position coordinate of the at least one correction point are added together, and a second direction sum coordinate is obtained after the second direction coordinate correction amount and the position coordinate of the at least one correction point are added together, wherein in the step (2), a pixel value of the original image corresponding to the first direction sum coordinate and the second direction sum coordinate is used as the pixel value of the at least one correction point.

11. The image distortion correction method according to claim 10, wherein in the step (2), the pixel value of the original image corresponding to the first direction sum coordinate and the second direction sum coordinate is obtained by interpolating pixel values of plural neighboring points around the first direction sum coordinate and the second direction sum coordinate of the original image.

12. The image distortion correction method according to claim 10,

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are a horizontal direction coordinate correction amount and a vertical direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are a horizontal direction sum coordinate and a vertical direction sum coordinate, respectively;

or

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are the vertical direction coordinate correction amount and the horizontal direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are the vertical direction sum coordinate and the horizontal direction sum coordinate, respectively.

13. The image distortion correction method according to claim 8, wherein the neural network model comprises an input layer, a hidden layer and an output layer, and the hidden layer comprises plural neurons, wherein a transfer function of each neuron of the hidden layer is a non-linear function, wherein the first direction correction point parameter is outputted from the neural network model at least according to a first direction parameter group corresponding to at least one neuron of the hidden layer, and the second direction correction point parameter is outputted from the neural network model at least according to a second direction parameter group corresponding to the at least one neuron of the hidden layer, wherein the first direction parameter group comprises a first horizontal direction coordinate weight, a first vertical direction coordinate weight and a first direction bias value, and the second direction parameter group comprises a second horizontal direction coordinate weight, a second vertical direction coordinate weight and a second direction bias value.

14. The image distortion correction method according to claim 13, wherein the hidden layer comprises four neurons.

15. An image distortion correction method for using a neural network model to perform a correcting operation on an original image so as to obtain a correction image with at least one correction point, the image distortion correction method comprising steps of:

(1) inputting a position coordinate of the at least one correction point into the neural network model so as to allow the neural network model to output a coordinate correction amount, and performing an arithmetic operation on the coordinate correction amount and the position coordinate of the at least one correction point so as to obtain an arithmetic coordinate; and

(2) using a pixel value of the original image corresponding to the arithmetic coordinate as a pixel value of the at least one correction point.

16. The image distortion correction method according to claim 15, wherein in the step (2), the pixel value of the original image corresponding to the arithmetic coordinate is obtained by interpolating pixel values of plural neighboring points around the arithmetic coordinate of the original image.

17. The image distortion correction method according to claim 15, wherein the arithmetic operation is an addition operation, and the arithmetic coordinate is a sum coordinate.

18. The image distortion correction method according to claim 17, wherein the step (1) comprises a sub-step (1-1) of inputting the position coordinate of the at least one correction point into the neural network model so as to allow the neural network model to output a first direction coordinate correction amount, and inputting the position coordinate of the at least one correction point into the neural network model so as to allow the neural network model to output a second direction coordinate correction amount, wherein a first direction sum coordinate is obtained after the first direction coordinate correction amount and the position coordinate of the at least one correction point are added together, and a second direction sum coordinate is obtained after the second direction coordinate correction amount and the position coordinate of the at least one correction point are added together, wherein in the step (2), a pixel value of the original image corresponding to the first direction sum coordinate and the second direction sum coordinate is used as the pixel value of the at least one correction point.

19. The image distortion correction method according to claim 18,

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are a horizontal direction coordinate correction amount and a vertical direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are a horizontal direction sum coordinate and a vertical direction sum coordinate, respectively;

or

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are the vertical direction coordinate correction amount and the horizontal direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are the vertical direction sum coordinate and the horizontal direction sum coordinate, respectively.

20. The image distortion correction method according to claim 15, wherein the neural network model comprises an input layer, a hidden layer and an output layer, and the hidden layer comprises plural neurons, wherein a transfer function of each neuron of the hidden layer is a non-linear function, wherein the first direction coordinate correction amount is outputted from the neural network model at least according to a first direction parameter group corresponding to at least one neuron of the hidden layer, and the second direction coordinate correction amount is outputted from the neural network model at least according to a second direction parameter group corresponding to the at least one neuron of the hidden layer, wherein the first direction parameter group comprises a first horizontal direction coordinate weight, a first vertical direction coordinate weight and a first direction bias value, and the second direction parameter group comprises a second horizontal direction coordinate weight, a second vertical direction coordinate weight and a second direction bias value.

21. The image distortion correction method according to claim 20, wherein the hidden layer comprises four neurons.

22. An image distortion correction device for performing a correcting operation on an original image so as to obtain a correction image with plural correction points, the image distortion correction device comprising:

a data storage and control unit for obtaining a position coordinate of each correction point, a first direction parameter group and a second direction parameter group;

a neural network computing unit connected with the data storage and control unit for generating and outputting a first direction correction point parameter of each correction point according to the position coordinate of each correction point and the first direction parameter group, and generating and outputting a second direction correction point parameter of each correction point according to the position coordinate of each correction point and the second direction parameter group; and

a pixel value computing unit connected with the neural network computing unit for obtaining a pixel value of each correction point from the original image according to the first direction correction point parameter and the second direction correction point parameter of each correction point.

23. The image distortion correction device according to claim 22, wherein the first direction correction point parameter is a first direction coordinate correction amount, and the second direction correction point parameter is a second direction coordinate correction amount.

24. The image distortion correction device according to claim 23, wherein a first direction sum coordinate is obtained after the first direction coordinate correction amount and the position coordinate of each correction point are added together, and a second direction sum coordinate is obtained after the second direction coordinate correction amount and the position coordinate of each correction point are added together, wherein the pixel value of each correction point is obtained from the original image by the pixel value computing unit according to the first direction sum coordinate and the second direction sum coordinate of each correction point.

25. The image distortion correction device according to claim 24, wherein the pixel value computing unit comprises an interpolation unit for interpolating pixel values of plural neighboring points around the first direction sum coordinate and the second direction sum coordinate of the original image, thereby obtaining the pixel value of each correction point.

26. The image distortion correction device according to claim 24,

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are a horizontal direction coordinate correction amount and a vertical direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are a horizontal direction sum coordinate and a vertical direction sum coordinate, respectively;

or

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are the vertical direction coordinate correction amount and the horizontal direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are the vertical direction sum coordinate and the horizontal direction sum coordinate, respectively.

27. The image distortion correction device according to claim 22, wherein the neural network computing unit comprises plural hidden layer neuron computing units, wherein each of the hidden layer neuron computing unit comprises an input computing unit and a transfer function computing unit, wherein the first direction parameter group comprises a first horizontal direction coordinate weight, a first vertical direction coordinate weight and a first direction bias value corresponding to at least one hidden layer neuron computing unit of the neural network computing unit, and the second direction parameter group comprises a second horizontal direction coordinate weight, a second vertical direction coordinate weight and a second direction bias value corresponding to the at least one hidden layer neuron computing unit of the neural network computing unit.

28. The image distortion correction device according to claim 27, wherein the input computing unit is an index computing unit, and the transfer function computing unit is a non-linear function computing unit, wherein after the position coordinate of each correction point is received by the index computing unit, the index computing unit generates and outputs an index value to the non-linear function computing unit, and/or the plural hidden layer neuron computing units comprise four hidden layer neuron computing units, or the plural hidden layer neuron computing units comprise four first direction hidden layer neuron computing units and four second direction hidden layer neuron computing units.

29. The image distortion correction device according to claim 28, wherein the non-linear function computing unit comprises an integer index function computing unit, a decimal fraction index function computing unit, and a multiplication and integration computing unit connected with the integer index function computing unit and the decimal fraction index function computing unit, wherein the index value is divided into an integer index value and a decimal fraction index value, wherein the integer index function computing unit receives the integer index value and generates an integer index function value, the decimal fraction index function computing unit receives the decimal fraction index value and generates a decimal fraction index function value, and the multiplication and integration computing unit receives the integer index function value and the decimal fraction index function value and generates a product of the integer index function value and the decimal fraction index function value.

30. The image distortion correction device according to claim 29, wherein the integer index function computing unit has an integer index function table, wherein after the integer index value is received by the integer index function computing unit, the integer index function computing unit generates and outputs the integer index function value according to the coordinate rotation table.

31. The image distortion correction device according to claim 29,

wherein the decimal fraction index function computing unit comprises a coordinate rotation digital computer (CORDIC) for performing an index computation after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit has a decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to a Taylor series expansion after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit has the decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table and the Taylor series expansion after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit comprises the coordinate rotation digital computer (CORDIC) and the decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the index computation of the coordinate rotation digital computer (CORDIC) and the decimal fraction index function table.

32. The image distortion correction device according to claim 31, wherein if the decimal fraction index value is smaller than a predetermined value, the coordinate rotation digital computer performs the index computation to obtain the decimal fraction index function value, wherein if the decimal fraction index value is larger than the predetermined value, the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table.

33. The image distortion correction device according to claim 22, wherein the image distortion correction device is included in an image pickup device or an image projection device, and/or the image distortion correction device is implemented by a field-programmable gate array (FPGA) or a very-large-scale integration (VLSI) circuit.

34. An image distortion correction device for performing a correcting operation on an original image so as to obtain a correction image with plural correction points, the image distortion correction device comprising:

a data storage and control unit for obtaining a position coordinate of each correction point and a direction parameter group;

a neural network computing unit connected with the data storage and control unit for generating and outputting a coordinate correction amount of each correction point according to the position coordinate of each correction point and the direction parameter group;

an arithmetic computing unit connected with the neural network computing unit for performing an arithmetic operation on the position coordinate of each correction point and the corresponding coordinate correction amount, thereby obtaining an arithmetic coordinate of each correction point; and

a pixel value computing unit connected with the arithmetic computing unit for obtaining a pixel value of each correction point from the original image according to the arithmetic coordinate.

35. The image distortion correction device according to claim 34, wherein the arithmetic operation is an addition operation, and the arithmetic coordinate is a sum coordinate.

36. The image distortion correction device according to claim 35, wherein the direction parameter group comprises a first direction parameter group and a second direction parameter group, the coordinate correction amount of each correction point comprises a first direction coordinate correction amount and a second direction coordinate correction amount, and the sum coordinate comprises a first direction sum coordinate and a second direction sum coordinate, wherein the neural network computing unit generates and outputs the first direction coordinate correction amount of each correction point according to the position coordinate of each correction point and the first direction parameter group, and generates and outputs a second direction coordinate correction amount of each correction point according to the position coordinate of each correction point and the second direction parameter group, wherein the first direction sum coordinate is obtained after the first direction coordinate correction amount and the position coordinate of each correction point are added together, and the second direction sum coordinate is obtained after the second direction coordinate correction amount and the position coordinate of each correction point are added together.

37. The image distortion correction device according to claim 36, wherein the pixel value computing unit comprises an interpolation unit for interpolating pixel values of plural neighboring points around the first direction sum coordinate and the second direction sum coordinate of the original image, thereby obtaining the pixel value of each correction point.

38. The image distortion correction device according to claim 36,

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are a horizontal direction coordinate correction amount and a vertical direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are a horizontal direction sum coordinate and a vertical direction sum coordinate, respectively;

or

wherein the first direction coordinate correction amount and the second direction coordinate correction amount are the vertical direction coordinate correction amount and the horizontal direction coordinate correction amount, respectively, and the first direction sum coordinate and the second direction sum coordinate are the vertical direction sum coordinate and the horizontal direction sum coordinate, respectively.

39. The image distortion correction device according to claim 36, wherein the neural network computing unit comprises plural hidden layer neuron computing units, wherein each of the hidden layer neuron computing unit comprises an input computing unit and a transfer function computing unit, wherein the first direction parameter group comprises a first horizontal direction coordinate weight, a first vertical direction coordinate weight and a first direction bias value corresponding to at least one hidden layer neuron computing unit of the neural network computing unit, and the second direction parameter group comprises a second horizontal direction coordinate weight, a second vertical direction coordinate weight and a second direction bias value corresponding to the at least one hidden layer neuron computing unit of the neural network computing unit.

40. The image distortion correction device according to claim 39, wherein the input computing unit is an index computing unit, and the transfer function computing unit is a non-linear function computing unit, wherein after the position coordinate of each correction point is received by the index computing unit, the index computing unit generates and outputs an index value to the non-linear function computing unit, and/or the plural hidden layer neuron computing units comprise four hidden layer neuron computing units, or the plural hidden layer neuron computing units comprise four first direction hidden layer neuron computing units and four second direction hidden layer neuron computing units.

41. The image distortion correction device according to claim 40, wherein the non-linear function computing unit comprises an integer index function computing unit, a decimal fraction index function computing unit, and a multiplication and integration computing unit connected with the integer index function computing unit and the decimal fraction index function computing unit, wherein the index value is divided into an integer index value and a decimal fraction index value, wherein the integer index function computing unit receives the integer index value and generates an integer index function value, the decimal fraction index function computing unit receives the decimal fraction index value and generates a decimal fraction index function value, and the multiplication and integration computing unit receives the integer index function value and the decimal fraction index function value and generates a product of the integer index function value and the decimal fraction index function value.

42. The image distortion correction device according to claim 41, wherein the integer index function computing unit has an integer index function table, wherein after the integer index value is received by the integer index function computing unit, the integer index function computing unit generates and outputs the integer index function value according to the coordinate rotation table.

43. The image distortion correction device according to claim 41,

wherein the decimal fraction index function computing unit comprises a coordinate rotation digital computer (CORDIC) for performing an index computation after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit has a decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to a Taylor series expansion after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit has the decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table and the Taylor series expansion after the decimal fraction index value is received by the decimal fraction index function computing unit;

or

wherein the decimal fraction index function computing unit comprises the coordinate rotation digital computer (CORDIC) and the decimal fraction index function table, and the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the index computation of the coordinate rotation digital computer (CORDIC) and the decimal fraction index function table.

44. The image distortion correction device according to claim 43, wherein if the decimal fraction index value is smaller than a predetermined value, the coordinate rotation digital computer performs the index computation to obtain the decimal fraction index function value, wherein if the decimal fraction index value is larger than the predetermined value, the decimal fraction index function computing unit generates and outputs the decimal fraction index function value according to the decimal fraction index function table.

45. The image distortion correction device according to claim 34, wherein the image distortion correction device is included in an image pickup device or an image projection device, and/or the image distortion correction device is implemented by a field-programmable gate array (FPGA) or a very-large-scale integration (VLSI) circuit.