Image processing device, image processing method, image processing program

• US 8,947,588 B2
• Filed: 01/24/2013
• Issued: 02/03/2015
• Est. Priority Date: 02/14/2012
• Status: Active Grant

• Current Assignee: JVC Kenwood Corporation
• Original Assignee: JVC Kenwood Corporation
• Inventors: Shirakawa, Ryuji
• Primary Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US13/749,184
• Publication Number:
  US 20130208144A1
• Time in Patent Office: 740 Days
• Field of Search: 348370-376, 396 61- 62, 396106-110
• US Class Current: 348/371
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS

Abstract

A flash detection unit calculates a line average luminance of each line of the current screen of image data and a screen average luminance of a past screen at least one screen before the current screen and compares the calculated line average luminance with the calculated screen average luminance to detect whether the current screen includes a line of high luminance due to a flash. A holding unit holds the past screen of the image data. A flash correction unit, if it is detected that some lines of the current screen have high luminance, replaces the lines having high luminance in the current screen with corresponding lines of the past screen held in the holding unit to correct the image data.

Claims

1. An image processing device comprising:

a flash detection unit configured to calculate a line average luminance of each line of a current frame of image data and a frame average luminance of a past frame at least one frame before the current frame and to compare the calculated line average luminance of each line with the calculated frame average luminance to detect whether some lines of the current frame have high luminance due to a flash;

a holding unit configured to hold the past frame of the image data; and

a flash correction unit configured to replace the lines having high luminance in the current frame with corresponding lines of the past frame, which are held in the holding unit, to correct the image data, if the flash detection unit detects that some lines of the current frame have high luminance, wherein

the flash detection unit includes;

a block average luminance calculation section configured to calculate a block average luminance of each block of the blocks which are obtained by dividing each line of the current frame;

a line average luminance calculation section configured to calculate the line average luminance of each line of the current frame based on the block average luminances of each line calculated by the block average luminance calculation section;

a frame average luminance calculation section configured to calculate the frame average luminance of the past frame based on the line average luminance of each line of the past frame calculated by the line average luminance calculation section;

a difference calculation section configured to calculate a difference value between each block average luminance and the frame average luminance and a difference value between each line average luminance and the frame average luminance; and

a flash determination section configured to determine a presence of high luminance due to a flash if at least a predetermined number of consecutive lines are in a particular state, the particular state being a state in which the difference between the line average luminance and the frame average luminance is equal to or larger than a first threshold value and the difference between each block average luminance and the frame average luminance is equal to or larger than a second threshold value.

2. The image processing device according to claim 1, wherein the flash determination section determines that a current line is in the particular state if the difference value between each block average luminance of a line above the current line calculated by the block average luminance calculation section and the frame average luminance calculated by the frame average luminance calculation section is less than a third threshold value which is equal to or different from the second threshold value.

3. The image processing device according to claim 2, wherein the line above the current line is any number of lines between one and three before the current line.

4. The image processing device according to claim 1, wherein the flash determination section determines that a current line is in the particular state if the difference value between the block average luminance of each block of the current line and the block average luminance of the corresponding block of a line above the current line is equal to or larger than a third threshold value, the block average luminances being calculated by the block average luminance calculation section.

5. The image processing device according to claim 4, wherein the line above the current line is any number of lines between one and three before the current line.

6. The image processing device according to claim 1, wherein the flash determination section determines that a current line is not in the particular state if the difference between the line average luminance of the current line and the frame average luminance is less than a fourth threshold value which is smaller than the first threshold value.

7. An image processing device comprising:

a flash detection unit configured to calculate a line average luminance of each line of a current frame of image data and a frame average luminance of a past frame at least one frame before the current frame and to compare the calculated line average luminance of each line with the calculated frame average luminance to detect whether some lines of the current frame have high luminance due to a flash;

a holding unit configured to hold the past frame of the image data and

a flash correction unit configured to replace the lines having high luminance in the current frame with corresponding lines of the past frame, which are held in the holding unit, to correct the image data, if the flash detection unit detects that some lines of the current frame have high luminance, wherein

the flash detection unit detects a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line, and

the flash correction unit sets a first adjacent area which is located above the flash start line and is adjacent thereto and a second adjacent area which is located below the flash end line and is adjacent thereto,

in the first adjacent area, mixes image data of the current frame and image data of the past frame in such a manner that a proportion of image data of the current frame gradually decreases from the first line to the last line of the first adjacent area and a proportion of image data of the past frame gradually increases from the first line to the last line, and

in the second adjacent area, mixes image data of the current frame and image data of the past frame in such a manner that the proportion of image data of the current frame gradually increases from the first line to the last line of the second adjacent area and the proportion of image data of the past frame gradually decreases from the first line to the last line.

8. An image processing device comprising:

a flash detection unit configured to calculate a line average luminance of each line of a current frame of image data and a frame average luminance of a past frame at least one frame before the current frame and to compare the calculated line average luminance of each line with the calculated frame average luminance to detect whether some lines of the current frame have high luminance due to a flash;

a holding unit configured to hold the past frame of the image data; and

a flash correction unit configured to replace the lines having high luminance in the current frame with corresponding lines of the past frame, which are held in the holding unit, to correct the image data, if the flash detection unit detects that some lines of the current frame have high luminance, wherein

the flash detection unit detects a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line, and

the flash correction unit sets a first adjacent area which is located above the flash start line and adjacent thereto and a second adjacent area which is located below the flash end line and adjacent thereto,

in the first and second adjacent areas, the flash correction unit corrects the image data by using only pixel data of the current frame if a brightness of each pixel in the first and second adjacent areas is less than a first threshold value, using only pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is more than a second threshold value, which is larger than the first threshold value, and mixing the pixel data of the current frame and the pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is equal to or larger than the first threshold value and equal to or less than the second threshold value.

9. An image processing method comprising:

calculating a line average luminance of each line of a current frame of image data;

calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the current frame have high luminance due to a flash, by

holding the past frame of the image data in a holding unit; and

if it is detected that some lines of the current frame have high luminance, replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data,

the method further comprising;

calculating a block average luminance of each block of the blocks which are obtained by dividing each line of the current frame;

calculating the line average luminance of each line of the current frame based on the block average luminances of each line;

calculating the frame average luminance of the past frame based on the line average luminance of each line of the past frame;

calculating a difference value between each block average luminance and the frame average luminance and a difference value between each line average luminance and the frame average luminance and;

determining that some lines of the current frame have high luminance due to a flash if at least a predetermined number of consecutive lines are in a particular state, the particular state being a state in which the difference between the line average luminance of a line and the frame average luminance is equal to or larger than a first threshold value and the difference between the block average luminance of each block of the line and the frame average luminance is less than a second threshold value.

10. The image processing method according to claim 9, the method further comprising:

determining that a current line is in the particular state if the difference value between each block average luminance calculated for a line above the current line and the calculated frame average luminance is less than a third threshold value which is equal to or different from the second threshold value.

11. The image processing method according to claim 9, the method further comprising:

determining that a current line is in the particular state if the difference value between the block average luminance of each block calculated for the current line and the corresponding block average luminance calculated for a line above the current line is equal to or larger than a third threshold value.

12. An image processing method comprising:

calculating a line average luminance of each line of a current frame of image data;

calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the current frame have high luminance due to a flash, by

holding the past frame of the image data in a holding unit; and

if it is detected that some lines of the current frame have high luminance, replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data,

the method further comprising;

detecting a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line;

setting a first adjacent area which is located above the flash start line and is adjacent thereto and a second adjacent area which is located below the flash end line and is adjacent thereto;

correcting the image data in the first adjacent area by mixing image data of the current frame and image data of the past frame in such a manner that a proportion of the image data of the current frame gradually decreases from the first line to the last line of the first adjacent area while a proportion of the image data of the past frame gradually increases; and

correcting the image data in the second adjacent area by mixing the image data of the current frame and image data of the past frame in such a manner that the proportion of the image data of the current frame gradually increases from the first line to the last line of the first adjacent area as the proportion of the image data of the past frame decreases.

13. An image processing method comprising:

calculating a line average luminance of each line of a current frame of image data;

calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the current frame have high luminance due to a flash, by

holding the past frame of the image data in a holding unit; and

if it is detected that some lines of the current frame have high luminance, replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data,

the method further comprising;

detecting a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line;

setting a first adjacent area which is located above the flash start line and is adjacent thereto and a second adjacent area which is located below the flash end line and is adjacent thereto; and

correcting the image data in the first and second adjacent areas by using only pixel data of the current frame if a brightness of each pixel in the first and second adjacent areas is less than a first threshold value, using only pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is more than a second threshold value, which is larger than the first threshold value, and mixing the pixel data of the current frame and the pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is equal to or larger than the first threshold value and equal to or less than the second threshold value.

14. A non-transitory computer readable medium storing an image processing program for causing a computer to implement the following functions, comprising:

a function of calculating a line average luminance of each line of a current frame of image data;

a function of calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

a function of comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the current frame have high luminance due to a flash, by

a function of holding the past frame of the image data in a holding unit; and

a function of replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data, if it is detected that some lines of the current frame have high luminance,

the image processing program further implementing the following functions of;

calculating a block average luminance of each block of the blocks which are obtained by dividing each line of the current frame;

calculating the line average luminance of each line of the current frame based on the block average luminances of each line;

calculating the frame average luminance of the past frame based on the line average luminance of each line of the past frame;

calculating a difference between each block average luminance and the frame average luminance, and a difference between each line average luminance and the frame average luminance, and

determining that some lines of the current have high luminance due to a flash if at least a predetermined number of consecutive lines are in a particular state, the particular state being a state in which the difference between the line average luminance of a line and the frame average luminance is equal to or larger than a first threshold value and the difference between the block average luminance of each block of the line and the frame average luminance is less than a second threshold value.

15. The non-transitory computer readable medium storing the image processing program according to claim 14, the program causing the computer to further implement the following functions of:

determining that a current line is in the particular state if the difference value between the block average luminance of each block calculated for the current line and the corresponding block average luminance calculated for a line above the current line is equal to or larger than a third threshold value.

16. The non-transitory computer readable medium storing the image processing program according to claim 14, the program causing the computer to further implement the following functions of:

determining that a current line is in the particular state if the difference value between each block average luminance calculated for a line above the current line and the calculated frame average luminance is less than a third threshold value which is equal to or different from the second threshold value.

17. A non-transitory computer readable medium storing an image processing program for causing a computer to implement the following functions, comprising:

a function of calculating a line average luminance of each line of a current frame of image data;

a function of calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

a function of comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the frame have high luminance due to a flash, by

a function of holding the past frame of the image data in a holding unit; and

a function of replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data, if it is detected that some lines of the current frame have high luminance,

the image processing program further implementing the following functions of;

detecting a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line;

setting a first adjacent area which is located above the flash start line and is adjacent thereto and a second adjacent area which is located below the flash end line and is adjacent thereto;

mixing, in the first adjacent area, image data of the current frame and image data of the past frame in such a manner that a proportion of the image data of the current frame gradually decreases from the first line to the last line of the first adjacent area as a proportion of the image data of the past frame gradually increases; and

mixing, in the second adjacent area, the image data of the current frame and the image data of the past frame in such a manner that the proportion of the image data of the current frame gradually increases from the first line to the last line of the first adjacent area as the proportion of the image data of the past frame decreases.

18. A non-transitory computer readable medium storing an image processing program for causing a computer to implement the following functions, comprising:

a function of calculating a line average luminance of each line of a current frame of image data;

a function of calculating a frame average luminance of a past frame of the image data, the past frame being at least one frame before the current frame;

a function of comparing the line average luminance of each line and the frame average luminance to detect whether some lines of the frame have high luminance due to a flash, by

a function of holding the past frame of the image data in a holding unit; and

a function of replacing the lines of high luminance in the current frame with lines of the past frame held by the holding unit, to correct the image data, if it is detected that some lines of the current frame have high luminance,

the image processing program further implementing the following functions of;

detecting a flash start line as a first line that has high luminance due to a flash and a flash end line as a last line;

setting a first adjacent area which is located above the flash start line and is adjacent thereto and a second adjacent area which is located below the flash end line and is adjacent thereto;

correcting the image data in the first and second adjacent areas by using only pixel data of the current frame if a brightness of each pixel in the first and second adjacent areas is less than a first threshold value, using only pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is more than a second threshold value, which is larger than the first threshold value, and mixing the pixel data of the current frame and the pixel data of the past frame if the brightness of each pixel in the first and second adjacent areas is equal to or larger than the first threshold value and equal to or less than the second threshold value.