Method and apparatus for detecting text associated with video

• US 7,446,817 B2
• Filed: 02/14/2005
• Issued: 11/04/2008
• Est. Priority Date: 02/18/2004
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Moon, Youngsu, Kim, Jiyeun, Jung, Cheolkon
• Primary Examiner(s): Natnael; Paulos M
• Application Number:
  US11/056,361
• Publication Number:
  US 20050196043A1
• Time in Patent Office: 1,359 Days
• Field of Search: 348/468, 348/461, 381/176, 381/229, 381/321
• US Class Current: 348/468
• CPC Class Codes: G06V 20/62 Text, e.g. of license plates, overlay texts or captions on TV images
• CPC Class Codes: G06V 30/10 Character recognition

Abstract

A method and apparatus for detecting text associated with video are provided. The method of detecting the text of the video includes reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame, determining whether there is a text area detected from a previous frame which is a (t−N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame, and upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame. Upon determining that there is the text area detected from the previous frame in the current frame, the text area is detected from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame. Whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video is verified.

Claims

1. A method of detecting a text associated with video, the method comprising:

reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

determining whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame;

upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame; and

upon determining that there is the text area detected from the previous frame in the current frame, detecting the text area from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame and verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein the detecting of the text area from the current frame comprises

detecting a character block from the current frame,

determining whether the current frame is a text frame, using the detected character block, and

upon determining that the current frame is the text frame, post-processing the text frame and determining a text area from the text frame.

2. The method of claim 1, further comprising:

after the text area is detected or verified, determining whether the current frame is an end frame of the video, and

upon determining that the current frame is not the end frame of the video, increasing t by a value of N and reading the current frame.

3. The method of claim 1, wherein the detecting of the character block comprises:

obtaining a brightness difference between each pixel of the current frame and each pixel of the next frame, the pixels being placed in the same position, and determining a pixel of the current frame used in obtaining a brightness difference less than a first threshold value as a text candidate pixel,

dividing the current frame into sub-blocks having a predetermined size,

obtaining the number of text candidate pixels included in each of the sub-blocks,

determining a sub-block having the number of a text candidate pixel greater than a second threshold value as a text candidate block,

obtaining an edge density of each text candidate block, and

determining a text candidate block having the edge density greater than a third threshold value as the character block.

4. The method of claim 3, wherein the edgy density is obtained by dividing summed results of edge gradients of pixels included in the text candidate block by the number of pixels included in the text candidate block.

5. The method of claim 3, wherein the edge density is obtained by dividing a sum of results obtained by multiplying edge gradients of pixels, included in the text candidate block by corresponding weighted values, by the number of pixels included in the text candidate block.

6. The method of claim 5, wherein the weighted value is obtained by dividing the gradient by a maximum value of a gradient that may occur in the text candidate block.

7. The method of claim 1, wherein the determining whether the current frame is the text frame comprises determining whether the number of character blocks included in the current frame is greater than a fourth threshold value, and whether the current frame having the number of the character blocks greater than the fourth threshold value corresponds to the text frame.

8. The method of claim 1, wherein the post-processing of the text frame comprises:

upon determining that the current frame is the text frame, removing an isolated character block in which there is no adjacent character block, from the character block included in the text frame,

determining a group of adjacent character blocks in the text frame from which the isolated character block is removed, as a bounding box,

obtaining the number of sub-blocks included in each bounding box when the current frame is divided into sub-blocks having a predetermined size, and extracting only a bounding box having the number of sub-blocks greater than a fifth threshold value from bounding boxes included in the text frame, and

obtaining an average edge density with respect to each of the extracted bounding boxes and determining a bounding box having an average edge density greater than a sixth threshold value as the text area.

9. The method of claim 8, wherein the determining of the bounding box comprises:

obtaining the number of character blocks included in each row and each column of the text frame from which the isolated character block is removed,

selecting a row having the number of character blocks greater than a seventh threshold value and a column having the number of character blocks greater than an eighth threshold value, and

determining the bounding box by combining the selected rows with the selected columns,

wherein each row and each column are set when the current frame is divided into the sub-blocks.

10. The method of claim 9, wherein the seventh threshold value is the same as the eighth threshold value.

11. The method of claim 9, wherein the determining of the bounding box by combining the selected rows and the selected columns comprises:

determining a portion in which the character blocks are placed continuously in the selected row as a row area, and determining a portion in which the character blocks are placed continuously in the selected column, as a column area, and

determining a portion in which the determined row area and the determined column area overlap each other, as the bounding box.

12. The method of claim 8, wherein in the extracting of the bounding box, the bounding box having the number of sub-blocks greater than the fifth threshold value is extracted from the text frame using a connected component labeling method.

13. The method of claim 8, wherein the average edgy density corresponds to an average of edge densities of the sub-blocks included in the extracted bounding boxes.

14. The method of claim 8, wherein the average edge density corresponds to an average of edge densities of the text candidate blocks included in the extracted bounding boxes.

15. The method of claim 8, wherein the average edgy density corresponds to an average of edge densities of the character blocks included in the extracted bounding boxes.

16. The method of claim 1, wherein the detecting of the text area from the current frame further comprises searching for a starting frame, in which text associated with the video starts to be displayed on, among frames from the current frame to a frame prior to the previous frame.

17. The method of claim 16, wherein the searching for the starting frame comprises:

obtaining the number of brightness differences greater than a ninth threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t-1 )-th frame corresponding to the text area decoded in the t-th frame,

determining whether the number of the brightness differences greater than the ninth threshold value is greater than a tenth threshold value,

upon determining that the number of the brightness differences greater than the ninth threshold value is greater than the tenth threshold value, −

setting the t-th frame as the starting frame

upon determining that the number of the brightness differences greater than the ninth threshold value is not greater than the tenth threshold value, reducing t by a value of 1,

determining whether the t-th frame is a previous frame, and upon determining that the t-th frame is not the previous frame, proceeding the obtaining the number of the brightness differences, and

upon determining that the t-th frame is the previous frame, setting a (t−

N+1)-th frame is as the starting frame.

18. A method of detecting a text associated with video, the method comprising:

reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

determining whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame;

upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame; and

upon determining that there is the text area detected from the previous frame in the current frame, detecting the text area from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame and verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein the detecting of the text area from the remaining area of the current frame comprises;

detecting a character block from the remaining area of the current frame,

determining whether the current frame is a text frame based on the remaining area of the current frame, and

upon determining that the current frame is the text frame, post-processing the remaining area and determining the text area from the remaining area.

19. A method of detecting a text associated with video, the method comprising:

reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

determining whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame;

upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame; and

upon determining that there is the text area detected from the previous frame in the current frame, detecting the text area from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame and verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein the determining whether there is the text area in the next frame comprises;

obtaining the number of brightness differences greater than an eleventh threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t+N)-th frame corresponding to the text area determined in the t-th frame, and

upon determining that the number of the brightness differences greater than the eleventh threshold value is greater than a twelfth threshold value

if there is no text area in the next frame, setting the number of the brightness differences greater than the eleventh threshold value as greater than the twelfth threshold value.

20. The method of claim 19 wherein the determining whether there is the text area in the next frame comprises searching for an end frame in which text associated with video ends if it is determined that there is no text area in the next frame.

21. The method of claim 20, wherein the searching for the end frame comprises:

obtaining the number of brightness differences greater than a thirteenth threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t+1)-th frame corresponding to the text area determined in the t-th frame,

determining whether the number of the brightness differences greater the thirteenth threshold value is greater than a fourteenth threshold value,

upon determining that the number of the brightness differences greater than the thirteenth threshold value is greater than the fourteenth threshold value, determining the t-th frame as the end frame,

upon determining that the number of the brightness differences greater than the thirteenth threshold value is not greater than the fourteenth threshold value, increasing t by a value of 1,

determining whether the t-th frame is the next frame and if it is determined that the t-th frame is not the next frame, obtaining the number of the brightness differences, and

upon determining that the t-th frame is the next frame, determining a (t+N−

1)-th frame as the end frame.

22. An apparatus for detecting text associated with video, the apparatus comprising:

a frame inputting unit reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

a text area checking unit checking whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame and outputting the checked results as a first control signal;

a first text area detecting unit detecting text area from the current frame in response to the first control signal;

a second text area detecting unit detecting text area from a remaining area obtained by excluding an area corresponding to the text area detected from the previous frame from the current frame in response to the first control signal; and

a text area verifying unit verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video.

23. The apparatus of claim 22, further comprising:

an end frame checking unit checking whether the current frame inputted by the frame inputting unit is an end frame of the video and outputting the checked results as a second control signal in response to a signal inputted by the first text area detecting unit that represents whether detecting the text area is completed and a signal inputted by the text area verifying unit that represents whether verifying the text area is completed; and

a first t-adjusting unit increasing t by a value of N and outputting the increased value of t to the frame inputting unit in response to the second control signal.

24. The apparatus of claim 22, wherein the first text area detecting unit comprises:

a first character block detecting portion detecting a character block from the current frame in response to the first control signal,

a first text frame checking portion checking whether the current frame is a text frame using the detected character block and outputting the checked results as a third control signal, and

a first text area determining portion post-processing the text frame and determining a text area from the text frame in response to the third control signal.

25. The apparatus of claim 24, wherein the first character block detecting portion comprises:

a text candidate pixel determining part obtaining a brightness difference between each pixel of the current frame and each pixel of the next frame, the pixels being placed in the same position, and determining a pixel of the current frame used in obtaining a brightness difference less than a first threshold value as a text candidate pixel in response to the first control signal,

a text candidate block determining part dividing the current frame into sub-blocks having a predetermined size, obtaining the number of text candidate pixels included in each of the sub-blocks, and determining a sub-block having the number of a text candidate pixel greater than a second threshold value as a text candidate block, and

a character block determining part obtaining an edge density of each text candidate block and determining text candidate block having an edge density greater than a third threshold value as the character block.

26. The apparatus of claim 24, wherein the first text frame checking portion comprises:

a character block number checking part checking whether the number of character blocks included in the current frame is greater than a fourth threshold value, and

a text frame determining part determining the current frame having the number of character blocks greater than the fourth threshold value as the text frame.

27. The apparatus of claim 24, wherein the first text area determining portion comprises:

an isolated character block removing part removing isolated character block in which there is no adjacent character block, from the character blocks included in the text frame in response to the third control signal,

a bounding box determining part determining a group of adjacent character blocks from the text frame from which the isolated character block is removed, as a bounding box,

a bounding box extracting part obtaining the number of sub-blocks included in each bounding box and extracting the bounding box having the number of sub-blocks greater than a fifth threshold value from the text frame when the current frame is divided into sub-blocks having a certain size, and

a text area extracting part obtaining an average edge density with respect to each of extracted bounding boxes and determining a bounding box having an average edge density greater than a sixth threshold value as the text area.

28. The apparatus of claim 27, wherein the bounding box determining part comprises:

a character block number calculating part calculating the number of character blocks included in each row and each column of the text frame from which the isolated character block is removed,

a row and column selecting part selecting a row having the number of character blocks greater than a seventh threshold value and column having the number of character blocks greater than an eighth threshold value, and

a row and column combining part determining the bounding box by combining the selected rows with the selected columns,

wherein each row and each column are set when the current frame is divided into the sub-blocks.

29. The apparatus of claim 28, wherein the row and column combining part comprises:

an area determining part determining a portion in which character blocks are placed continuously in the selected row, as a row area and determining a portion in which character blocks are placed continuously in the selected column, as a column area, and

an overlapped area checking part determining a portion in which the determined row area and the determined column area overlap each other, as the bounding box.

30. The apparatus of claim 24, wherein the first text area detecting unit further comprises a starting frame searching portion searching for a starting frame in which text associated with the video starts to be displayed, among frames from a current frame to a frame prior to the previous frame in response to a signal inputted by the first text area determining portion that represents whether determining the text area is completed.

31. The apparatus of claim 30, wherein the starting frame searching portion comprises:

a first brightness difference number calculating part obtaining the number of brightness differences greater than a ninth threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t−

1 )-th frame corresponding to the text area determined in the t-th frame in response to a signal inputted by the first text area determining portion that represents whether determining the text area is completed,

a first number comparing part comparing the number of the brightness differences greater than the ninth threshold value with a tenth threshold value and outputting the compared results,

a second t-adjusting part reducing t by a value of 1 in response to the compared results of the first number comparing part,

a previous frame checking part checking whether the t-th frame adjusted by the second t-adjusting part is the previous frame and outputting the checked results; and

a starting frame determining the t-th frame as the starting frame in response to the compared results of the first number comparing part or determining the (t−

N+1 )-th frame as the starting frame in response to the checked results of the previous frame checking part,

wherein the first brightness difference number calculating part calculates the number of the brightness differences in response to the checked results of the previous frame checking part.

32. The apparatus of claim 22, wherein the second text area detecting unit comprises:

a second character block detecting portion detecting a character block from the remaining area of the current frame in response to the first control signal ,a second text frame checking portion checking whether the current frame is text frame using the character block extracted from the remaining area of the current frame and outputting the checked results as a fourth control signal, and

a second text area determining portion post-processing the remaining area and determining text area from the remaining area in response to the fourth control signal.

33. The apparatus of claim 22, wherein the text area verifying unit comprises:

a second brightness difference number calculating portion calculating the number of brightness differences greater than an eleventh threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t+N)-th frame corresponding to the text area determined in the t-th frame in response to a signal inputted by the second text area detecting unit that represents whether determining the text area is completed, and

a second number comparing portion comparing the number of the brightness differences greater than the eleventh threshold value with a twelfth threshold value,

wherein when there is text area in the next frame, the number of the brightness differences greater than the eleventh threshold value is not greater than the twelfth threshold value.

34. The apparatus of claim 33, wherein the text area verifying unit further comprises an end frame searching portion searching for an end frame in which text associated with the video ends, in response to the compared results of the second number comparing portion.

35. The apparatus of claim 34, wherein the end frame searching portion comprises:

a third brightness difference number calculating part obtaining the number of brightness differences greater than a thirteenth threshold value among brightness differences which are differences between brightness values of pixels in the text area determined in the t-th frame and brightness values of pixels in the text area of the (t+1)-th frame corresponding to the text area determined in the t-th frame in response to the compared results of the second number comparing portion,

a third number comparing part comparing the number of the brightness differences greater than the thirteenth threshold value with a fourteenth threshold value,

a third t-adjusting part increasing t by a value of 1 in response to the compared results of the third number comparing part,

a next frame checking part checking whether the t-th frame adjusted by the third t-adjusting part is the next frame and outputting the checked results; and

an end frame determining part determining the t-th frame as the end frame in response to the compared results of the third number comparing part or setting the (t+N−

1 )-th frame as the end frame in response to the checked results of the next frame checking part,

the third brightness difference number calculating part calculates the number of the brightness differences in response to the checked results of the next frame checking part.

36. A method of detecting a text associated with a video, comprising:

detecting the text area from a remaining area obtained by excluding from a current frame an area corresponding to the text area detected from a previous frame; and

verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein the detecting the text area comprises;

detecting a character block from the remaining area of the current frame,

determining whether the current frame is a text frame based on the remaining area of the current frame, and

upon determining that the current frame is the text frame, post-processing the remaining area and determining the text area from the remaining area.

37. An apparatus for detecting text associated with video, the apparatus comprising:

a first text area detecting unit detecting a first text area from a current frame in response to a first control signal;

a second text area detecting unit detecting a second text area from a remaining area obtained by excluding an area corresponding to the first text area detected from the previous frame from the current frame in response to a first control signal; and

a text area verifying unit verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein the first text area detecting unit comprises;

a first character block detecting portion detecting a character block from the current frame in response to the first control signal,

a first text frame checking portion checking whether the current frame is a text frame using the detected character block and outputting the checked results as a third control signal, and

a first text area determining portion post-processing the text frame and determining a text area from the text frame in response to the third control signal.

38. An apparatus for detecting a text associated with video, comprising:

an isolated character block removing part removing an isolated character block in which there is no adjacent character block, from the character blocks included in a text frame in response to a control signal;

a bounding box determining part determining a group of adjacent character blocks from the text frame, from which the isolated character block is removed, as a bounding box;

a bounding box extracting part obtaining a number of sub-blocks included in each bounding box and extracting the bounding box having the number of sub-blocks greater than a first threshold value from the text frame when the current frame is divided into sub-blocks having a certain size; and

a text area extracting part obtaining an average edge density with respect to each of extracted bounding boxes and determining a bounding box having an average edge density greater than a second threshold value as the text area.

39. The apparatus of claim 38, wherein the bounding box determining part comprises:

a character block number calculating part calculating a number of character blocks included in each row and each column of the text frame from which the isolated character block is removed,

a row and column selecting pad selecting a row having the number of character blocks greater than a third threshold value and column having the number of character blocks greater than an fourth threshold value, and

a row and column combining part determining the bounding box by combining the selected rows with the selected columns, wherein each row and each column are set when the current frame is divided into the sub-blocks.

40. The apparatus according to claim 39, wherein the row and column combining part comprises:

an area determining part determining a portion in which character blocks are placed continuously in the selected row, as a row area and determining a portion in which character blocks are placed continuously in the selected column, as a column area, and

an overlapped area checking pad determining a portion in which the determined row area and the determined column area overlap each other, as the bounding box.

41. A computer-readable storage storing a computer-readable program which controls a computer system to execute detecting text associated with video, by:

reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

determining whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame;

upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame; and

upon determining that there is the text area detected from the previous frame in the current frame, detecting the text area from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame and verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video,

wherein after the text area is detected or verified, determining whether the current frame is an end frame of the video, and

upon determining that the current frame is not the end frame of the video, increasing t by a value of N and reading the current frame.

42. A computer-readable storage storing a computer-readable program which controls a computer system to execute detecting text associated with video, by:

reading a t-th frame (where t is a positive integer) among frames forming the video as a current frame;

determining whether there is a text area detected from a previous frame which is a (t−

N)-th (where N is a positive integer) frame among the frames forming the video, in the current frame;

upon determining that there is no text area detected from the previous frame in the current frame, detecting the text area in the entire current frame;

upon determining that there is the text area detected from the previous frame in the current frame, detecting the text area from a remaining area obtained by excluding from the current frame an area corresponding to the text area detected from the previous frame and verifying whether there is a text area in a next frame which is a (t+N)-th frame among the frames forming the video;

detecting a character block from the current frame;

determining whether the current frame is a text frame, using the detected character block; and

upon determining that the current frame is the text frame, post-processing the text frame and determining a text area from the text frame.