Video processing device and its method

• US 7,817,856 B2
• Filed: 07/15/2005
• Issued: 10/19/2010
• Est. Priority Date: 07/20/2004
• Status: Expired due to Fees

• Current Assignee: Socionext Inc.
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Tanikawa, Yuji, Okada, Masahiro, Yoshii, Tsuyoshi
• Primary Examiner(s): Carter; Aaron W
• Application Number:
  US11/632,967
• Publication Number:
  US 20080085051A1
• Time in Patent Office: 1,922 Days
• Field of Search: 382182-187, 382/176, 382/177, 348/461, 348/465, 348/468, 725 32- 36, 725/137
• US Class Current: 382/182
• CPC Class Codes: G06V 20/635 Overlay text, e.g. embedded captions in a TV program
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 21/478 Supplemental services, e.g. displaying phone caller identification, shopping application
• CPC Class Codes: H04N 21/4882 for displaying messages, e.g. warnings, reminders arrangements for providing short real-time information to substation in data switching networks H04L12/1895
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 5/782 on tape
• CPC Class Codes: H04N 5/85 on discs or drums
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal

Abstract

A video processing method for selectively processing character information (Ic) included in an inputted video stream (Svc). The video stream (Svc) is separated in units of frame into a brightness frame (Y) representing brightness information (VY) and a plurality of color difference frames (Cb, Cr) representing color difference information (VCb, VCr). Character information (Ic) included according to at least either of the brightness information (VY) and the color difference information (VCb, VCr) is recognized. The recognized character information (Ic) is subjected to a processing by at least one of deletion, movement, and enlargement.

Claims

1. A video processing method of selectively processing character information included in an inputted video stream, the video processing method comprising:

a video stream separating step of separating the video stream into a brightness frame representing brightness information and a plurality of color difference frames representing color difference information;

a character recognizing step of recognizing the included character information based on at least either of the brightness information and the color difference information; and

a character processing step of performing at least one of deletion, movement, and enlargement on the recognized character information.

2. The video processing method according to claim 1, wherein the character recognizing step comprises:

a brightness determining step of comparing a brightness of a pixel in the brightness frame with a threshold value;

a character pixel recognizing step of recognizing a pixel having a brightness higher than the threshold value, as a character pixel representing the character information;

a character pixel registering step of registering the recognized character pixel in a character display pixel table; and

a character information frame registering step of registering a frame where at least one recognized character pixel is present, in a character display frame table;

as a frame including the character information.

3. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a color difference value determining step of determining whether a color difference value of the character pixel registered in the character display pixel table is within a predetermined range; and

a character same color recognizing step of recognizing that the character information is the same color when it is determined that the color difference value is within the predetermined range.

4. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a pixel selecting step of selecting the pixel whose brightness is compared with the threshold value, by the brightness determining step.

5. The video processing method according to claim 2, wherein the character recognizing step further comprises:

an inter-character pixel distance calculating step of calculating separation distances between the character pixels registered in the character display pixel table;

an average separation distance calculating step of calculating an average separation distance by averaging the calculated separation distances; and

a non-character pixel recognizing step of deleting the character pixel whose separation distance from the adjoining character pixel is larger than the average separation distance by a predetermined distance, from the character display table.

6. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a horizontal direction character pixel continuity detecting step of detecting continuity, in a horizontal direction, of the character pixel registered in the character display pixel table;

a vertical direction character pixel continuity detecting step of detecting continuity, in a vertical direction, of the character pixel registered in the character display pixel table; and

a character line width recognizing step of recognizing a character line width based on the continuity in the horizontal direction and the continuity in the vertical direction.

7. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a peripheral pixel color difference value determining step of determining whether the color difference values of peripheral pixels of the character pixel registered in the character display pixel table are uniform; and

an edge recognizing step of recognizing that the character information is edged with a uniform color when it is determined that the color difference values of the peripheral pixels are uniform.

8. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a character information occurrence period detecting step of detecting an occurrence period of the character information registered in the character display frame table;

a character information non-occurrence period detecting step of detecting a non-occurrence period of the character information registered in the character display frame table; and

a character information blinking display recognizing step of recognizing that a display method of the character information is blinking display in which display and deletion are repeated at constant time intervals, based on the detected occurrence period and non-occurrence period.

9. The video processing method according to claim 2, wherein the character recognizing step further comprises:

a first pixel distribution detecting step of detecting a first distribution of the character pixel in an arbitrary first frame on a time axis, based on the character display pixel table and the character display frame table;

a second pixel distribution detecting step of detecting a second distribution of the character pixel in an arbitrary second frame on a different time axis from the first frame, based on the character display pixel table and the character display frame table;

a third pixel distribution detecting step of detecting a third distribution of the character pixel on an arbitrary time axis between the first frame and the second frame, based on the first distribution;

a distribution difference detecting step of detecting a difference between the second distribution and the third distribution; and

a moving display recognizing step of recognizing that a display method of the character information is moving display in which the character information moves at a constant speed, based on the difference.

10. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of linearly interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel; and

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

11. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table; and

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

12. The video processing method according to claim 9, further comprising:

a pixel value interpolating step of interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value (VCb, of the corresponding pixel a frame, close in time, not registered in the character display frame table; and

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

13. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

a character pixel after movement position determining step of obtaining a position of the character pixel registered in the character display pixel table when the character pixel is moved by a predetermined distance;

an after movement pixel determining step of determining the pixel corresponding to the determined after movement position; and

a destination pixel value changing step of overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

14. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table, respectively;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

a character pixel after movement position determining step of obtaining a position of the character pixel registered in the character display pixel table when the character pixel is moved by a predetermined distance;

an after movement pixel determining step of determining the pixel corresponding to the determined after movement position; and

a destination pixel value changing step of overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

15. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

an enlarging step of enlarging coordinate values of the character pixel by a predetermined enlargement amount;

an after movement pixel determining step of determining the pixel corresponding to the enlarged position; and

an enlarged pixel value changing step of overwriting the brightness and the color difference value of the determined pixel with the brightness and the color difference value of the character pixel, wherein the character information is enlarged by the enlargement amount.

16. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table, respectively;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

an enlarging step of enlarging coordinate values of the character pixel by a predetermined enlargement amount;

an after movement pixel determining step of determining the pixel corresponding to the enlarged position; and

an enlarged pixel value changing step of overwriting the brightness and the color difference value of the determined pixel with the brightness and the color difference value of the character pixel, respectively, wherein the character information is enlarged by the enlargement amount.

17. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel, respectively;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

a non-attention attracting area recognizing step of recognizing a non-attention attracting area in the video stream;

a separation distance detecting step of calculating a separation distance between the non-attention attracting area and the character information;

a character pixel after movement position determining step of obtaining a position of the character pixel when the character pixel is moved by the separation distance;

an after movement pixel determining step of determining the pixel corresponding to the determined after movement position; and

a destination pixel value changing step of overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

18. The video processing method according to claim 2, further comprising:

a pixel value interpolating step of interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table, respectively;

a frame signal correcting step of overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

a non-attention attracting area recognizing step of recognizing a non-attention attracting area in the video stream;

a separation distance detecting step of calculating a separation distance between the non-attention attracting area and the character information;

a character pixel after movement position determining step of obtaining a position of the character pixel when the character pixel is moved by the separation distance;

an after movement pixel determining step of determining the pixel corresponding to the determined after movement position; and

a destination pixel value changing step of overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

19. A video processing device that selectively processes character information included in an inputted video stream, the video processing device comprising:

video stream separating means for separating the video stream into a brightness frame representing brightness information in units of frame and a plurality of color difference frames representing color difference information;

character recognizing means for recognizing the included character information based on at least either of the brightness information and the color difference information; and

character processing means for performing at least one of deletion, movement, and enlargement on the recognized character information.

20. The video processing device according to claim 19, wherein the character recognizing means comprises:

brightness determining means for comparing a brightness of a pixel in the brightness frame with a threshold value;

character pixel recognizing means for recognizing a pixel having a brightness higher than the threshold value, as a character pixel representing the character information;

character pixel registering means for registering the recognized character pixel in a character display pixel table; and

character information frame registering means for registering a frame where at least one recognized character pixel is present, in a character display frame table as a frame including the character information.

21. The video processing device according to claim 20, wherein the character recognizing means further comprises:

color difference value determining means for determining whether a color difference value of the character pixel registered in the character display pixel table is within a predetermined range; and

character same color recognizing means for recognizing that the character information is the same color when it is determined that the color difference value is within the predetermined range.

22. The video processing device according to claim 20, wherein the character recognizing means further comprises:

pixel selecting means for selecting the pixel whose brightness is compared with the threshold value, by the brightness determining means.

23. The video processing device according to claim 20, wherein the character recognizing means further comprises:

inter-character pixel distance calculating means for calculating separation distances between the character pixels registered in the character display pixel table;

average separation distance calculating means for calculating an average separation distance by averaging the calculated separation distances; and

non-character pixel recognizing means for deleting the character pixel whose separation distance from the adjoining character pixel is larger than the average separation distance by a predetermined distance, from the character display table.

24. The video processing device according to claim 20, wherein the character recognizing means further comprises:

horizontal direction character pixel continuity detecting means for detecting continuity, in a horizontal direction, of the character pixel registered in the character display pixel table;

vertical direction character pixel continuity detecting means for detecting continuity, in a vertical direction, of the character pixel registered in the character display pixel table; and

character line width recognizing means for recognizing a character line width based on the continuity in the horizontal direction and the continuity in the vertical direction.

25. The video processing device according to claim 20, wherein the character recognizing means further comprises:

peripheral pixel color difference value determining means for determining whether the color difference values of peripheral pixels of the character pixel registered in the character display pixel table are uniform; and

edge recognizing means for recognizing that the character information is edged with a uniform color when it is determined that the color difference values of the peripheral pixels are uniform.

26. The video processing device according to claim 20, wherein the character recognizing means further comprises:

character information occurrence period detecting means for detecting an occurrence period of the character information registered in the character display frame table;

character information non-occurrence period detecting means for detecting a non-occurrence period of the character information registered in the character display frame table; and

character information blinking display recognizing means for recognizing that a display method of the character information is blinking display in which display and deletion are repeated at constant time intervals, based on the detected occurrence period and non-occurrence period.

27. The video processing device according to claim 20, wherein the character recognizing means further comprises:

first pixel distribution detecting means for detecting a first distribution of the character pixel in an arbitrary first frame on a time axis, based on the character display pixel table and the character display frame table;

second pixel distribution detecting means for detecting a second distribution of the character pixel in an arbitrary second frame on a different time axis from the first frame, based on the character display pixel table and the character display frame table;

third pixel distribution detecting means for detecting a third distribution of the character pixel on an arbitrary time axis between the first frame and the second frame, based on the first distribution;

distribution difference detecting means for detecting a difference between the second distribution and the third distribution; and

moving display recognizing means for recognizing that a display method of the character information is moving display in which the character information moves at a constant speed, based on the difference.

28. The video processing device according to claim 20, further comprising:

pixel value interpolating means for linearly interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel; and

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

29. The video processing device according to claim 20, further comprising:

pixel value interpolating means for interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table, respectively; and

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

30. The video processing device according to claim 27, further comprising:

pixel value interpolating means for interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table; and

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value, wherein the character information is deleted from the overwritten brightness signal and color difference signal.

31. The video processing device according to claim 20, further comprising:

pixel value interpolating means for linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

character pixel after movement position determining means for obtaining a position of the character pixel registered in the character display pixel table when the character pixel is moved by a predetermined distance;

after movement pixel determining means for determining the pixel corresponding to the determined after movement position; and

destination pixel value changing means for overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

32. The video processing device according to claim 20, further comprising:

pixel value interpolating means for interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

character pixel after movement position determining means for obtaining a position of the character pixel registered in the character display pixel table when the character pixel is moved by a predetermined distance;

after movement pixel determining means for determining the pixel corresponding to the determined after movement position; and

destination pixel value changing means for overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

33. The video processing device according to claim 20, further comprising:

pixel value interpolating means for linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel, respectively;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

enlarging means for enlarging coordinate values of the character pixel by a predetermined enlargement amount;

after movement pixel determining means for determining the pixel corresponding to the enlarged position; and

enlarged pixel value changing means for overwriting the brightness and the color difference value of the determined pixel with the brightness and the color difference value of the character pixel, respectively, wherein the character information is enlarged by the enlargement amount.

34. The video processing device according to claim 20, further comprising:

pixel value interpolating means for interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

enlarging means for enlarging coordinate values of the character pixel by a predetermined enlargement amount;

after movement pixel determining means for determining the pixel corresponding to the enlarged position; and

enlarged pixel value changing means for overwriting the brightness and the color difference value of the determined pixel with the brightness and the color difference value of the character pixel, respectively, wherein the character information is enlarged by the enlargement amount.

35. The video processing device according to claim 20, further comprising:

pixel value interpolating means for linearly interpolating the brightness and the color difference value of the character pixel registered in the character display pixel table, based on the brightness and the color difference value of the pixel adjoining the character pixel, respectively;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the linearly interpolated brightness and color difference value;

non-attention attracting area recognizing means for recognizing a non-attention attracting area in the video stream;

separation distance detecting means for calculating a separation distance between the non-attention attracting area and the character information;

character pixel after movement position determining means for obtaining a position of the character pixel when the character pixel is moved by the separation distance;

after movement pixel determining means for determining the pixel corresponding to the determined after movement position; and

destination pixel value changing means for overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

36. The video processing device according to claim 20, further comprising:

pixel value interpolating means for interpolating a brightness and a color difference value of the character pixel registered in the character display pixel table in the frame registered in the character display frame table, by the brightness and the color difference value of the corresponding pixel in a frame, close in time, not registered in the character display frame table;

frame signal correcting means for overwriting the brightness signal and the color difference signal of the character pixel with the interpolated brightness and color difference value;

non-attention attracting area recognizing means for recognizing a non-attention attracting area in the video stream;

separation distance detecting means for calculating a separation distance between the non-attention attracting area and the character information;

character pixel after movement position determining means for obtaining a position of the character pixel when the character pixel is moved by the separation distance;

after movement pixel determining means for determining the pixel corresponding to the determined after movement position; and

destination pixel value changing means for overwriting the brightness and the color difference value of the determined pixel, with the brightness and the color difference value of the character pixel, respectively, wherein the character information is moved by a predetermined distance.

37. A video playback device comprising playback means for playing back a video stream processed by the video processing device according to claim 20.

38. A video playback display device according to claim 37, further comprising presenting means for presenting an audiovisual image based on the played back video stream.

39. A video recording device having recording means for recording a video stream processed by the video processing device according to claim 20.