Image signal converting method and image signal converting apparatus

• US 5,946,044 A
• Filed: 06/20/1996
• Issued: 08/31/1999
• Est. Priority Date: 06/30/1995
• Status: Expired due to Term

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kondo, Tetsujiro, Uchida, Masashi, Fujimori, Yasuhiro, Nakaya, Hideo
• Primary Examiner(s): Faile, Andrew I.
• Assistant Examiner(s): Srivastava, Vivek
• Application Number:
  US08/667,846
• Time in Patent Office: 1,167 Days
• Field of Search: 348/458, 348/409, 348/410, 348/411, 348/412, 348/413, 348/407, 348/405, 348/402, 348/401, 348/400, 348/390, 348/384, 348/438, 348/394, 348/397, 348/399, 348/415, 348/416 , 348/403, 348/404, 348/406, 348/408, 382/299, 382/300
• US Class Current: 348/458
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/98 Adaptive-dynamic-range coding [ADRC]
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/0135 involving interpolation processes interpolation-based image scaling G06T3/4007; interpolation for video coding H04N19/587, H04N19/59
• CPC Class Codes: H04N 7/0137 dependent on presence/absence of motion, e.g. of motion zones H04N7/014 takes precedence; movement detection in television signals H04N5/144
• CPC Class Codes: H04N 7/0145 the interpolation being class adaptive, i.e. it uses the information of class which is determined for a pixel based upon certain characteristics of the neighbouring pixels

Abstract

In an image signal converting method and an image signal converting apparatus, it is possible to perform a proper classification considering various signal characteristics of input image data and convert low-resolution input image data to high-resolution image data by using not only a classifying means 56 for classifying input images in accordance with the level distribution pattern of the input images but also a classifying means 50 for classifying input images in accordance with the movement of the input images and deciding the final class D32 of input image signals by both CLASS0 and CLASS1 obtained by both classifying means 50 and 56.

Claims

1. An image signal converting apparatus for converting a first input image signal to a second image signal different from said first image signal, comprising:

a class code deciding means for deciding the class of a noticed pixel in accordance with the movement indicated by a change of at least one pixel in the vicinity of the noticed pixel between at least a first input image and a second input image of said first input image signal and patterns of at least one pixel in the vicinity of the noticed pixel in at least one of said first or second input images of said first input image signal, and generating a class code showing the class;

a predictive coefficient data storage means for storing predictive coefficient data serving as the information for converting said first input image signal to said second input image signal for each class code to output said predictive coefficient data corresponding to said class code; and

a predictive operating means for generating said second image signal by using the predictive coefficient data output from said predictive coefficient data storage means corresponding to said class code and a plurality of pixels spatially and/or temporally present in the vicinity of said noticed pixel and thereby performing predictive operation.

2. The image signal converting apparatus according to claim 1, wherein

said class code deciding means, comprising:

a first classifying means for deciding the first class of said noticed pixel in accordance with the movement of the circumference of said noticed pixel in said first input image data to output a first class code showing the first class; and

a second classifying means for deciding the second class of said noticed pixel in accordance with level distribution patterns of a plurality of pixel spatially and/or temporally present at the circumference of said noticed pixel to output a second class code showing the second class;

wherein

a class code is generated based on said first class code and said second class code.

3. The image signal converting apparatus according to claim 1, wherein

said first input image signal is a low-resolution image signal and said second image signal is a higher-resolution image signal than said first input image signal.

4. The image signal converting apparatus according to claim 1, wherein

said second image signal is an image signal having larger number of pixels than said first image signal.

5. The image signal converting apparatus according to claim 2, including

a classified pixel selecting means for adaptively selecting said pixels used for deciding the second class by said second classifying means corresponding to said first class code.

6. The image signal converting apparatus according to claim 2, including

a predictively-operated pixel selecting means for adaptively selecting said pixels used for predictive operation by said predictive operating means corresponding to said first class code.

7. The image signal converting apparatus according to claim 2, including:

a classified pixel selecting means for adaptively selecting said pixels used for deciding the second class by said second classifying means corresponding to said first class code; and

a predictively-operated pixel selecting means for adaptively selecting said pixels used for predictive operation by said predictive operating means corresponding to said first class code.

8. The image signal converting apparatus according to claim 5, wherein:

said first classifying means decides the first class of said noticed pixels in accordance with motion vectors of images around said noticed pixel; and

said classified pixel selecting means selects said pixels long extended in the direction of movement and distributed in wider area for larger movement.

9. The image signal converting apparatus according to claim 6, wherein:

said first classifying means decides the first class of said noticed pixel in accordance with motion vectors of the images around said noticed pixel; and

said classified pixel selecting means selects said pixels long extended in the direction of movement and distributed in wider area for larger movement.

10. The image signal converting apparatus according to claim 5, wherein:

said first classifying means decides the first class of said noticed pixel in accordance with if images around said noticed pixel move; and

said classified pixel selecting means selects said pixels present in the same field when said images move and selects said pixels present in a plurality of fields when they do not move.

11. The image signal converting apparatus according to claim 6, wherein:

said first classifying means decides the first class of said noticed pixel in accordance with if images around said noticed pixel move; and

said classified pixel selecting means selects said pixels present in the same field when said images move and selects said pixels present in a plurality of fields when they do not move.

12. The image signal converting apparatus according to claim 5, wherein

said first classifying means includes:

a predicted value calculating means having a plurality of coefficients previously selected by considering the movement of a plurality of types of images, for calculating a plurality of predicted values of said noticed pixel by using the coefficients and the pixel data around said noticed pixel;

a difference value calculating means for calculating difference values between a plurality of predicted values calculated by said predicted value calculating means and the pixel values of said noticed pixel; and

a first class code generating means for judging the movement around said noticed pixel based on said difference values and outputting said first class code.

13. The image signal converting apparatus according to claim 6, wherein

said first classifying means includes:

a predicted value calculating means having a plurality of coefficients previously selected by considering the movement of a plurality of types of images, for calculating a plurality of predicted values of said noticed pixel by using the coefficients and the pixel data around said noticed pixel;

a difference value calculating means for calculating difference values between a plurality of predicted values calculated by said predicted value calculating means and the pixel values of said noticed pixel; and

a first class code generating means for judging the movement around said noticed pixel based on said difference values and outputting said first class code.

14. The image signal converting apparatus according to claim 5, wherein

said first classifying means includes:

a first blocking means for forming a predetermined-size block mainly including said noticed pixel in the present frame or field of said input image data;

a second blocking means for forming a plurality of blocks arranged in a plurality of directions about the position of said block formed in said present frame or field in a past frame or field of said input image data;

a difference value calculating means for calculating difference values between pixels of said block formed by said first blocking means and those of said blocks formed by said second blocking means; and

a first class code generating means for judging the movement around said noticed pixel based on said difference values and outputting said first class code.

15. The image signal converting apparatus according to claim 6, wherein

said first classifying means includes:

a first blocking means for forming a predetermined-size block mainly including said noticed pixel in the present frame or field of said input image data;

a second blocking means for forming a plurality of blocks arranged in a plurality of directions about the position of said block formed in said present frame or field in a past frame or field of said input image data;

a difference value calculating means for calculating difference values between pixels of said block formed by said first blocking means and those of said blocks formed by said second blocking means; and

a first class code generating means for judging the movement around said noticed pixel based on said differences and outputting said first class code.

16. The image signal converting apparatus according to claim 5, wherein

said first classifying means outputs said first class code by calculating the inter-frame difference value between a predetermined-size block allocated from the frame including said noticed pixel and a block allocated from a frame adjacent to said frame and at the spatially same position as said block, comparing said inter-frame difference value with a predetermined threshold value, and judging the movement around said noticed pixel in accordance with said comparison result.

17. The image signal converting apparatus according to claim 6, wherein

said first classifying means outputs said first class code by calculating the inter-frame difference value between a predetermined-size block allocated from the frame including said noticed pixel and a block allocated from a frame adjacent to said frame and at the spatially same position as said block, comparing said inter-frame difference value with a predetermined threshold value, and judging the movement around said noticed pixel in accordance with said comparison result.

18. An image signal converting apparatus for converting a first input image signal to a second image signal different from said first image signal, comprising:

a class code deciding means for deciding the class of a noticed pixel in accordance with the movement indicated by a change of at least one pixel in the vicinity of the noticed pixel between at least a first input image and a second input image of said first input image signal and patterns of at least one pixel in the vicinity of the noticed pixel in at least one of said first or second input images of said first input image signal, and generating a class code showing the class; and

a predicted value data storage means for storing predicted value data for converting said first input image signal to said second input image signal for each class code to output said predicted value data corresponding to said class code.

19. The image signal converting apparatus according to claim 18, wherein

said class code deciding means, comprising:

a first classifying means for deciding the first class of said noticed pixel in accordance with the movement of the circumference of said noticed pixel in said first input image data to output a first class code showing the first class;

a second classifying means for deciding the second class of said noticed pixel in accordance with level distribution patterns of a plurality of pixel spatially and/or temporally present at the circumference of said noticed pixels to output a second class code showing the second class;

wherein

a class code is generated based on said first class code and said second class code.

20. The image signal converting apparatus according to claim 18, wherein

said first input image signal is a low-resolution image signal and said second image signal is a higher-resolution image signal than said first input image signal.

21. The image signal converting apparatus according to claim 18, wherein

said second image signal is an image signal having larger number of pixels than said first image signal.

22. The image signal converting apparatus according to claim 19, including

a classified pixel selecting means for adaptively selecting said pixels used for deciding the second class by said second classifying means corresponding to said first class code.

23. An image signal converting method for converting a first input image signal to a second image signal different from said first image signal, comprising the steps of:

deciding the class of a noticed pixel in accordance with the movement indicated by a change of at least one pixel in the vicinity of the noticed pixel between at least a first input image and a second input image of said first input image signal and patterns of at least one pixel in the vicinity of the noticed pixel in at least one of said first or second input images of said first input image signal, and generating a class code showing the class;

storing predictive coefficient data for converting said first input image signal to said second input image signal for each class code, to output said predictive coefficient data corresponding to said class code; and

generating said second image signal by using the predictive coefficient data output from said predictive coefficient storage step corresponding to said class code and a plurality of pixels spatially and/or temporally present in the vicinity of said noticed pixel and thereby performing predictive operation.

24. The image signal converting method according to claim 23, wherein

said class code deciding step, comprising:

a first classifying step for deciding the first class of said noticed pixel in accordance with the movement of the circumference of said noticed pixel in said first input image data to output a first class code showing the first class;

a second classifying step for deciding the second class of said noticed pixel in accordance with level distribution patterns of a plurality of pixels spatially and/or temporally present at the circumference of said noticed pixel to output a second class code showing the second class;

wherein

a class code is generated based on said first class code and said second class code.

25. The image signal converting method according to claim 23, wherein

said first input image signal is a low-resolution image signal and said second image signal is a higher-resolution image signal than said first input image signal.

26. The image signal converting method according to claim 23, wherein

said second image signal is an image signal having larger number of pixels than said first image signal.

27. The image signal converting method according to claim 23, including

a step for adaptively selecting said pixels used for deciding the second class corresponding to said first class code.

28. The image signal converting method according to claim 23, including

a step for adaptively selecting said pixels used for predictive operation corresponding to said first class code.

29. The image signal converting method according to claim 23, including:

a step for adaptively selecting said pixels used for generating said second class code corresponding to said first class code; and

a step for adaptively selecting said pixels used for predictive operation corresponding to said first class code.

30. The image signal converting method according to claim 27, wherein:

said first classifying step decides the first class of said noticed pixel in accordance with motion vectors of images around said noticed pixel; and

the pixels long extended in the direction of movement and distributed in wider area for larger movement are selected as said plural pixels used for deciding said second class code.

31. The image signal converting method according to claim 28, wherein:

said first classifying step decides the first class of said noticed pixel in accordance with motion vectors of the images around said noticed pixel; and

the pixels long extended in the direction of movement and distributed in wider area for larger movement are selected as said plural pixels used for deciding a second class code.

32. The image signal converting method according to claim 28, wherein:

said first classifying step decides the first class of said noticed pixel in accordance with if images around said noticed pixel move; and

the pixels present in the same field are selected when said images move and the pixels present in a plurality of fields are selected when they do not move.

33. The image signal converting method according to claim 28, wherein:

said first classifying step decides the first class of said noticed pixel in accordance with if images around said noticed pixel move; and

the pixels present in the same field are selected when said images move and the pixels present in a plurality of fields are selected when they do not move.

34. The image signal converting method according to claim 27, wherein

said first classifying step includes the steps of:

calculating a plurality of predicted values of said noticed pixel by using the coefficients and the pixel data around said noticed pixel by considering the movement of a plurality of types of images;

calculating difference values between a plurality of predicted values and the pixel values of said noticed pixel; and

judging the movement around said noticed pixels based on said difference values and outputting said first class code in accordance with said judged result.

35. The image signal converting method according to claim 28, wherein

said first classifying step includes the steps of:

calculating a plurality of predicted values of said noticed pixel by using the coefficients and the pixel data around said noticed pixel by considering the movement of a plurality of types of images;

calculating difference values between said plurality of predicted values thus calculated and the pixel values of said noticed pixels; and

judging the movement around said noticed pixel based on said difference values and outputting said first class code in accordance with said judged result.

36. The image signal converting method according to claim 27, wherein

said first classifying step includes the steps of:

forming a first predetermined-size block mainly including said noticed pixel in the present frame or field of said input image data;

forming a plurality of second blocks arranged in a plurality of directions about the position of said block formed in said present frame or field in a past frame or field of said input image data;

calculating difference values between pixels of said first block and said second blocks respectively; and

judging the movement around said noticed pixels based on said difference values and outputting said first class code in accordance with said judged result.

37. The image signal converting method according to claim 28, wherein

said first classifying step includes the steps of:

forming a first predetermined-size block mainly including said noticed pixel in the present frame or field of said input image data;

forming a plurality of second blocks arranged in a plurality of directions about the position of said block formed in said present frame or field in a past frame or field of said input image data;

calculating difference values between pixels of said first block and said second blocks respectively; and

judging the movement around said noticed pixels based on said differences and outputting said first class code in accordance with said judged result.

38. The image signal converting method according to claim 27, wherein

said first classifying step comprising the steps of:

calculating the inter-frame difference value between a predetermined-size block allocated from the frame including said noticed pixel and a block allocated from a frame adjacent to said frame and at the spatially same position as said block;

comparing said inter-frame difference value with a predetermined threshold value; and

judging the movement around said noticed pixels in accordance with said comparison result and outputting said first class code in accordance with said judged result.

39. The image signal converting method according to claim 28, wherein

said first classifying step comprising the steps of:

calculating the inter-frame difference value between a predetermined-size block allocated from the frame including said noticed pixel and a block allocated from a frame adjacent to said frame and at the spatially same position as said block;

comparing said inter-frame difference value with a predetermined threshold value; and

judging the movement around said noticed pixels in accordance with said comparison result and outputting said first class code in accordance with said judged result.

40. An image signal converting method for converting a first input image signal to a second image signal different from said first image signal, comprising the steps of:

deciding the class of a noticed pixel in accordance with the movement indicated by a change of at least one pixel in the vicinity of the noticed pixel between at least a first input image and a second input image of said first input image signal and patterns of at least one pixel in the vicinity of the noticed pixel in at least one of said first or second input images of said first input image signal, and generating a class code showing the class; and

storing predicted value data for converting said first input image signal to said second input image signal for each class code to output said predicted value data corresponding to said class code.

41. The image signal converting method according to claim 40, wherein

said class code deciding step, comprising the steps of:

a first classifying step for deciding the first class of said noticed pixel in accordance with the movement of the circumference of said noticed pixel in said first input image data to output a first class code showing the first class;

a second classifying step for deciding the second class of said noticed pixel in accordance with level distribution patterns of a plurality of pixels spatially and/or temporally present at the circumference of said noticed pixels and outputting a second class code showing the second class;

wherein

the class code is generated based on said first class code and said second class code.

42. The image signal converting method according to claim 40, wherein

said first input image signal is a low-resolution image signal and said second image signal is a higher-resolution image signal than said first input image signal.

43. The image signal converting method according to claim 40, wherein

said second image signal is an image signal having larger number of pixels than said first image signal.

44. The image signal converting method according to claim 41, including

a step for adaptively selecting said pixels used for generating said second class code corresponding to said first class code.