Image processing apparatus and method, and its computer program and storage medium

• US 20010048770A1
• Filed: 06/05/2001
• Published: 12/06/2001
• Est. Priority Date: 06/06/2000
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Maeda, Mitsuru
• Granted Patent: US 6,909,810 B2
• Time in Patent Office: Days
• US Class Current: 382/243
• CPC Class Codes: H04N 19/13 Adaptive entropy coding, e.g. adaptive variable length coding [AVLC] or context adaptive binary arithmetic coding [CABAC]
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream
• CPC Class Codes: H04N 19/467 characterised by the embedded information being invisible, e.g. watermarking
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/63 using sub-band based transform, e.g. wavelets
• CPC Class Codes: H04N 19/647 using significance based coding, e.g. Embedded Zerotrees of Wavelets [EZW] or Set Partitioning in Hierarchical Trees [SPIHT]
• CPC Class Codes: H04N 19/649 the transform being applied to non rectangular image segments

Abstract

MPEG-4 encoded data is input, and a shape code decoder decodes shape data contained in the encoded image data to obtain ROI information contained in that image. The frequency transforms of the decoded image data are computed to generate transform coefficients. A bit shift unit bit-shifts transform coefficients, corresponding to the ROI, of the generated transform coefficients, to upper bit planes, stuffs “0”s in blank fields outside the ROI, which are generated by the bit shift process, and stuffs audio data from an audio buffer in blank fields within the ROI, which are generated by the bit shift process.

Claims

1. An image processing apparatus comprising:

image input means for inputting image data;

information input means for inputting information data;

region of interest setting means for setting a region of interest on the basis of the image data;

transformation means for generating transform coefficients by computing frequency transforms of the image data; and

control means for bit-shifting transform coefficients, which correspond to the region of interest, of the transform coefficients generated by said transformation means to upper bit planes, stuffing zeros in blank fields outside the region of interest, which are generated by the bit shift process, and stuffing the information data in blank fields within the region of interest, which are generated by the bit shift process.

2. The apparatus according to

claim 1

, further comprising quantization means for quantizing the transform coefficients generated by said transformation means.

3. The apparatus according to

claim 1

, wherein said transformation means executes discrete wavelet transformation.

4. The apparatus according to

claim 1

, wherein the information data is audio information.

5. The apparatus according to

claim 1

, wherein the information data is meta data which pertains to a description of the image data.

6. The apparatus according to

claim 1

, wherein the information data includes an Intellectual Property right information.

7. The apparatus according to

claim 1

, further comprising encoding means for decomposing the transform coefficients stuffed by said control means into bit planes, and encoding the bit planes.

8. An image processing apparatus comprising:

encoded data input means for inputting encoded data;

decoding means for decoding the encoded data input by said encoded data input means;

region of interest extraction means for extracting a region of interest from a decoding result decoded by said decoding means; and

information data extraction means for extracting information data from lower bit planes of the region of interest extracted by said region of interest extraction means.

9. An image processing method comprising:

an image input step of inputting image data;

an information input step of inputting information data;

a region of interest setting step of setting a region of interest on the basis of the image data;

a transformation step of generating transform coefficients by computing frequency transforms of the image data; and

a control step of bit-shifting transform coefficients, which correspond to the region of interest, of the transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region of interest, which are generated by the bit shift process, and stuffing the information data in blank fields within the region of interest, which are generated by the bit shift process.

10. The method according to

claim 9

, further comprising a quantization step of quantizing the transform coefficients generated in said transformation step.

11. The method according to

claim 9

, wherein in said transformation step, a discrete wavelet transformation is executed.

12. The method according to

claim 9

, wherein the information data is audio information.

13. The method according to

claim 9

, wherein the information data is meta data which pertains to a description of the image data.

14. The method according to

claim 9

, wherein the information data is an Intellectual Property right information.

15. The method according to

claim 9

, further comprising an encoding step of decomposing the transform coefficients output in said control step into bit planes, and encoding the bit planes.

17. A computer readable storage medium storing a program for implementing an image processing method according to

claim 9

.

16. An image processing method comprising:

an encoded data input step of inputting encoded data;

a decoding step of decoding the encoded data;

a region of interest extraction step of extracting a region of interest from a decoding result decoded in said decoding step; and

an information data extraction step of extracting information data from lower bit planes of the extracted region of interest.

18. A computer readable storage medium storing a program for implementing an image processing method according to

claim 16

.

19. An image processing apparatus comprising:

generation means for generating object image data which represents an object image, and background image data to be composed in a background of the object image;

transformation means for generating first transform coefficients by computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image, and generating second transform coefficients by computing frequency transforms of the background image data corresponding to at least the region of the object image; and

control means for bit-shifting bits, which correspond to the region of the object image, of the first transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the second transform coefficients corresponding to the interior of the region in blank fields within the region, which are generated by the bit shift process.

20. An image processing apparatus comprising:

generation means for generating object image data which represents an object image, and background image data to be composed in a background of the object image;

first transformation means for computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image;

second transformation means for computing frequency transforms of the background image data corresponding to at least the region of the object image; and

control means for bit-shifting bits, which correspond to the region of the object image, of the transform coefficients obtained by said first transformation means to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the transform coefficients, which are obtained by said second transformation means and correspond to the interior of the region, in blank fields within the region, which are generated by the bit shift process.

21. An image processing apparatus comprising:

shape information extraction means for extracting shape information of an object from image data;

object texture information extraction means for extracting texture information of the object from the image data;

background texture information extraction means for extracting texture information of a background from the image data;

first frequency transformation means for computing frequency transforms of the texture information of the object and the texture information of the background on the basis of the shape information extracted by said shape information extraction means;

second frequency transformation means for computing frequency transforms of the texture information of the background;

stuffing means for stuffing zeros in a region outside a region of the object on the basis of an output from said first frequency transformation means, and the shape information; and

bit plane encoding means for decomposing an output from said stuffing means into bit planes and encoding the bit planes, and decomposing an output from said second frequency transformation means into bit planes and encoding the bit planes.

22. The apparatus according to

claim 21

, further comprising quantization means for quantizing transform coefficients computed by said first and second frequency transformation means.

23. The apparatus according to

claim 21

, further comprising image input means for inputting the image data on the basis of a captured image.

24. The apparatus according to

claim 21

, wherein the image data is encoded data encoded by MPEG-4.

25. The apparatus according to

claim 21

, wherein at least one of said first and second frequency transformation means executes discrete wavelet transformation.

26. The apparatus according to

claim 21

, further comprising shape correction means for expanding the shape information on the basis of the shape information extracted by said shape information extraction means, and a frequency transformation scheme of said first frequency transformation means.

27. An image processing apparatus comprising:

input means for inputting encoded data;

first bit plane decoding means for decoding a first group of bit planes;

shape information extraction means for extracting shape information of an object from a decoding result of said first bit plane decoding means;

first inverse frequency transformation means for computing inverse frequency transforms of the decoding result of said first bit plane decoding means;

object texture information extraction means for extracting texture information of the object from a transformation result of said first inverse frequency transformation means;

second bit plane decoding means for decoding a second group of bit planes;

second inverse frequency transformation means for computing inverse frequency transforms of a decoding result of said second bit plane decoding means;

background texture information extraction means for extracting texture information of a background from a transformation result of said second inverse frequency transformation means;

object shape information encoding means for generating object shape information encoded data by encoding the shape information of the object;

object encoding means for generating texture encoded data of the object by encoding an output from said first inverse frequency transformation means;

background encoding means for generating texture encoded data of the background by encoding a transformation result of said second inverse frequency transformation means; and

output means for outputting, as object encoded data, the object shape encoded data, the texture encoded data of the object, and the texture encoded data of the background.

29. The apparatus according to

claim 27

, further comprising dequantization means for dequantizing the decoding results of said first and second bit plane decoding means.

30. The apparatus according to

claim 27

, wherein said first and second inverse frequency transformation means execute inverse discrete wavelet transformation.

31. The apparatus according to

claim 27

, further comprising shape information correction means for reducing the shape information on the basis of the shape information extracted by said shape information extraction means, and a transformation scheme of said first and second inverse frequency transformation means.

32. The apparatus according to

claim 27

, wherein the object encoded data is encoded data encoded by MPEG-4.

28. An image processing apparatus comprising:

input means for inputting encoded data;

first bit plane decoding means for decoding a first group of bit planes;

shape information extraction means for extracting shape information of an object from a decoding result of said first bit plane decoding means;

first inverse frequency transformation means for computing inverse frequency transforms of the decoding result of said first bit plane decoding means;

second bit plane decoding means for decoding a second group of bit planes;

object texture information extraction means for extracting texture information of the object from a transformation result of said first inverse frequency w transformation means;

second inverse frequency transformation means for computing inverse frequency transforms of a decoding result of said second bit plane decoding means; and

background texture extraction means for extracting texture information of a background from a transformation result of said second inverse frequency transformation means.

33. The apparatus according to

claim 28

, further comprising dequantization means for dequantizing the decoding results of said first and second bit plane decoding means.

34. The apparatus according to

claim 28

, wherein said first and second inverse frequency transformation means execute inverse discrete wavelet transformation.

35. The apparatus according to

claim 28

, further comprising shape information correction means for reducing the shape information on the basis of the shape information extracted by said shape information extraction means, and a transformation scheme of said first and second inverse frequency transformation means.

36. The apparatus according to

claim 28

, wherein the object encoded data is encoded data encoded by MPEG-4.

37. An image processing method comprising:

a generation step of generating object image data which represents an object image, and background image data to be composed in a background of the object image;

a transformation step of generating first transform coefficients by computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image, and generating second transform coefficients by computing frequency transforms of the background image data corresponding to at least the region of the object image; and

a control step of bit-shifting bits, which correspond to the region of the object image, of the first transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the second transform coefficients corresponding to the interior of the region in blank fields within the region, which are generated by the bit shift process.

51. A computer readable storage medium storing a program for implementing an image processing method according to

claim 37

.

38. An image processing method comprising:

a generation step of generating object image data which represents an object image, and background image data to be composed in a background of the object image;

a first transformation step of computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image;

a second transformation step of computing frequency transforms of the background image data corresponding to at least the region of the object image; and

a control step of bit-shifting bits, which correspond to the region of the object image, of the transform coefficients obtained in the first transformation step to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the transform coefficients, which are obtained in the second transformation step and correspond to the interior of the region, in blank fields within the region, which are generated by the bit shift process.

52. A computer readable storage medium storing a program for implementing an image processing method according to

claim 38

.

39. An image processing method comprising:

a shape information extraction step of extracting shape information of an object from image data;

an object texture information extraction step of extracting texture information of the object from the image data;

a background texture information extraction step of extracting texture information of a background from the image data;

a first frequency transformation step of computing frequency transforms of the texture information of the object and the texture information of the background on the basis of the shape information extracted in said shape information extraction step;

a second frequency transformation step of computing frequency transforms of the texture information of the background;

a stuffing step of stuffing zeros in a region outside a region of the object on the basis of an output by said first frequency transformation step, and the shape information; and

a bit plane encoding step of decomposing an output of the stuffing step into bit planes and encoding the bit planes, and decomposing an output of the second frequency transformation step into bit planes and encoding the bit planes.

40. The method according to

claim 39

, further comprising a quantization step of quantizing transform coefficients computed in said first and second frequency transformation steps.

41. The method according to

claim 39

, further comprising an image input step of inputting the image data on the basis of a captured image.

42. The method according to

claim 39

, wherein the image data is encoded data encoded by MPEG-4.

43. The method according to

claim 39

, wherein at least one of said first and second frequency transformation steps, a discrete wavelet transformation is executed.

44. The method according to

claim 39

, further comprising a shape correction step of expanding the shape information on the basis of the shape information extracted in said shape information extraction step, and a frequency transformation scheme in said first frequency transformation step.

53. A computer readable storage medium storing a program for implementing an image processing method according to

claim 39

.

45. An image processing method comprising:

an input step of inputting encoded data;

a first bit plane decoding step of decoding a first group of bit planes;

a shape information extraction step of extracting shape information of an object from a decoding result of the first bit plane decoding step;

a first inverse frequency transformation step of computing inverse frequency transforms of the decoding result of the first bit plane decoding step;

an object texture information extraction step of extracting texture information of the object from a transformation result of the first inverse frequency transformation step;

a second bit plane decoding step of decoding a second group of bit planes;

a second inverse frequency transformation step of computing inverse frequency transforms of a decoding result in said second bit plane decoding step;

a background texture information extraction step of extracting texture information of a background from a transformation result in said second inverse frequency transformation step;

an object shape information encoding step of generating object shape information encoded data by encoding the shape information of the object;

an object encoding step of generating texture encoded data of the object by encoding an output from said first inverse frequency transformation step;

a background encoding step of generating texture encoded data of the background by encoding a transformation result in said second inverse frequency transformation step; and

an output step of outputting, as object encoded data, the object shape encoded data, the texture encoded data of the object, and the texture encoded data of the background.

47. The method according to

claim 45

, further comprising a dequantization step of dequantizing the decoding results in said first and second bit plane decoding steps.

48. The method according to

claim 45

, wherein in said first and second inverse frequency transformation steps, an inverse discrete wavelet transformation is executed.

49. The method according to

claim 45

, further comprising a shape information correction step of reducing the shape information on the basis of the shape information extracted in said shape information extraction step, and a transformation scheme at said first and second inverse frequency transformation steps.

50. The method according to

claim 45

, wherein the object encoded data is encoded data encoded by MPEG-4.

46. An image processing method comprising:

a step of inputting encoded data;

a first bit plane decoding step of decoding a first group of bit planes;

a shape information extraction step of extracting shape information of an object from a decoding result in said first bit plane decoding step;

a first inverse frequency transformation step of computing inverse frequency transforms of the decoding result in said first bit plane decoding step;

an object texture information extraction step of extracting texture information of the object from a transformation result in said first inverse frequency transformation step;

a second bit plane decoding step of decoding a second group of bit planes;

a second inverse frequency transformation step of computing inverse frequency transforms of a decoding result in said second bit plane decoding step; and

a background texture extraction step of extracting texture information of a background from a transformation result in said second inverse frequency transformation step.

54. A computer program comprising:

an image input program code for inputting image data;

an information input program code for inputting information data;

a region of interest setting program code for setting a region of interest on the basis of the image data;

a transformation program code for generating transform coefficients by computing frequency transforms of the image data; and

a control program code for bit-shifting transform coefficients, which correspond to the region of interest, of the transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region of interest, which are generated by the bit shift process, and stuffing the information data in blank fields within the region of interest, which are generated by the bit shift process.

55. A computer program comprising:

an encoded data input program code for inputting encoded data;

a decoding program code for decoding the encoded data;

a region of interest extraction program code for extracting a region of interest from the decoding result; and

an information data extraction program code for extracting information data from lower bit planes of the extracted region of interest.

56. A computer program comprising:

a generation program code for generating object image data which represents an object image, and background image data to be composited in a background of the object image;

a transformation program code for generating first transform coefficients by computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image, and generating second transform coefficients by computing frequency transforms of the background image data corresponding to at least the region of the object image; and

a control program code for bit-shifting bits, which correspond to the region of the object image, of the first transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the second transform coefficients corresponding to the interior of the region in blank fields within the region, which are generated by the bit shift process.

57. A computer program comprising:

a generation program code for generating object image data which represents an object image, and background image data to be composited in a background of the object image;

a first transformation program code for computing frequency transforms of the object image data and the background image data corresponding to a region outside a region of the object image;

a second transformation program code for computing frequency transforms of the background image data corresponding to at least the region of the object image; and

a control program code for bit-shifting bits, which correspond to the region of the object image, of the obtained transform coefficients to upper bit planes, stuffing zeros in blank fields outside the region, which are generated by the bit shift process, and stuffing the transform coefficients, which are obtained by executing the second transformation program code and correspond to the interior of the region, in blank fields within the region, which are generated by the bit shift process.

58. A computer program comprising:

a shape information extraction program code for extracting shape information of an object from image data;

an object texture information extraction program code for extracting texture information of the object from the image data;

a background texture information extraction program code for extracting texture information of a background from the image data;

a first frequency transformation program code for computing frequency transforms of the texture information of the object and the texture information of the background on the basis of the shape information extracted by the shape information extraction program code;

a second frequency transformation program code for computing frequency transforms of the texture information of the background;

a stuffing program code for stuffing zeros in a region outside a region of the object on the basis of an output of the first frequency transformation program code, and the shape information; and

a bit plane encoding program code for decomposing an output of the stuffing program code into bit planes and encoding the bit planes, and decomposing an output of the second frequency transformation program code into bit planes and encoding the bit planes.

59. A computer program comprising:

an input program code for inputting encoded data;

a first bit plane decoding program code for decoding first bit planes;

a shape information extraction program code for extracting shape information of an object from a decoding result of the first bit planes;

a first inverse frequency transformation program code for computing inverse frequency transforms of the decoding result of the first bit plane decoding program code;

an object texture information extraction program code for extracting texture information of the object from a transformation result of the first inverse frequency transformation program code;

a second bit plane decoding program code for decoding second bit planes;

a second inverse frequency transformation program code for computing inverse frequency transforms of a decoding result of the second bit plane decoding program code;

a background texture information extraction program code for extracting texture information of a background from a transformation result of the second inverse frequency transformation program code;

an object shape information encoding program code for generating object shape information encoded data by encoding the shape information of the object;

an object encoding program code for generating texture encoded data of the object by encoding an output of the first inverse frequency transformation program code;

a background encoding program code for generating texture encoded data of the background by encoding an output of the second inverse frequency transformation program code; and

an output program code for outputting, as object encoded data, the object shape encoded data, the texture encoded data of the object, and the texture encoded data of the background.

60. A computer program comprising:

a program code for inputting encoded data;

a first bit plane decoding program code for decoding first bit planes;

a shape information extraction program code for extracting shape information of an object from a decoding result of the first bit plane decoding program code;

a first inverse frequency transformation program code for computing inverse frequency transforms of the decoding result of the first bit plane decoding program code;

an object texture information extraction program code for extracting texture information of the object from a transformation result of the first inverse frequency transformation program code;

a second inverse frequency transformation program code for computing inverse frequency transforms of a decoding result of a second bit plane decoding program code; and

a background texture extraction program code for extracting texture information of a background from a transformation result of the second inverse frequency transformation program code.