Method and apparatus for coding and decoding image data

• US 7,085,419 B2
• Filed: 05/15/2002
• Issued: 08/01/2006
• Est. Priority Date: 05/15/2001
• Status: Expired due to Fees

• Current Assignee: JBF Partners Incorporated
• Original Assignee: Monolith Company Limited
• Inventors: Akiyoshi, Kozo, Akiyoshi, Nobuo
• Primary Examiner(s): Do, Anh Hong
• Application Number:
  US10/144,804
• Publication Number:
  US 20030043909A1
• Time in Patent Office: 1,539 Days
• Field of Search: 382/232, 382/233, 382/291, 382/209, 382/266, 382/151, 382/195, 382/217, 382/218, 382/107, 358/1.14, 348/208.4, 348/155, 348/207.99, 717/144, 717/156, 235/462.11
• US Class Current: 382/232
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/523 with sub-pixel accuracy
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

A matching processor detects correspondence information between key frames. A judging unit detects motion vectors from the correspondence information to determine variation of a field of view between the key frames. When variation of the field of view is sufficiently large, added information is generated related to the variation of the field of view. A stream generator generates a coded data stream by incorporating the added information. An intermediate image generator (for example, at a decoding side) detects the added information and trims edges of a generated intermediate image to account for the variation in the field of view, so that an unnatural invalidated region is cut out.

Claims

1. An image coding method, comprising:

detecting corresponding point data between key frames by image matching;

determining added information by which to invalidate at least a region of a reproduction image which is to be generated at a decoding side based on the corresponding point data; and

generating a coded data stream incorporating the corresponding point data and added information.

2. An image coding method according to claim 1, wherein the added information includes indication of a region to be invalidated.

3. An image coding method according to claim 1, wherein the added information includes image data by which to replace data of a region to be invalidated.

4. An image coding method according to claim 1, wherein the corresponding point data are incorporated into the coded data stream after data which correspond to a region to be invalidated are deleted from the corresponding point data.

5. An image coding method according to claim 1, wherein the key frames are photographed by a camera, the method further comprising:

judging the magnitude of variation of field of view of the camera which photographed the key frames, based on the corresponding point data,

wherein the added information is effectively generated or effectively incorporated only when it is judged that the variation of the field of view is substantially large.

6. An image coding method according to claim 1, wherein the image matching is computed in a manner such that critical points are extracted in each key frame and thereby correspondence of the critical points between the key frames is evaluated.

7. An image coding method, comprising:

detecting corresponding point data between key frames by image matching; and

incorporating data of the key frames having at least a region that is invalidated, and the detected corresponding point data, so as to generate a coded data stream.

8. An image coding method according to claim 7, wherein the corresponding point data are incorporated into the coded data stream in a form such that data which correspond to the invalidated region are deleted from the corresponding point data.

9. An image coding method according to claim 7, wherein the key frames are photographed by a camera, the method further comprising:

judging the magnitude of variation of field of view of the camera which photographed the key frames, based on the detected corresponding point data,

wherein the region is invalidated only when it is judged that the variation of the field of view is substantially large.

10. An image coding method according to claim 7, wherein the image matching is computed in a manner such that critical points are extracted in each key frame and thereby correspondence of the critical points between the key frames is evaluated.

11. An image coding method, comprising:

detecting corresponding point data between key frames by image matching;

determining added information by which to invalidate at least a region of a reproduction image which is to be generated at a decoding side based on the detected corresponding point data, and incorporating the added information into the corresponding point data; and

generating a coded data stream comprising the corresponding point data.

12. An image coding method according to claim 11, wherein the added information includes indication of a region to be invalidated.

13. An image coding method according to claim 11, wherein the added information includes image data by which to replace data of a region to be invalidated.

14. An image coding method according to claim 11, wherein the corresponding point data are incorporated into the coded data stream after data which correspond to a region to be invalidated are deleted from the corresponding point data.

15. An image coding method according to claim 11, wherein the key frames are photographed by a camera, the method further comprising:

judging the magnitude of movement of viewpoint of the camera which photographed the key frames, based on the detected corresponding point data,

wherein the added information is effectively generated only when it is judged that the movement of the viewpoint is substantially large.

16. An image coding method according to claim 11, wherein the image matching is computed in a manner such that a critical point is extracted from each key frame and thereby correspondence of the points between the key frames is evaluated.

17. An image coding method, comprising:

generating a coded data stream incorporating corresponding point data between key frames and added information by which to invalidate at least a region of an intermediate image which is to be generated at a decoding side based on the corresponding point data.

18. An image coding method according to claim 17, wherein a region to be invalidated includes left and right edges of the frames when motion vectors between corresponding points of the key frames are aligned approximately in one direction parallel to the top or bottom side of the frames.

19. An image coding method according to claim 18, wherein the width of the region to be invalidated is proportional to the magnitude of the motion vectors.

20. An image coding method according to claim 18, wherein whether or not the motion vectors are aligned approximately in one direction parallel to the top or bottom side of the frames is detected in a manner such that vectors in one direction are assumed and then the sum of differences between the assumed vectors and actual vectors are taken to determine whether or not the sum is less than a predetermined threshold value.

21. An image coding method according to claim 17, wherein a variation of the field of view between the key frames is presumed in advance for a plurality of directions and motion vectors between corresponding points of the key frames are made into a template, so that motion of a camera that photographed the key frames is specified and a region to be invalidated is determined.

22. An image coding method according to claim 17, wherein a region to be invalidated is provided all around an image when motion vectors between corresponding points of the key frames are aligned approximately in an upper right direction.

23. An image coding method according to claim 22, wherein the width of the region to be invalidated in top and bottom sides as well as left and right sides of the image is proportional to longitudinal and transverse directional components of the respective motion vector.

24. An image coding method according to claim 17, wherein a reference region of motion vectors between corresponding points of the key frames is narrowed down to a close vicinity of the center of key frames when variation of the field of view between the key frames is substantially large.

25. An image coding method, including:

incorporating corresponding point data between key frames and data of key frames having at least a region that is invalidated, so as to generate a coded data stream.

26. An image coding method according to claim 25, wherein an invalidated region includes left and right edges of the frames when motion vectors between corresponding points of the key frames are aligned approximately in one direction parallel to the top or bottom side of the frames.

27. An image coding method according to claim 26, wherein the width of the invalidated region is proportional to the magnitude of the motion vectors.

28. An image coding method according to claim 26, wherein whether or not the motion vectors are aligned approximately in one direction parallel to the top or bottom side of the frames is detected in a manner such that vectors in one direction are assumed and then the sum of differences between the assumed vectors and actual vectors are taken to determine whether or not the sum is less than a predetermined threshold value.

29. An image coding method according to claim 25, wherein a variation of the field of view between the key frames is presumed in advance for a plurality of ways and motion vectors between corresponding points of the key frames are made into a template, so that motion of a camera that photographed the key frames is specified and a region to be invalidated is determined.

30. An image coding method according to claim 25, wherein an invalidated region is provided all around an image when motion vectors between corresponding points of the key frames are aligned approximately in an upper right direction.

31. An image coding method according to claim 30, wherein the width of the invalidated region in top and bottom sides as well as left and right sides of the image is proportional to longitudinal and transverse directional components of the respective motion vector.

32. An image coding method according to claim 25, wherein a reference region of motion vectors between corresponding points of the key frames is narrowed down to a close vicinity of the key frames when variation of the field of view between the key frames is substantially large.

33. An image coding method, including:

incorporating added information by which to invalidate at least part of region of a reproduction image which is to be generated at a decoding side based on corresponding point data between key frames, into the corresponding point data, so as to generate a coded data stream.

34. An image coding method according to claim 33, wherein a region to be invalidated includes left and right edges of the frames when motion vectors between corresponding points of the key frames are aligned approximately in one direction parallel to the top or bottom side of the frames.

35. An image coding method according to claim 34, wherein the width of the region to be invalidated is made proportional to the magnitude of the motion vectors.

36. An image coding method according to claim 34, wherein whether or not the motion vectors are aligned approximately in one direction parallel to the top or bottom side of the frames is detected in a manner such that vectors of the one same direction are assumed and then the sum of difference between the assumed vectors and actual vectors are taken to determine whether or not the sum is less than a predetermined threshold value.

37. An image coding method according to claim 33, wherein variation of the field of view between the key frames are presumed in advance for a plurality of ways and motion vectors between corresponding points of the key frames are made into a template, so that motion of a camera that photographed the key frames is specified and a region to be invalidated is determined.

38. An image coding method according to claim 33, wherein a region to be invalidated is provided all around an image when motion vectors between corresponding points of the key frames are aligned approximately toward an upper right direction.

39. An image coding method according to claim 38, wherein the width of the region to be invalidated in top and bottom sides as well as left and right sides of the image is made proportional to longitudinal and transverse directional components of the respective motion vector.

40. An image coding method according to claim 33, wherein a reference region of motion vectors between corresponding points of the key frames is narrowed down to a close vicinity of the key frames when variation of the field of view between the key frames is substantially large.

41. An image coding apparatus, comprising:

a matching processor which detects corresponding point data by image matching; and

a stream generator which generates a coded data stream by incorporating the corresponding point data and added information by which to invalidate at least a region of a reproduction image which is to be generated at a decoding side based on the corresponding point data.

42. An image coding apparatus according to claim 41, wherein the added information includes indication of a region to be invalidated.

43. An image coding apparatus according to claim 41, wherein the added information includes image data by which to forcibly replace data of a region to be invalidated.

44. An image coding apparatus according to claim 41, wherein the corresponding point data are incorporated into the coded data stream after data which correspond to a region to be invalidated are deleted from the corresponding point data.

45. An image coding apparatus according to claim 41, wherein the key frames are photographed by a camera, the apparatus further comprising:

a judging unit which judges the magnitude of variation of field of view of the camera which photographed the key frames, based on the corresponding point data,

wherein the added information is effectively generated or effectively incorporated only when it is judged that the variation of the field of view is substantially large.

46. An image coding apparatus according to claim 41, wherein said matching processor evaluates correspondence of critical points between the key frames by extracting the critical point at each key frame.

47. An image coding apparatus, comprising:

a matching processor which detects corresponding point data between key frames by image matching; and

a stream generator which generates a coded data stream by incorporating data of the key frames having at least a region that is invalidated and the detected corresponding point data.

48. An image coding apparatus according to claim 47, wherein the corresponding point data are incorporated into the coded data stream in a form such that data which correspond to the invalidated region are deleted.

49. An image coding apparatus according to claim 47, wherein the key frames are photographed by a camera, the apparatus further comprising:

a judging unit which judges the magnitude of variation of field of view of the camera which photographed the key frames, based on the detected corresponding point data,

wherein the region is invalidated only when it is judged that the variation of the field of view is substantially large.

50. An image coding apparatus according to claim 47, wherein said matching processor evaluates correspondence of critical points between the key frames by extracting critical points from each key frame.

51. An image coding apparatus, comprising:

a matching processor which detects corresponding point data between key frames by image matching; and

a stream generator which generates a coded data stream by incorporating added information by which to invalidate at least part of a region of a reproduction image which is to be generated at a decoding side based on the detected corresponding point data, into the corresponding point data, so as to generate a coded data stream.

52. An image coding apparatus according to claim 51, wherein the added information includes indication of a region to be invalidated.

53. An image coding apparatus according to claim 51, wherein the added information includes image data by which to forcibly replace data of a region to be invalidated.

54. An image coding apparatus according to claim 51, wherein the corresponding point data are incorporated into the coded data stream after data which correspond to a region to be invalidated are deleted.

55. An image coding apparatus according to claim 51, wherein the key frames are photographed by a camera, the apparatus further comprising:

a judging unit which judges the magnitude of movement of viewpoint of the camera which photographed the key frames, based on the detected corresponding point data,

wherein the added information is effectively generated only when it is judged that the movement of the viewpoint is substantially large.

56. An image coding apparatus according to claim 51, wherein said matching processor evaluates correspondence of critical points between the key frames by extracting the critical point at each key frame.

57. An image coding apparatus, including:

a stream generator which generates a coded data stream by incorporating corresponding point data between key frames, and added information by which to invalidate at least part of region of an intermediate image which is to be generated at a decoding side based on the corresponding point data.

58. An image coding apparatus, including:

a stream generator which generates a coded data stream by incorporating corresponding point data between key frames, and data of key frames whose at least part of region is invalidated.

59. An image coding apparatus, including:

a stream generator which generates a coded data stream by incorporating added information by which to invalidate at least part of region of a reproduction image which is to be generated at a decoding side based on corresponding point data between key frames, into the corresponding point data.

60. An image decoding method, comprising:

acquiring a coded data stream of an image; and

generating an intermediate frame based on corresponding point data between key frames included in the acquired coded data stream,

wherein the intermediate frame is generated in a manner that at least part of region thereof is invalidated.

61. An image decoding method according to claim 60, wherein said generating is processed by extracting indication of a region to be invalidated, from the coded data stream.

62. An image decoding method according to claim 60, wherein said generating is processed by extracting image data which forcibly replaces data of a region to be invalidated, from the coded data stream.

63. An image decoding method according to claim 60, further comprising:

judging the magnitude of variation of field of view of a camera which photographed the key frames based on the corresponding point data,

wherein said invalidating is executed only when it is judged that the variation of the field of view is substantially large.

64. An image decoding apparatus, comprising:

an image input unit which acquires a coded data stream of an image; and

an intermediate image generator which generates an intermediate frame based on corresponding point data between key frames included in the acquired coded data stream,

wherein said intermediate image generator generates the intermediate frame in a manner that at least part of region thereof is invalidated.

65. An image decoding apparatus according to claim 64, wherein said intermediate image generator extracts indication of a region to be invalidated, from the coded data stream.

66. An image decoding apparatus according to claim 64, wherein said intermediate image generator extracts image data which forcibly replaces data of a region to be invalidated, from the coded data stream.

67. An image decoding apparatus according to claim 64, further comprising:

a judging unit which judges the magnitude of variation of field of view of a camera which photographed the key frames based on the corresponding point data,

wherein said invalidating is executed only when it is judged that the variation of the field of view is substantially large.