Method and apparatus for a declarative representation of distortion correction for add-on graphics in broadcast video

• US 6,850,250 B2
• Filed: 08/28/2001
• Issued: 02/01/2005
• Est. Priority Date: 08/29/2000
• Status: Expired due to Fees

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Electronics Inc. (Sony Group Corp.)
• Inventors: Hoch, Michael
• Primary Examiner(s): BRIER, JEFFERY A
• Application Number:
  US09/942,251
• Publication Number:
  US 20020070952A1
• Time in Patent Office: 1,253 Days
• Field of Search: 345/632, 345/633, 345/647, 348/239, 348/589
• US Class Current: 345/632
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/434 Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/440263 by altering the spatial resolution, e.g. for displaying on a connected PDA
• CPC Class Codes: H04N 21/4438 Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4755 for defining user preferences, e.g. favourite actors or genre retrieval personalisation and generation of user profiles for the retrieval of video data G06F16/739; user profiles in network data switching protocols H04L67/306
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

An apparatus and a method for distorting add-on graphics to be composited with a streaming video feed comprising, receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame, measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame, creating a virtual camera, setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data, generating add-on graphics having a size and an insertion position within the image frame based on the virtual camera, creating a distortion grid, distorting the add-on graphics to match the image distortions of the image frame, and compositing the distorted add-on graphics with the image frame of the streaming video feed.

Claims

1. A method of distorting add-on graphics to be composited with a streaming video feed comprising:

receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and distortion parameters corresponding to the image distortions of the image frame;

creating a virtual camera;

setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

generating add-on graphics having a size and an insertion position within the image frame based on the virtual camera;

creating a distortion grid;

distorting the add-on graphics to match the image distortions of the image frame; and

compositing the distorted add-on graphics with the image frame of the streaming video feed.

2. The method of claim 1 wherein the virtual camera renders the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera.

3. The method of claim 1 wherein the size and the insertion position of the add-on graphics within the image frame corresponds to the size and the position of an image of a real asset within the image frame.

4. The method of claim 1 wherein the grid has a set of coordinate values based on the distortion parameters.

5. The method of claim 4 wherein distorting of the add-on graphics to match the image distortions of the image frame uses the set of coordinate values.

6. The method of claim 1 wherein the virtual camera is created using a declarative markup language.

7. The method of claim 6 wherein the declarative markup language is an extension of VRML.

8. The method of claim 6 wherein the declarative markup language includes at least:

a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

9. The method of claim 8 wherein the camera view point node comprises at least one of:

a first field for receiving and storing parameters corresponding to the field of view of the real camera;

a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

a description field for receiving and storing data corresponding to the viewing direction of the real camera.

10. The method of claim 6 wherein the declarative markup language includes at least:

a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

11. The method of claim 10 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

12. The method of claim 10 wherein the grid node comprises at least one of:

a dimension field for receiving the dimension of the image frame; and

a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

13. The method of claim 12 wherein the distort field includes at least one of:

a parameter representing the optical center shift of the real camera in an x direction;

a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

at least one parameter representing the radial distortion coefficient of the real camera.

14. The method of claim 6 wherein the declarative markup language includes:

a first node for receiving the plurality of camera instrumentation data; and

a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

15. The method of claim 1 wherein the distortion grid is created using a declarative markup language.

16. The method of claim 15 wherein the declarative markup language is an extension of VRML.

17. The method of claim 15 wherein the declarative markup language includes at least:

a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

18. The method of claim 17 wherein the camera viewpoint node comprises at least one of:

a first field for receiving and storing parameters corresponding to the field of view of the real camera;

a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

a description field for receiving and storing data corresponding to the viewing direction of the real camera.

19. The method of claim 15 wherein the declarative markup language includes at least:

a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

20. The method of claim 19 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

21. The method of claim 19 wherein the grid node comprises at least one of:

a dimension field for receiving the dimension of the image frame; and

a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

22. The method of claim 21 wherein the distort field includes at least one of:

a parameter representing the optical center shift of the real camera in an x direction;

a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

at least one parameter representing the radial distortion coefficient of the real camera.

23. The method of claim 15 wherein the declarative markup language includes:

a first node for receiving the plurality of camera instrumentation data; and

a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

24. The method of claim 1 wherein the camera instrumentation data from the corresponding real camera includes parameter values for at least one of:

pan, tilt, twist, zoom level and image distortion parameters.

25. The method of claim 1 wherein the image distortion parameters include at least one of:

radial lens distortion and optical center shift in x and y direction.

26. A computer program product embodied in a computer readable medium for distorting add-on graphics to be composited with a streaming video feed comprising:

code means for receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

code means for measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

code means for creating a virtual camera;

code means for setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

code means for generating add-on graphics having a size and an insertion position within the image frame based on the virtual camera;

code means for creating a distortion grid;

code means for distorting the add-on graphics to match the image distortions of the image frame; and

code means for compositing the distorted add-on graphics with the image frame of the streaming video feed.

27. The computer program product of claim 26 wherein the virtual camera renders the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera.

28. The computer program product of claim 26 wherein the size and the insertion position of the add-on graphics within the image frame corresponds to the size and the position of an image of a real asset within the image frame.

29. The computer program product of claim 26 wherein the grid has a set of coordinate values based on the distortion parameters.

30. The computer program product of claim 29 wherein distorting of the add-on graphics to match the image distortions of the image frame uses the set of coordinate values.

31. The computer program product of claim 26 wherein the virtual camera is created using a declarative markup language.

32. The computer program product of claim 31 wherein the declarative markup language is an extension of VRML.

33. The computer program product of claim 31 wherein the declarative markup language includes at least:

code means for a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

34. The computer program product of claim 33 wherein the camera view point node comprises at least one of:

code means for a first field for receiving and storing parameters corresponding to the field of view of the real camera;

code means for a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

code means for a description field for receiving and storing data corresponding to the viewing direction of the real camera.

35. The computer program product of claim 31 wherein the declarative markup language includes at least:

code means for a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

36. The computer program product of claim 35 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

37. The computer program product of claim 35 wherein the grid node comprises at least one of:

code means for a dimension field for receiving the dimension of the image frame; and

code means for a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

38. The computer program product of claim 37 wherein the distort field includes at least one of:

code means for a parameter representing the optical center shift of the real camera in an x direction;

code means for a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

code means for at least one parameter representing the radial distortion coefficient of the real camera.

39. The computer program product of claim 31 wherein the declarative markup language includes:

code means for a first node for receiving the plurality of camera instrumentation data; and

code means for a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

40. The computer program product of claim 26 wherein the distortion grid is created using a declarative markup language.

41. The computer program product of claim 40 wherein the declarative markup language is an extension of VRML.

42. The computer program product of claim 40 wherein the declarative markup language includes at least:

code means for a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

43. The computer program product of claim 42 wherein the camera viewpoint node comprises at least one of:

code means for a first field for receiving and storing parameters corresponding to the field of view of the real camera;

code means for a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

code means for a description field for receiving and storing data corresponding to the viewing direction of the real camera.

44. The computer program product of claim 40 wherein the declarative markup language includes at least:

code means for a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

45. The computer program product of claim 44 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

46. The computer program product of claim 44 wherein the grid node comprises at least one of:

code means for a dimension field for receiving the dimension of the image frame; and

code means for a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

47. The computer program product of claim 46 wherein the distort field includes at least one of:

code means for a parameter representing the optical center shift of the real camera in an x direction;

code means for a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

code means for at least one parameter representing the radial distortion coefficient of the real camera.

48. The computer program product of claim 40 wherein the declarative markup language includes:

code means for a first node for receiving the plurality of camera instrumentation data; and

code means for a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

49. The computer program product of claim 26 wherein the camera instrumentation data from the corresponding real camera includes parameter values for at least one of:

pan, tilt, twist, zoom level and image distortion parameters.

50. The computer program product of claim 26 wherein the image distortion parameters include at least one of:

radial lens distortion and optical center shift in x and y direction.

51. An apparatus of distorting add-on graphics to be composited with a streaming video feed comprising:

means for receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

means for measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

means for creating a virtual camera;

means for setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

means for generating add-on graphics having a size and an insertion position within the image frame based on the virtual camera;

means for creating a distortion grid;

means for distorting the add-on graphics to match the image distortions of the image frame; and

means for compositing the distorted add-on graphics with the image frame of the streaming video feed.

52. The apparatus of claim 51 wherein the virtual camera renders the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera.

53. The apparatus of claim 51 wherein the size and the insertion position of the add-on graphics within the image frame corresponds to the size and the position of an image of a real asset within the image frame.

54. The apparatus of claim 51 wherein the grid has a set of coordinate values based on the distortion parameters.

55. The apparatus of claim 54 wherein distorting of the add-on graphics to match the image distortions of the image frame uses the set of coordinate values.

56. The apparatus of claim 51 wherein the virtual camera is created using a declarative markup language.

57. The apparatus of claim 56 wherein the declarative markup language is an extension of VRML.

58. The apparatus of claim 56 wherein the declarative markup language includes at least:

a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

59. The apparatus of claim 58 wherein the camera view point node comprises at least one of:

a first field for receiving and storing parameters corresponding to the field of view of the real camera;

a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

a description field for receiving and storing data corresponding to the viewing direction of the real camera.

60. The apparatus of claim 56 wherein the declarative markup language includes at least:

a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

61. The apparatus of claim 60 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

62. The apparatus of claim 60 wherein the grid node comprises at least one of:

a dimension field for receiving the dimension of the image frame; and

a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

63. The apparatus of claim 62 wherein the distort field includes at least one of:

a parameter representing the optical center shift of the real camera in an x direction;

a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

at least one parameter representing the radial distortion coefficient of the real camera.

64. The apparatus of claim 56 wherein the declarative markup language includes;

a first node for receiving the plurality of camera instrumentation data; and

a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

65. The apparatus of claim 51 wherein the distortion grid is created using a declarative markup language.

66. The apparatus of claim 65 wherein the declarative markup language is an extension of VRML.

67. The apparatus of claim 65 wherein the declarative markup language includes at least:

a camera viewpoint node for receiving the camera instrumentation data used to drive the virtual camera.

68. The apparatus of claim 67 wherein the camera viewpoint node comprises at least one of:

a first field for receiving and storing parameters corresponding to the field of view of the real camera;

a distort field for receiving and storing distortion parameters corresponding to the radial lens distortions of the real camera; and

a description field for receiving and storing data corresponding to the viewing direction of the real camera.

69. The apparatus of claim 65 wherein the declarative markup language includes at least:

a grid node for creating a distortion grid used as a form for distorting the add-on graphics.

70. The apparatus of claim 69 wherein the form is adjusted to match the distortions parameters by adjusting the set of coordinate values.

71. The apparatus of claim 69 wherein the grid node comprises at least one of:

a dimension field for receiving the dimension of the image frame; and

a distort field for receiving the distortion parameters corresponding to radial lens distortion values of the real camera.

72. The apparatus of claim 71 wherein the distort field includes at least one of:

a parameter representing the optical center shift of the real camera in an x direction;

a parameter representing the optical center shift of the real camera in a y direction orthogonal to said x direction; and

at least one parameter representing the radial distortion coefficient of the real camera.

73. The apparatus of claim 65 wherein the declarative markup language includes:

a first node for receiving the plurality of camera instrumentation data; and

a second node for receiving data corresponding to the view point and field of view of the real camera, the data being used to set the virtual view point and the virtual field of view.

74. The apparatus of claim 51 wherein the camera instrumentation data from the corresponding real camera includes parameter values for at least one of:

pan, tilt, twist, zoom level and image distortion parameters.

75. The apparatus of claim 51 wherein the image distortion parameters include at least one of:

radial lens distortion and optical center shift in x and y direction.

76. A system for creating add-on graphics into a streaming video feed comprising:

means for generating a streaming video feed;

means for collecting camera instrumentation data;

means for transmitting the streaming video feed and the camera instrumentation data down to a receiver;

means for receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

means for measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

means for creating a virtual camera;

means for setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

means for generating add-on graphics having a size and an insertion position within the image frame based on the virtual camera;

means for creating a distortion grid;

means for distorting the add-on graphics to match the image distortions of the image frame; and

means for compositing the distorted add-on graphics with the image frame of the streaming video feed.

77. A method of distorting add-on graphics to be composited with a streaming video feed comprising:

receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

creating a virtual camera;

using a declarative markup language, said virtual camera being used in generating add-on graphics having a size and an insertion position within the image frame, the virtual camera rendering the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera, the size and the insertion position of the add-on graphics within the image frame corresponding to the size and the position of an image of a real asset within the image frame;

setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

creating a distortion grid using the declarative markup language, the grid having a set of coordinate values based on the distortion parameters;

distorting the add-on graphics to match the image distortions of the image frame using the set of coordinate values; and

compositing the distorted add-on graphics with the image frame of the streaming video feed.

78. An apparatus of distorting add-on graphics to be composited with a streaming video feed comprising:

means for receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

means for measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

means for creating a virtual camera;

using a declarative markup language, said virtual camera being used in generating add-on graphics having a size and an insertion position within the image frame, the virtual camera rendering the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera , the size and the insertion position of the add-on graphics within the image frame corresponding to the size and the position of an image of a real asset within the image frame;

means for setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

means for creating a distortion grid using the declarative markup language, the grid having a set of coordinate values based on the distortion parameters;

means for distorting the add-on graphics to match the image distortions of the image frame using the set of coordinate values; and

means for compositing the distorted add-on graphics with the image frame of the streaming video feed.

79. A computer program product embodied in a computer readable medium for distorting add-on graphics to be composited with a streaming video feed comprising;

code means for receiving a streaming video feed captured using a real camera, said streaming video feed including an image frame taken from a particular view point and a particular field of view, said real camera introducing image distortions into the image frame;

code means for measuring a plurality of camera instrumentation data corresponding to at least said view point, said field of view, and a distortion parameters corresponding to the image distortions of the image frame;

code means for creating a virtual camera;

using a declarative markup language, said virtual camera being used in generating add-on graphics having a size and an insertion position within the image frame, the virtual camera rendering the add-on graphics at a virtual view point and a virtual field of view corresponding to the view point and the field of view of the real camera, the size and the insertion position of the add-on graphics within the image frame corresponding to the size and the position of an image of a real asset within the image frame;

code means for setting the virtual view point and the virtual field of view of the virtual camera using the plurality of camera instrumentation data;

code means for creating a distortion grid using the declarative markup language, the grid having a set of coordinate values based on the distortion parameters;

code means for distorting the add-on graphics to match the image distortions of the image frame using the set of coordinate values; and

code means for compositing the distorted add-on graphics with the image frame of the streaming video feed.