Derivation of studio camera position and motion from the camera image

• US RE38,420 E1
• Filed: 03/25/1998
• Issued: 02/10/2004
• Est. Priority Date: 08/12/1992
• Status: Expired due to Term

• Current Assignee: British Broadcasting Corporation
• Original Assignee: British Broadcasting Corporation
• Inventors: Thomas, Graham Alexander
• Primary Examiner(s): Kostak, Victor R.
• Application Number:
  US09/047,892
• Time in Patent Office: 2,148 Days
• Field of Search: 348/140, 348/25, 348/700, 348/703, 348/699, 348/701, 348/702, 348/135, 348/169, 348/592, 348/27, 348/35, 348/137, 348/581, 348/650, 348/17, 348/348, 348/165, 382/286, 382/289
• US Class Current: 348/140
• CPC Class Codes: G06K 9/209 Sensor details, e.g. position, configuration, special lenses G06K9/2018 takes precedence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/53 Multi-resolution motion estimation; Hierarchical motion estimation
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

Studio camera position and motion may be derived from the camera image by separating out the background and deriving from a background having a number of areas of hue and/or brightness different from adjacent areas estimates of movement from one image to the next. The initial image is used as a reference and amended with predicted motion value. The amended image is compared with incoming images and the result used to derive translation and scale change information. Once the proportion of the reference image contained in an incoming image falls below a threshold a fresh reference image is adopted.

Claims

1. A method of measuring the translation and scale change in a sequence of video images, the method comprising:

storing a first image in said sequence to form a stored image;

forming a prediction of the translation and scale change from said stored image to a further image in said sequence;

comparing said stored image to said further image by transforming at least one of said stored image byand said further image based on said prediction of the translation and scale change;

to form a transformed first image;

comparing a further image in said sequence with said transformed first image;

deriving measurements of translation and scale change between said stored image and said further image from said comparison and said prediction; and

replacing said first image with a new incoming image when the image area common to both said first image and incoming image falls below a given proportion of the whole image area of said incoming image.

2. The method of claim 1, wherein only the background areas of said video images are used in the measurement of translation and scale change.

3. The method of claim 2, wherein a signal is used to separate foreground and background and said signal is derived using chroma-key techniques.

4. The method of claim 2, wherein a signal is used to separate foreground and background portions of the images and said signal is derived using motion detection methods to identify objects moving with a different motion from that predicted for the background.

5. The method of claim 1, wherein:

each image is a single-component signal derived from a camera viewing a scene containing a background of near-uniform color;

said background is divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow the generation of a key signal by chroma-key techniques; and

wherein said single-component signal is formed from a three-component camera signal so to accentuate differences in hue and/or brightness of individual areas of said background to enable motion estimation.

9. The method of claim 1 or 6, wherein said translation and scale change are predicted by the computation of a number of simultaneous equations each of which relate said translation and scale change to spatial and temporal gradients at a point in the reference image and which are solved to yield a least-squares solution for the motion parameters.

10. The method of claim 1 or 6, comprising:

selecting a number of measurement points in said first image for motion estimation; and

replacing said first image with a new incoming image when the number of measurement points which lie in areas of background visible in both said first image and a given incoming image falls below a given proportion of the total number of measurement points.

11. The method of claim 1 or 6, comprising replacing said first image if the scale change between an incoming image and the reference image exceeds a given factor.

12. The method of claim 1 or 6, comprising spatially prefiltering said first image prior to storage and comparison.

13. The method of claim 10, wherein said measurement points lie in a regular array in the image.

14. The method of claim 10 wherein said measurement points are chosen to lie at points of high spatial gradient.

15. The method of claim 1 or 6, comprising storing replaced reference images for later use.

27. The method of claim 1 wherein said step of comparing said stored image to said further image comprises substeps of:

6. A method of measuring the translation and scale change in a sequence of video images of a camera signal derived by a camera, comprising:

storing a first image of said sequence to form a stored image said image comprising a single component signal derived by a camera viewing a scene containing a background of near-uniform color, and said background being divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow generation of a key signal by chroma-key techniques;

forming said single-component signal from said camera signal so as to accentuate differences in hue and/or brightness of individual areas of the background to enable motion estimation;

forming a prediction of the translation and scale change from said stored image to a further image in said sequence;

comparing said stored image to said further image by transforming at least one of said stored image byand said further image based on said prediction of the translation and scale change;

to form a transformed first image;

comparing a further image in said sequence with said transformed first image; and

deriving measurements of translation and scale change between said stored image and said further image from said comparison and said prediction.

7. The method of claim 6, wherein said background is divided into a plurality of areas, each area having one of two hues and/or brightnesses.

8. The method of claim 6, wherein said areas of the background are square.

28. The method of claim 6 wherein said step of comparing said stored image to said further image comprises substeps of:

16. Apparatus for measuring the translation and scale change in a sequence of video images, comprising:

means for acquiring said sequence of images;

storage means for storing a first image in said sequence to form a stored image;

means for forming a prediction of the translation and scale change from said stored image to a further image in said sequence;

means for comparing said stored image to said further image by transforming at least one of said stored image by and said further image based on said prediction of the translation and scale change to form a transformed first image;

means for comparing said transformed first image with a further image in the sequence ;

means for deriving measurements of translation and scale change between said stored image and said further image from said comparison and said prediction; and

means for replacing said first image with a new incoming image when the image area common to both said first image and incoming image falls below a given proportion of the whole image area of said incoming images.

17. The apparatus of claim 16, wherein said means for deriving operates only on the background areas of the images, said means for deriving comprising means for separating foreground and background portions of the images using motion techniques to identify objects moving with a different motion from that predicted for the background.

18. The apparatus of claim 16, wherein said means for deriving operates only on the background areas of the images, and said means for deriving comprising a key generator for generating a chroma-key to separate foreground and background.

19. The apparatus of claim 16, comprising:

means for generating a single-component signal from a camera viewing a scene containing a background of near-uniform color, said background being divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow the generation of a key signal by chroma-key techniques, and wherein the single-component signal is formed from a three-component camera signal so as to accentuate differences in hue and/or brightness of individual areas of the background to enable motion estimation.

21. The apparatus of claim 19 or 20, wherein said background is divided into a plurality of areas each area having one or two hues and/or brightnesses.

22. The apparatus of claim 19 or 20, wherein said areas of the background are square.

23. The apparatus of claim 19 or 20, comprising:

means for predicting the translation and scale change by the computation of a number of simultaneous equations each of which relate the translation and scale change to spatial and temporal gradients at a point in the first image and which are solved to yield a least-squares solution for the motion parameters.

24. The apparatus of claim 19 or 20, wherein the replacing means comprises:

means for selecting a number of measurement points in the first image for motion estimation and for replacing the first image with a new incoming image when the number of measurement points which lie in areas of background visible in both the first image and a given incoming image falls below a given proportion of the total number of measurement points.

25. The apparatus of claim 16 or 20, comprising a spatial filter for filtering the images prior to storage and comparison.

26. The apparatus of claim 16 or 20, comprising a further storage means for storing a replaced reference image for future use.

29. The apparatus of claim 16 wherein said means for comparing said stored image to said further image comprises:

20. Apparatus for measuring the translation and scale change in a sequence of video images derived by a camera, comprising:

means for storing a first image of said sequence to form a stored image, said image comprising a single component signal derived by a camera viewing a scene containing a background of near-uniform color, said background being divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow generation of a key signal by chroma-key techniques;

means for forming said single-component signal from said camera signal so as to accentuate differences in hue and/or brightness of individual areas of the background to enable motion estimation;

means for forming a prediction of the translation and scale change from said stored image to a further image in said sequence;

means for comparing said stored image to said further image by transforming at least one of the stored image by a and said further image based on said prediction of the translation and scale change to form a transformed first image;

means for comparing a further image in said sequence with said transformed first image; and

means for deriving, from said comparison and said prediction, measurements of translation of scale change between said stored image and said further image.

30. The apparatus of claim 20 wherein said means for comparing said stored image to said further image comprises:

31. Apparatus for measuring the translation and scale change in a sequence of video images, comprising:

32. The apparatus of claim 31, wherein said measurement unit operates only on the background areas of the images, said measurement unit separating foreground and background portions of the images using motion techniques to identify objects moving with a different motion from that predicted for the background.

33. The apparatus of claim 31, wherein said measurement unit operates only on the background areas of the images, and said measurement unit comprises a key generator for generating a chroma-key to separate foreground and background.

34. The apparatus of claim 31, comprising:

means for generating a single-component signal from a camera viewing a scene containing a background of near-uniform color, said background being divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow the generation of a key signal by chroma-key techniques, and wherein the single-component signal is formed from a three-component camera signal so as to accentuate differences in hue and/or brightness of individual areas of the background to enable motion estimation.

35. Apparatus for measuring the translation and scale change in a sequence of video images derived by a camera, comprising:

an image storage unit that stores a first image of said sequence to form a stored image, said image comprising a single component signal derived by a camera viewing a scene containing a background of near-uniform color, said background being divided into a plurality of areas each having a hue and/or brightness different from the hue and/or brightness of adjacent areas to allow generation of a key signal by chroma-key techniques;

a signal former that forms said single-component signal from said camera signal so as to accentuate differences in hue and/or brightness of individual areas of the background to enable motion estimation;

36. The apparatus of claim 35 wherein said background is divided into a plurality of areas each area having one or two hues and/or brightnesses.

37. The apparatus of claim 35, wherein said areas of the background are square.

97. The apparatus of claim 35 wherein said areas of the background are quadrilaterals.

38. A method of deriving a measure of translation and scale change from a camera output signal, the method comprising:

39. A method according to claim 38, wherein the first and second tones differ in hue.

40. A method according to claim 38, wherein the first and second tones differ in brightness.

41. A method according to claim 38, wherein the first and second tones are of substantially equal brightness.

42. A method according to claim 38, wherein the pattern comprises a chequered pattern.

43. A method according to claim 38, wherein said chroma-keying is used to remove foreground objects from the image prior to determination of translation and scale change.

44. A method according to claim 38 wherein the camera output signal is spatially filtered prior to storage and comparison.

45. A method according to claim 38, wherein said measures of translation and scale change are determined using a stored reference image.

46. A method according to claim 45, wherein translation and scale change are determined by transforming the reference image and comparing with the camera image.

47. A method according to claim 46, wherein said transforming is based on a prediction of the translation and scale change.

48. A method according to claim 45, wherein translation and scale change are determined by computation of a number of simultaneous equations each of which relate the translation and scale change to spatial and temporal gradients at a point in the reference image and which are solved to yield a least-squares solution for the motion parameters.

49. A method according to claim 45, wherein the stored reference image is replaced when the accumulated translation and scale change exceeds a given threshold.

50. A method according to claim 38, wherein translation and scale change are determined from a sub-set of the pixels in the camera image.

51. A method according to claim 50, wherein the sub-set of pixels lie in a regular array in the camera image.

52. A method according to claim 50, wherein the sub-set of pixels are chosen to lie at points of high spatial gradient.

53. A method according to claim 38, wherein a stored reference image is compared to the camera image after transforming the reference image based on a prediction of translation and scale change and a measure of translation and scale change derived from the comparison.

54. A method according to claim 53, wherein the reference image comprises an earlier camera image.

55. A method according to claim 54, wherein the reference image is replaced when the overlap between the reference image and the current camera image falls below a given proportion.

56. A method according to claim 38, wherein data obtained from a sensor on the camera is used in derivation of said measures of translation and scale change.

98. The method of claim 38 wherein said areas of the first tone and areas of the second tone are quadrilaterals.

57. Apparatus for deriving at least one parameter representing the movement of a camera from a camera image, said camera viewing at least a portion of a background having a pattern comprising a plurality of areas of a first tone and a plurality of areas of a second tone arranged so that transitions between areas occur along at least two axes, whereby the transitions provide reference points from which measures of both translation and scale change may be determined, the apparatus comprising:

58. Apparatus according to claim 57 further comprising said background.

59. Apparatus according to claim 57 further comprising said camera.

60. Apparatus according to claim 57 further comprising said background and said camera.

61. Apparatus according to claim 57, wherein the first and second tones differ in hue.

62. Apparatus according to claim 57, wherein the first and second tones differ in brightness.

63. Apparatus according to claim 57, wherein the first and second tones are of substantially equal brightness.

64. Apparatus according to claim 57, wherein the pattern comprises a chequered pattern.

65. Apparatus according to claim 57, wherein said detecting means is arranged to remove foreground objects from the image prior to determination of translation and scale change.

66. Apparatus according to claim 57 comprising means for spatially filtering the camera image prior to storage and comparison.

67. Apparatus according to claims 57, further comprising an image store that stores a reference image for use by said deriving means.

68. Apparatus according to claim 57, wherein the deriving means is configured to determine translation and scale change by transforming the reference image and comparing with the camera image.

69. Apparatus according to claim 68, wherein said transforming is based on a prediction of the translation and scale change.

70. Apparatus according to claim 67, wherein translation and scale change are determined by computation of a number of simultaneous equations each of which relate the translation and scale change to spatial and temporal gradients at a point in the reference image and which are solved to yield a least-squares solution for the motion parameters.

71. Apparatus according to claims 67 wherein the stored reference image is replaced when the accumulated translation and scale change exceeds a given threshold.

72. Apparatus according to claim 57, wherein said deriving means is configured to determine translation and scale change from a sub-set of the pixels in the camera image.

73. Apparatus according to claim 72, wherein the sub-set of pixels lie in a regular array in the camera image.

74. Apparatus according to claim 72 wherein the sub-set of pixels are chosen to lie at points of high spatial gradient.

75. Apparatus according to claim 57, further comprising a sensor on the camera that provides data for use in derivation of said measures of translation and scale change.

76. Apparatus according to claim 57, wherein said deriving means comprises digital signal processing apparatus.

99. The apparatus of claim 57 wherein said areas of the first tone and areas of the second tone are quadrilaterals.

77. Apparatus for deriving at least one parameter representing the movement of a camera from a camera image, said camera viewing at least a portion of a background having a pattern comprising a plurality of areas of a first tone and a plurality of areas of a second tone arranged so that transitions between areas occur along at least two axes, whereby the transitions provide reference points from which measures of both translation and scale change may be determined, the apparatus comprising:

78. Apparatus according to claim 77 further comprising said background.

79. Apparatus according to claim 77 further comprising said camera.

80. Apparatus according to claim 77 further comprising said background and said camera.

81. Apparatus according to claim 77, wherein the first and second tones differ in hue.

82. Apparatus according to claim 77, wherein the first and second tones differ in brightness.

83. Apparatus according to claim 77, wherein the first and second tones are of substantially equal brightness.

84. Apparatus according to claim 77, wherein the pattern comprises a chequered pattern.

85. Apparatus according to claim 77, wherein said signal processing system removes foreground objects from the image prior to determination of translation and scale change.

86. Apparatus according to claim 77, wherein said signal processing system determines translation and scale change by transforming the reference image and comparing with the camera image.

87. Apparatus according to claim 77, further comprising an image store that stores a reference image for use by said signal processing system.

88. Apparatus according to claim 87 comprising a filter that spatially filters the camera image prior to storage and comparison.

89. Apparatus according to claim 86, wherein said transforming is based on a prediction of the translation and scale change.

90. Apparatus according to claim 87, wherein translation and scale change are determined by computation of a number of simultaneous equations each of which relate the translation and scale change to spatial and temporal gradients at a point in the reference image and which are solved to yield a least-squares solution for the motion parameters.

91. Apparatus according to claim 87, arranged to replace the stored reference image when the accumulated translation and scale change exceeds a given threshold.

92. Apparatus according to claim 77, wherein said signal processing system determines translation and scale change from a sub-set of the pixels in the camera image.

93. Apparatus according to claim 92, wherein the sub-set of pixels lie in a regular array in the camera image.

94. Apparatus according to claim 92 wherein the sub-set of pixels are chosen to lie at points of high spatial gradient.

95. Apparatus according to claim 77, further comprising a sensor on the camera that provides data for use in derivation of said measures of translation and scale change.

96. Apparatus according to claim 77, wherein said deriving means comprises digital signal processing apparatus.

100. The apparatus of claim 77 wherein said areas of the first tone and areas of the second tone are quadrilaterals.