Position determination

• US 6,556,722 B1
• Filed: 11/30/1999
• Issued: 04/29/2003
• Est. Priority Date: 05/30/1997
• Status: Expired due to Term

• Current Assignee: British Broadcasting Corporation
• Original Assignee: British Broadcasting Corporation
• Inventors: Russell, Richard Thomas, Thomas, Graham Alexander
• Primary Examiner(s): COUSO, YON JUNG
• Application Number:
  US09/449,634
• Time in Patent Office: 1,246 Days
• Field of Search: 382/291, 382/306, 382/287, 382/294, 382/151, 348/94, 348/95, 348/87
• US Class Current: 382/291
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/875 for determining attitude
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/163 Determination of attitude using inertial means G01C9/00; control of attitude G05D1/49
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

In the preferred implementation, the position of an object, for example a studio camera is determined by means of a camera which views several markers disposed about a studio ceiling, the markers being patterened as a series of light and dark rings to encode information in binary form enabling the markers to be identified as the camera moves about the studio. Methods and apparatus of more general applicability are also disclosed.

Claims

1. A method of determining the three-dimensional position of an object in a reference frame, the method comprising:

providing three or more markers at respective reference positions, at least three of the markers being patterned as a series of concentric shapes encoding a plurality of bits of identification information in the darkness or color of the shapes, the center of the concentric shapes of each patterned marker providing a reference point for each marker, wherein each marker has a respective distance from a reference plane of the reference frame, the distances varying between the markers;

storing information including a measure of the three-dimensional positions of the markers in the reference frame and information identifying the patterned markers;

obtaining an image of at least a sub-set of said markers from a camera associated with the object;

processing the image to identify the positions of said markers in the image and, for each patterned marker in the sub-set, identifying the respective reference point and decoding said identification information;

determining a measure of the three-dimensional position of the object in the reference frame based on said processing and decoding and based on said stored information.

2. A method according to claim 1, wherein the patterned markers encode a plurality of bits of information in binary form as a series of light and dark shapes.

5. A method according to claim 2, wherein the shapes are circular rings.

6. A method according to claim 2, wherein 3 or more bits of identification information are encoded.

7. A method according to claim 2, wherein 4 or more shapes are provided.

8. A method according to claim 2, wherein each bit of the information is assigned to a corresponding shape.

9. A method according to claim 2, wherein at least one outer region of a predetermined brightness is provided.

10. A method according to claim 2, wherein at least one parity or other error-correcting bit is included in the encoded information.

11. A method according to claim 2, wherein dark shapes are substantially black.

12. A method according to claim 2, wherein light shapes are retro-reflective so as to reflect light directed from adjacent said camera substantially towards said camera.

13. A method according to claim 12, further including illuminating the markers with a light source substantially coincident with the axis of said camera means so that light falling on said light shapes is retro-reflected towards said camera.

14. A method according to claim 1, wherein each shape is coloured, patterned or shaded to encode a plurality of bits of identification information.

15. A method according to claim 1, wherein all markers are patterned to encode identification information.

16. A method according to claim 1, wherein at least some markers encode identification information uniquely identifying that marker among the plurality of markers.

17. A method according to claim 1, wherein the markers are so disposed that the position of the object can be uniquely determined at all points within a given volume of the reference frame.

18. A method according to claim 1 including adjusting the determined position to smooth out sudden changes in determined position as markers are revealed or obscured.

19. A method according to claim 18, wherein adjusting comprises storing pre-stored or measured marker positions as initial marker positions;

dynamically applying correction factors to said initial marker positions to produce respective corrected marker positions;

determining the object position based on the corrected marker positions, wherein

said dynamically applying correction factors comprises calculating correction factors so that the corrected marker positions are corrected to become mutually self-consistent values, the rate of variation of the correction factors being limited to a pre-determined level by limiting the rate of change of correction factors.

24. A method according to claim 1, wherein said object is a video camera having a field of view differing from the field of view of said camera.

25. A method according to claim 24 further comprising providing at least one supplementary marker at a given position so as to be viewable by the video camera, the or each marker having identification information encoded thereon in binary form as a series of concentric rings of two tones, the tones being selected to enable the marker to be keyed out of the video camera image by chroma-keying, said determining the position of the video camera being based on the position of the supplementary marker, if present, in the video camera image.

36. A set of at least three patterned markers mountable in a reference frame to enable the three-dimensional position of an object with respect to the reference frame to be determined in accordance with the method of claim 1, each marker comprising a series of concentric rings, each marker encoding an identifier of the marker as a plurality of bits of information in binary form as the brightness or reflectivity and/or thickness of each ring, the identifier of the markers varying between the markers, the center of the rings providing a reference point for the markers.

40. A virtual production studio having at least three patterned markers mounted on the ceiling thereof at mutually different distances from the ceiling to enable the three-dimensional position of an object with respect to the studio reference frame to be determined in accordance with the method of claim 1, each marker comprising a series of concentric rings, each marker encoding an identifier of the marker as a plurality of bits of information in binary form as the brightness or reflectivity and/or thickness of each ring, the identifier of the markers varying between the markers, the center of the rings providing a reference point for the markers.

41. A virtual production studio according to claim 40, having at least 10 markers mounted on the ceiling thereof.

42. A virtual production studio according to claim 40 including a store of the three-dimensional positions and identifiers of the markers for use in position determination.

43. A virtual production studio according to claim 40 including at least one studio camera arranged to capture a subject in a working space of the studio, the studio camera having apparatus for determination of the three-dimensional position of the studio camera in the studio reference frame, the apparatus for determination of the three-dimensional position comprising:

a camera for mounting on the studio camera and arranged to point upwards at the markers when the studio camera is upright and pointed at a subject in said working space;

a memory storing measures of the three-dimensional positions of said markers in the reference frame and to store information identifying patterned encoded markers which markers comprise a series of concentric shapes encoding identification information as the darkness or color of shapes, the center of the concentric shapes of each patterned marker providing a reference point for each marker;

image processing means arranged to process an image output by the camera to identify the positions of markers in the image, to decode information encoded in the patterned markers and to determine the position of said reference points; and

position determining means arranged to determine the three-dimensional position of the studio camera in the studio reference frame based on the output of the image processing means and the information stored in the memory means.

44. A method of providing a virtual production, the method comprising determining the position of a video camera in accordance with claim 24;

and

applying a video effect to the output of the video camera based on the determined position.

3. A method of determining the three-dimensional position of an object in a reference frame, the method comprising:

providing three or more markers at respective reference positions, at least three of the markers being patterned to encode a plurality of bits of identification information in binary form in a series of concentric light and dark regions, the center of the concentric regions providing a reference point, wherein each marker has a respective distance from a reference plane of the reference frame, the distances varying between the markers;

;

storing information including a measure of the three-dimensional positions of the markers in the reference frame and information identifying the patterned markers;

obtaining an image of at least a sub-set of said plurality of markers from a camera associated with the object;

processing the image to identify the positions of said markers in the image and, for each patterned marker in the sub-set, decoding said identification information and identifying the reference point; and

determining a measure of the three-dimensional position of the object in the reference frame based on said processing and decoding and based on said stored information.

4. A method of producing a set of at least three markers for use in determining the three-dimensional position of an object in a reference frame in a method according to claim 3, the markers having a mutually different identifiers encoded as a plurality of bits of information, the method comprising:

providing mutually different identifiers to be encoded in binary form; and

providing on a mountable backing a series of concentric shapes, each shape having either a high or a low brightness or reflectivity, the thickness or brightness or reflectivity of each ring shape selected according to a pre-determined coding scheme to encode the information, wherein the center of the shapes provides a reference point.

39. A method according to claim 3, wherein the reference frame comprises a production studio having at least 10 markers positioned on the ceiling thereof, wherein the camera is an upward-pointing camera mounted on a studio camera.

20. A method of determining the position of an object based on identification of a plurality of markers at respective reference positions, the method comprising:

storing pre-stored or measured marker positions as initial marker positions;

dynamically applying correction factors to said initial marker positions to produce respective corrected marker positions;

determining the object position based on the corrected marker positions, wherein

said dynamically applying correction factors comprises calculating correction factors so that the corrected marker positions are corrected to become mutually self-consistent values, a rate of variation of the correction factors being limited to a pre-determined level by limiting the rate of change of correction factors.

21. A method according to claim 20, wherein the pre-determined level is selected so that transitions in the determined image position resulting from one or more identified markers becoming unusable or new markers becoming usable are smoothed over a period of at least about 2 seconds.

22. A method according to claim 20, wherein a measure of the accuracy of position determination is obtained.

23. A method according to claim 22, wherein said measure is based on determination of said correction factors.

26. Apparatus for determination of the three-dimensional position of an object in a reference frame, the apparatus comprising:

a camera for mounting on an object whose position is to be determined;

a memory arranged to store measures of the three-dimensional positions of at least three markers in the reference frame and to store information identifying patterned encoded markers which markers comprise a series of concentric shapes encoding identification information as the darkness or color of the shapes, the center of the concentric shapes of each patterned marker providing a reference point for each marker;

image processing means arranged to process an image output by the camera to identify the positions of markers in the image, to decode information encoded in the patterned markers and to determine the position of said reference points; and

position determining means arranged to determine the three-dimensional position of the object in the reference frame based on the output of the image processing means and the information stored in the memory means.

27. Apparatus according to claim 26 wherein the memory, image processing means and position determining means are integrated into a single computer.

28. Apparatus according to claim 26 further including a set of at least three markers, including a subset of at least three patterned markers having identification information encoded as a series of concentric rings of light and dark regions, the center of the rings providing a reference point.

29. Apparatus according to claim 26 for use with markers including retro-reflective portions so as to reflect light directed from adjacent said camera substantially towards said camera, the apparatus having light-projection means substantially optically coincident with the axis of said camera.

30. Apparatus according to claim 29 wherein the light-projection means is pulsed, arranged to be pulsed on at a time when said camera means is integrating.

31. Apparatus according to claim 29 wherein the camera means is provided with filter means having a transmission window selected to pass light transmitted by the light-projection means but to inhibit at least some other light.

32. Apparatus according to claim 31, wherein the light-projection means comprises n infra-red LED and said filter inhibits visible light but passes infra-red.

33. Apparatus according to claim 26 including means for applying correction factors to stored or measured marker positions to produce respective corrected marker positions and determining the object position based on the corrected marker positions, the means for applying correction factors comprising:

further memory for storing pre-stored or measured marker positions as initial marker positions;

means for dynamically applying correction factors to said initial marker positions to produce respective corrected marker positions;

means for determining the object position based on the corrected marker positions, wherein

said means for dynamically applying correction factors comprises means for calculating correction factors so that the corrected marker positions are corrected to become mutually self-consistent values, the rate of variation of the correction factors being limited to a pre-determined level by limiting the rate of change of correction factors.

34. Apparatus according to claim 26 arranged to determine the position of a video camera wherein said camera comprises a further camera mounted on the video camera, said camera facing in a different direction to the video camera.

35. Apparatus according to claim 34, including means for detecting the zoom and/or focus settings of the video camera.

37. Marker apparatus for positioning in the field of view of a video camera for use in determining the relative position of the camera and the marker apparatus, the apparatus comprising:

a base;

a first patterned marker mounted at a first predetermined relative positions on a substantially rigid base, the first patterned marker having identification information encoded thereon in binary form as a series of concentric rings of two tones, the tones being selected to enable the marker to be keyed out of the video camera image by chroma-keying, the center of the rings providing a reference point for the marker, the first marker being at a first height from the base;

a second patterned marker mounted at a second predetermined relative positions on a substantially rigid base, the second patterned marker having identification information encoded thereon in binary form as a series of concentric rings of two tones, the tones being selected to enable the marker to be keyed out of the video camera image by chroma-keying, the center of the rings providing a reference point for the marker, the second marker being at a second height from the base;

a third patterned marker mounted at a third predetermined relative positions on a substantially rigid base, the third patterned marker having identification information encoded thereon in binary form as a series of concentric rings of two tones, the tones being selected to enable the marker to be keyed out of the video camera image by chroma-keying, the center of the rings providing a reference point for the marker, the third marker being at a third height from the base;

wherein

said first height, said second height and said third height are all mutually different, whereby a three dimensional position fix relative to the base can be obtained from the markers.

38. A method of determining the relative three-dimensional positions of at least one camera and an object having at least three markers mounted thereon, the relative positions of the markers and the object being substantially fixed, each marker being patterned to encode identification information in binary form as the darkness or color of a series of concentric rings, the center of the rings providing a reference point, the method comprising:

storing the relative three dimensional positions of the markers with respect to the object and information enabling the identification information to be decoded;

receiving an image from the camera;

identifying the positions of the markers in the camera image;

decoding the identification information encoded on each marker;

determining the position of the reference point for each marker; and

based on the stored information, determining a measure of the relative three-dimensional positions and orientations of said at least one camera and object.