Measurments using a single image

• US 20070002039A1
• Filed: 06/30/2005
• Published: 01/04/2007
• Est. Priority Date: 06/30/2005
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Pendleton, Rand, White, Marvin
• Granted Patent: US 7,680,301 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 7/536 from perspective effects, e.g. by using vanishing points

Abstract

A method used in broadcasts of events is disclosed for identifying the coordinates of an object in world space from a video frame, where the object is not on the geometric model of the environment. Once the world coordinates of the object are identified, a graphic may be added to a video replay showing the object. The method may also be expanded in a further embodiment to identify a trajectory of an object over time moving through world space from video images of the start and end of the trajectory, where the object is not on the geometric model of the environment. Once the trajectory of the object in world space is identified, a graphic may be added to a video replay showing the trajectory.

Claims

1. A method of determining a location within an environment having a known geometry, the method comprising the step of:

(a) determining the a 3-dimensional location of an object not residing on the known geometry using the known geometry and only one image.

2. A method of determining a location as recited in claim 1, further comprising the steps of:

(b) receiving a first image from a camera, said only one image being from the camera;

(c) converting the 3-dimensional location of the object to a position in the first image corresponding to an image of the object; and

(d) modifying the first image based on the position in the first image in order to identify the object in the first image.

3. A method of determining a location as recited in claim 1, said step a of determining the a 3-dimensional location of an object comprising the steps of:

(a1) calculating a first ray from the camera through the object;

(a2) identifying a point on the known geometry relating to a projection of the object in a known direction onto the known geometry;

(a3) calculating a second ray in a direction of the projection of the object; and

(a4) calculating a point on the first ray that represents the point of closest approach to the second ray, the point calculated in said step (a4) representing a 3-dimensional position of the object in the environment.

4. A method of determining a location as recited in claim 3, said step (a1) of calculating the first ray comprising the step of selecting a point in the 2-dimensional video frame, and converting from that point to the first ray using known transformation equations for the camera within the environment.

5. A method of determining a location as recited in claim 3, said step (a2) of identifying a point on the known geometry comprising the step of selecting a 2-dimensional point in the video frame, converting from the 2-dimensional point to a line of position ray from the camera through a 3-dimensional point indicated by the 2-dimensional point using known transformation equations for the camera within the environment, and then calculating a location of intersection between the line of position through the 3-dimensional point and the known geometry.

6. A method of making a measurement relating to a position of an object within an environment from a known geometry within the environment, the object not residing on the known geometry, the method comprising the steps of:

(a) calculating a 3-dimensional position of the object from the known geometry and a video image from a single camera calibrated within the environment; and

(b) associating a time with the video image.

7. A method of making a measurement relating to a position of an object within an environment as recited in claim 6, said step (a) of calculating a 3-dimensional position of the object from the known geometry and a video image from a single camera calibrated within the environment comprising the step of calculating a first ray in 3-dimensional space passing from the camera through the object.

8. A method of making a measurement relating to a position of an object within an environment as recited in claim 7, said step (a) of calculating a 3-dimensional position of the object from the known geometry and a video image from a single camera calibrated within the environment further comprising the step of identifying a 3-dimensional location of a point on the known geometry, a second ray passing in a known direction from the point through or near the first ray.

9. A method of making a measurement relating to a position of an object within an environment as recited in claim 8, said step (a) of calculating a 3-dimensional position of the object from the known geometry and a video image from a single camera calibrated within the environment further comprising the step of calculating a point on the first ray that represents the point of closest approach to the second ray, the point representing the point of closest approach representing a 3-dimensional position of the object in the environment.

10. A method of making a measurement relating to a position of an object as recited in claim 7, said step (a) of calculating the first ray comprising the step of selecting a point in the 2-dimensional video image, and converting from that point to the first ray using known transformation equations for the camera within the environment.

11. A method of making a measurement relating to a position of an object as recited in claim 8, said step (b) of identifying a point on the known geometry comprising the step of selecting a 2-dimensional point in the video image, converting from the 2-dimensional point to a line of position ray from the camera through a 3-dimensional point indicated by the 2-dimensional point using known transformation equations for the camera within the environment, and then calculating a location of intersection between the line of position through the 3-dimensional point and the known geometry.

12. A method of making a measurement relating to a position of an object within an environment as recited in claim 6, wherein the object is a football.

13. A method of making a measurement relating to a position of an object within an environment as recited in claim 6, wherein the video image is a video frame generated during a broadcast of a sporting event.

14. A method of making a measurement relating to a position of an object within an environment as recited in claim 6, wherein the video image is a video frame generated during a broadcast of a football game.

15. A method of making measurements relating to one or more positions of an object within an environment from a known geometry within the environment, the object moving in a trajectory in the environment over a time t₁ to a time t₂, the method comprising the steps of:

(a) calculating a 3-dimensional position of the object at time t₁ from the known geometry and a first video image from a single camera calibrated within the environment;

(b) associating the time t₁ with the first video image;

(c) calculating a 3-dimensional position of the object at time t₂ from the known geometry and a second video image from the single camera;

(d) associating the time t₂ with the second video image;

(e) calculating the trajectory of the object from time t₁ to time t₂ from the 3-demensional positions of the object at time t₁ and time t₂, an elapsed time between time t₁ and time t₂ and known forces acting on the object as it moves through the trajectory.

16. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 15, said step (e) of calculating the trajectory of the object further comprising the step of using information of the position of the object at a time t₃ between the times t₁ and t₂.

17. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 16, said step of using information of the position of the object at a time t₃ comprising the step of using information of the 2-dimensional position of the object at the time t₃.

18. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 16, said step of using information of the position of the object at a time t₃ comprising the step of using information of the intersection of a 3-dimensional line of position determined for the object at t₃ with a determined plane in which the object is moving.

19. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 15, said step (a) of calculating a 3-dimensional position of the object at time t₁ from the known geometry and a first video image from a single camera calibrated within the environment comprising the steps of:

(a)(1) calculating a first ray in 3-dimensional space passing from the camera through the object at time to;

(a)(2) identifying a 3-dimensional location of a point on the known geometry at time t₁, a second ray passing in a known direction from the point through or near the first ray; and

(a)(3) calculating a point on the first ray that represents the point of closest approach to the second ray, the point representing the point of closest approach representing a 3-dimensional position of the object in the environment at time t₁.

20. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 19, said step (c) of calculating a 3-dimensional position of the object at time t₂ from the known geometry and a second video image from the single camera calibrated within the environment comprising the steps of:

(c)(1) calculating a third ray in 3-dimensional space passing from the camera through the object at time t₂;

(c)(2) identifying a 3-dimensional location of a point on the known geometry at time t₂, a fourth ray passing in a known direction from the point identified in said step (c)(2) through or near the third ray; and

(c)(4) calculating a point on the third ray that represents the point of closest approach to the fourth ray, the point representing the point of closest approach representing the 3-dimensional position of the object in the environment at time t₂.

21. A method of making measurements relating to one or more positions of an object as recited in claim 19, said step (a)(1) of calculating the first ray comprising the step of selecting a point in the first video image, and converting from that point to the first ray using known transformation equations for the camera within the environment.

22. A method of making measurements relating to one or more positions of an object as recited in claim 19, said step (a)(2) of identifying a point on the known geometry comprising the step of selecting a 2-dimensional point in the first video image, converting from the 2-dimensional point to a line of position ray from the camera through a 3-dimensional point indicated by the 2-dimensional point using known transformation equations for the camera within the environment, and then calculating a location of intersection between the line of position through the 3-dimensional point and the known geometry.

23. A method of making measurements relating to one or more positions of an object as recited in claim 20, said step (c)(1) of calculating the first ray comprising the step of selecting a point in the first video image, and converting from that point to the first ray using known transformation equations for the camera within the environment.

24. A method of making measurements relating to one or more positions of an object as recited in claim 20, said step (c)(2) of identifying a point on the known geometry comprising the step of selecting a 2-dimensional point in the second video image, converting from the 2-dimensional point to a line of position ray from the camera through a 3-dimensional point indicated by the 2-dimensional point using known transformation equations for the camera within the environment, and then calculating a location of intersection between the line of position through the 3-dimensional point and the known geometry.

25. A method of making measurements relating to one or more positions of within an environment as recited in claim 15, wherein the first and second video images are video frames generated during a broadcast of a sporting event.

26. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 15, wherein the first and second video images are video frames generated during a broadcast of a football game.

27. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 15, further comprising blending a graphic into video images between the first and second video images during a broadcast of the video images using the 4-dimensional measurement.

28. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 15, further comprising adding a visual indication of the trajectory of the object to video images between the first and second video images during a broadcast of the video images using the 4-dimensional measurement.

29. A method of making measurements relating to one or more positions of an object within an environment as recited in claim 28, said step of adding a visual indication of the trajectory of the object comprising the step of selecting a 3-dimensional position of the object at a given time, translating that 3-dimensional position to a 2-dimensional video image position, and adding the visual indication to the 2-dimensional video image position when the video image is broadcast.

30. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition, the playing field having a known geometry, a camera calibrated with respect to the known geometry capturing an event over a time t₁ to a time t₂ in a plurality of video images, the captured event including movement of the ball, the method comprising the steps of:

(a) selecting a first video image from the plurality of video images, the first video image depicting a start of the captured event at time t₁;

(b) calculating a 3-dimensional position of the ball at time t₁ from the known geometry and the first video image;

(c) associating the time t₁ with the first video image;

(d) selecting a second video image from the plurality of video images, the second video image depicting an end of the captured event at time t₂;

(e) calculating a 3-dimensional position of the ball at time t₂ from the known geometry and the second video image;

(f) associating the time t₂ with the second video image;

(g) calculating the movement of the ball from time t₁ to time t₂ from the 3-demensional positions of the ball at time t₁ and time t₂, an elapsed time between time t₁ and time t₂ and known forces acting on the ball as it moves.

31. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition as recited in claim 30, said step (a) of selecting a first video image from the plurality of video images comprises selecting a video image of a player throwing the ball.

32. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition as recited in claim 31, said step (d) of selecting a second video image from the plurality of video images comprises selecting a video image of a player catching the ball.

33. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition as recited in claim 30, said step (b) of calculating a 3-dimensional position of the ball at time t₁ comprising the steps of:

(b)(1) calculating a first ray in 3-dimensional space passing from the camera through the ball at time t₁;

(b)(2) identifying a 3-dimensional location of a point on the playing field at time t₁ where a player in possession or recent possession of the ball is in contact with the playing field, a second ray passing in a known direction from the point on the playing field through or near the first ray; and

(b)(3) calculating a point on the first ray that represents the point of closest approach to the second ray, the point representing the point of closest approach representing a 3-dimensional position of the ball at time t₁.

34. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition as recited in claim 33, said step (b)(1) of calculating the first ray comprising the step of selecting a point in the first video image at a center of the football and converting from that point to the first ray using known transformation equations for the camera.

35. A method of making measurements relating to one or more positions of a ball during a broadcast of a sporting competition as recited in claim 33, said step (b)(2) of identifying a 3-dimensional location of a point on the playing field comprising the step of selecting a 2-dimensional point in the first video image of the portion of the player in contact with the playing surface, converting from the 2-dimensional point to a line of position ray from the camera through a 3-dimensional point indicated by the 2-dimensional point using known transformation equations for the camera, and then calculating a location of intersection between the line of position through the 3-dimensional point and the playing surface.

36. A system for making measurements relating to movement of an object captured in a plurality of video images over a time t₁ to a time t₂ during a broadcast of an event using a known geometry of the broadcast event, the system comprising:

a camera for generating the plurality of video images, the camera being calibrated with respect to the known geometry;

a processing device for receiving video images from the camera;

a monitor associated with the processing device, the processor capable of displaying the plurality of video images on the monitor at a controlled rate;

a pointing device associated with the processing device for selecting points on one or more of the plurality of video images, the pointing device capable of selecting first and second points on a first video image of the plurality of video images and capable of selecting third and fourth points on a second video image of the plurality of video images;

a processing device for calculating a 3-dimensional position of the object at time t₁ from the first and second points selected in the first video image and a transformation matrix for converting between 2-dimensional video image space and 3-dimensional world space, and for calculating a 3-dimensional position of the object at time t₂ from the third and fourth points selected in the second video image and the transformation matrix;

a processing device for determining the movement of the object from the 3-dimensional positions of the object at times t₁ and t₂, the elapsed time between times t₁ and t₂ and known forces acting on the object between t₁ and t₂; and

a processing device for adding a graphic to a replay of the plurality of video images, the graphic visually indicating the motion of the object between t₁ and t₂.

37. A system for making measurements relating to movement of an object as recited in claim 36, the processing device for determining the movement of the object further using information of the position of the object at a time t₃ between the times t₁ and t₂.

38. A system for making measurements relating to movement of an object as recited in claim 36, wherein one or more of the processing device for receiving video images from the camera, the processing device for calculating a 3-dimensional position of the object, the processing device for determining the movement of the object and the processing device for adding a graphic to a replay of the plurality of video images comprise a single processing device.

39. A system for making measurements relating to movement of an object as recited in claim 36, wherein one or more of the processing device for receiving video images from the camera, the processing device for calculating a 3-dimensional position of the object, the processing device for determining the movement of the object and the processing device for adding a graphic to a replay of the plurality of video images comprise different processing devices.

40. A system for making measurements relating to movement of an object as recited in claim 36, wherein the calibrated camera is a stationarily mounted camera.

41. A system for making measurements relating to movement of an object as recited in claim 36, wherein the calibrated camera is an aerial camera mounted for at least one of translation and rotation.

42. A system for making measurements relating to movement of an object as recited in claim 36, wherein the event is a sporting event.

43. A system for making measurements relating to movement of an object as recited in claim 36, wherein the event is a football game.