3D textured objects for virtual viewpoint animations

• US 8,073,190 B2
• Filed: 05/15/2008
• Issued: 12/06/2011
• Est. Priority Date: 11/16/2007
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Gloudemans, James R., Hsiao, Walter
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): GORADIA, SHEFALI DINESH
• Application Number:
  US12/121,655
• Publication Number:
  US 20090129630A1
• Time in Patent Office: 1,300 Days
• Field of Search: 382/100, 382/154, 345/419, 345/630
• US Class Current: 382/100
• CPC Class Codes: G06T 15/20 Perspective computation

Abstract

3d textured objects are provided for virtual viewpoint animations. In one aspect, an image of an event is obtained from a camera and an object in the image is automatically detected. For example, the event may be a sports event and the object may be a stationary object which is detected based on a known location, color and shape. A 3d model of the object is combined with a textured 3d model of the event to depict a virtual viewpoint which differs from a viewpoint of the camera. The textured 3d model of the event has texture applied from an image of the event, while the 3d model of the object does not have such texture applied, in one approach. In another aspect, an object in the image such as a participant in a sporting event is represented in the virtual viewpoint by a textured 3d kinematics model.

Claims

1. A method for providing image data of an event, comprising:

obtaining an image of the event from a camera, the image including a playing field, field lines on the playing field and a stationary structure on the playing field;

automatically detecting pixels in the image which make up the stationary structure; and

responsive to the automatically detecting, providing a 3d model of the stationary structure to replace the pixels in the image which make up the stationary structure; and

providing an animation which depicts the event from a viewpoint of the camera and from a virtual viewpoint which differs from the viewpoint of the camera, the animation includes the 3d model of the stationary structure, and a textured 3d model of the playing field and the field lines, wherein the textured 3d model of the playing field and the field lines has texture from one or more camera-captured images of the event, and the 3d model of the stationary structure does not have texture from one or more camera-captured images of the event.

2. The method of claim 1, wherein:

the stationary structure is automatically detected based on a known physical shape of the stationary structure.

3. The method of claim 1, wherein:

the stationary structure is automatically detected based on a known color of the stationary structure.

4. The method of claim 1, wherein:

the stationary structure is automatically detected by examining pixels in a portion of the obtained image of the event which correspond to a known location of the stationary structure on the playing field.

5. The method of claim 4, further comprising:

using registration data of the camera to assist in the automatically detecting.

6. The method of claim 1, wherein:

the event comprises a sporting event and the stationary structure comprises a goal structure at the sporting event.

7. The method of claim 1, further comprising:

removing the pixels in the obtained image of the event which make up the stationary structure; and

blending in the removed pixels with surrounding pixels in the image.

8. The method of claim 1, further comprising:

broadcasting the animation in a television signal.

9. The method of claim 1, wherein:

the playing field is in a stadium which includes grandstands; and

the animation includes a textured 3d model of the grandstands, the grandstands in the textured 3d model having video texture from one or more camera-captured images of the event.

10. The method of claim 1, further comprising:

detecting a participant on the playing field in the event; and

providing the animation with a textured 3d model of the participant, the participant in the textured 3d model having texture from one or more camera-captured images of the event.

11. The method of claim 10, wherein:

the textured 3d model of the participant comprises a 3d kinematics model.

12. The method of claim 11, wherein:

the 3d kinematics model is textured using images from multiple cameras.

13. The method of claim 11, wherein:

the 3d kinematics model is textured using the image obtained from the viewpoint of the camera.

14. A system for providing image data of an event, comprising:

at least one camera for capturing an image of the event, the image including a playing field, field lines on the playing field and a stationary structure on the playing field; and

at least one processing facility which receives the image, the at least one processing facility;

a) automatically detects the stationary structure in the image, and b) responsive to the automatically detecting, combines a 3d model of the stationary structure with a textured 3d model of the playing field and the field lines, to depict a virtual viewpoint which differs from a viewpoint of the camera, where the textured 3d model of the playing field and the field lines has texture applied from one or more camera-captured images of the event, and the 3d model of the at least a first object stationary structure does not have texture applied to it from one or more camera-captured images of the event.

15. The system of claim 14, wherein:

the stationary structure is automatically detected based on a known physical shape of the at least a first object.

16. The system of claim 14, wherein:

the stationary structure is automatically detected based on a known color of the stationary structure.

17. The system of claim 14, wherein:

the stationary structure is automatically detected based on a known location of the stationary structure in the event.

18. The system of claim 14, wherein:

the event comprises a sporting event and the stationary structure comprises a goal structure at the sporting event.

21. The method of claim 14, wherein:

the 3d model of the event includes a 3d model of a stationary structure in the event facility which does not have texture from one or more camera-captured images of the event.

19. A method for providing image data of an event, comprising:

obtaining an image of the event from a camera;

detecting a participant in the event in the image;

providing a 3d model of the event, the 3d model of the event includes a 3d model of an event facility having texture from one or more camera-captured images of the event;

providing a textured 3d kinematics model of the participant, the 3d kinematics model is based on movement of different parts of a body in a known manner relative to one another within certain limitations; and

combining the textured 3d model of the event and the textured 3d kinematics model, to depict at least one virtual viewpoint which differs from a viewpoint of the camera.

20. The method of claim 19, wherein:

the 3d kinematics model is textured using images from multiple cameras.

22. The method of claim 19, wherein:

the stationary structure comprises a goal structure on a playing field in the event facility; and

the 3d model of the event includes a 3d model of the playing field having texture from one or more camera-captured images of the event.