Telestrator system

• US 7,075,556 B1
• Filed: 10/21/1999
• Issued: 07/11/2006
• Est. Priority Date: 10/21/1999
• Status: Expired due to Term

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Hsiao, Walter, White, Marvin S., Gloudemans, James R., Cavallaro, Richard H., Honey, Stanley K., Meier, Kevin R.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/425,992
• Time in Patent Office: 2,455 Days
• Field of Search: 345/629, 345/630, 345/589, 345/591, 345/634, 345/419, 358/2.1, 358/2.99, 382/284, 382/276
• US Class Current: 345/629
• CPC Class Codes: H04N 5/262 Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A telestrator system is disclosed that allows a broadcaster to annotate video during or after an event. For example, while televising a sporting event, an announcer (or other user) can use the present invention to draw over the video of the event to highlight one or more actions, features, etc. In one embodiment, when the announcer draws over the video, it appears that the announcer is drawing on the field or location of the event. Such an appearance can be performed by mapping the pixels location from the user'"'"'s drawing to three dimensional locations at the event. Other embodiments include drawing on the video without obscuring persons and/or other specified objects, and/or smoothing the drawings in real time.

Claims

1. A method for annotating video, comprising:

receiving video, said video depicts a surface at a live event;

receiving a graphic manually created by a human operator during said live event; and

blending said graphic with said video such that said graphic appears to be drawn on said surface, said step of blending being performed during said live event;

wherein said step of blending includes performing a flicker filter.

2. A method for annotating video, comprising:

receiving video, said video depicts a surface at a live event, said video is captured by a first camera;

receiving a graphic manually created by a human operator during said live event, said graphic includes a curve, said curve is represented as a set of quadrilaterals; and

blending said graphic with said video such that said graphic appears to be drawn on said surface, said step of blending being performed during said live event, said step of blending includes tessellating said quadrilaterals if said first camera has been zoomed past a threshold.

3. A method for annotating video, comprising the steps of:

receiving video, at least a portion of said video depicts at least a portion of a surface at a live event and a set of one or more objects occluding said surface;

receiving a graphic manually created by a human operator during said live event; and

blending said graphic with said portion of said video without drawing said graphic over said objects, said steps of receiving video, receiving a graphic and blending are performed during said live event.

4. A method according to claim 3, wherein:

said step of receiving a graphic includes receiving two dimensional position information for said graphic created in relation to a two dimensional image.

5. A method according to claim 4, further comprising the steps of:

converting a first set of one or more two dimensional positions to one or more three dimensional locations, said first set of one or more two dimensional positions are based on said two dimensional position information for said graphic; and

converting said one or more three dimensional locations to a second set of one or more two dimensional positions, said step of blending includes blending said graphic with said video based on said second set of one or more two dimensional positions.

6. A method according to claim 5, wherein:

said two dimensional image is part of a first video frame; and

said step of blending includes blending said graphic with a second video frame different from said first video frame.

7. A method according to claim 3, further comprising the step of:

smoothing said graphic prior to said step of blending, said graphic is a drawing.

8. A method according to claim 3, wherein said step of blending includes:

adding said graphic to a first video image at a first position in said first video image;

adding said graphic to a second video image at a second position in said second video image;

said first position is different than said second position;

said first position corresponds to a position of an image of a first portion of said surface in said first video image; and

said second position corresponds to a position of said image of said first portion of said surface in said second video image.

9. A method according to claim 3, wherein said step of blending includes the steps of:

accessing color data for said video;

using said color data to access stored blending values, said stored blending values are organized by colors; and

causing a blending of said graphic with said video based on using said blending values.

10. A method according to claim 9, further comprising the steps of:

receiving a selection of a portion of an image, said portion of said image includes a set of colors;

receiving a first blending value for said set of colors; and

storing said first blending values.

11. A method according to claim 9, wherein:

said step of using includes the steps of;

comparing a first color value for a first pixel in said video to a color map, said color map stores color values and corresponding blending values, and

identifying a first blending value corresponding to said first color value based on said color map; and

said step of causing a blending includes using said first blending value as part of a process to blend said first pixel with a corresponding pixel in said graphic.

12. A method for annotating video, comprising the steps of:

receiving video, said video depicts a surface at said live event;

receiving two dimensional position information for at least a portion of a graphic created in relation to a two dimensional image;

converting a first set of one or more two dimensional positions to one or more three dimensional locations, said one or more two dimensional positions correspond to said two dimensional position information;

converting said one or more three dimensional locations to a second set of one or more two dimensional positions; and

blending said graphic with said video based on said second set of one or more two dimensional positions.

13. A method according to claim 12, further comprising the steps of:

receiving camera sensor data for a first camera, said video being captured by said first camera, said step of converting said one or more three dimensional locations to a second set of one or more two dimensional positions is performed using said camera sensor data for said first camera.

14. A method according to claim 13, further comprising the steps of:

receiving camera sensor data for a second camera, said two dimensional image being captured by said second camera, said step of converting said first set of one or more two dimensional positions to one or more three dimensional locations uses said camera sensor data for said second camera.

15. A method according to claim 12, further comprising the steps of:

receiving camera sensor data for a first camera, said two dimensional image being captured by said first camera, said step of converting said first set of one or more two dimensional positions to one or more three dimensional locations uses said camera data for said first camera.

16. A method according to claim 12, wherein:

said two dimensional position information is different than said first set of one or more two dimensional positions.

17. A method according to claim 12, further comprising the steps of:

dividing said graphic into segments based on said two dimensional position information, said first set of one or more two dimensional positions are end points of said segments; and

thickening said graphic after said step of converting a first set of one or more two dimensional positions to one or more three dimensional locations.

18. A method according to claim 12, wherein:

said two dimensional image is part of a first video frame; and

said step of blending includes blending said graphic with a second video frame different from said first video frame.

19. A method according to claim 12, wherein:

said two dimensional image is part of a first video frame from a first camera; and

said step of blending includes blending said graphic with a second video frame from a second camera.

20. A method according to claim 12, wherein:

said first set of one or more two dimensional positions pertains to only a new portion of said graphic; and

said one or more three dimensional locations pertain to said graphic as a whole.

21. A method according to claim 12, further comprising the step of:

creating a three dimensional model of at least a portion of said surface, said step of converting said first set of one or more two dimensional positions to one or more three dimensional locations uses said model.

22. A method according to claim 12, further comprising the step of:

smoothing said graphic prior to said step of blending, said graphic is a drawing.

23. A method for annotating video, comprising:

receiving video, said video depicts a surface at a live event;

receiving at least a portion of a drawing manually created by a human operator during said live event;

smoothing said drawing, said smoothing includes;

receiving a new point,

adding said new point to a control group if there are is a sufficient distance between said new point and another point in said control group,

drawing a line between said new point and said another point in said control group if there is not more than two points in said control group, and

removing a next to last point in said control group and fitting a Bezier spline through points in said control group, if there are more than two points in said control, group; and

blending said smoothed drawing with said video during said live event.

24. A method for blending images, comprising the steps of:

storing blending values for a set of colors;

receiving a first image;

receiving a second image after said step of storing;

accessing color data for said second image;

comparing said color data to said set of colors; and

causing a blending of said first image with said second image based on using one or more blending values for colors of said set of colors that match said color data for said second image.

25. A method according to claim 24, further comprising the steps of:

receiving a selection of a portion of a third image, said portion of said third image includes a subset of said set of colors; and

receiving one or more blending values for said subset of said set of colors.

26. A method according to claim 25, wherein:

said second image is part of a video of an event; and

said first image is an effect to be added to said video of said event.

27. A method according to claim 26, wherein:

said video is live video; and

said step of receiving one or more blending values includes receiving a number from an operator indicating how to blend said first image with said second image.

28. A method according to claim 24, wherein:

said blending values are alpha percentages.

29. A method according to claim 24, wherein:

said step of comparing includes the steps of;

comparing a first color value for a first pixel in said second image to a color map, said color map stores color values for said set of colors and said blending values for said set of colors, and

identifying a first blending value corresponding to said first color value based on said color map; and

said step of causing a blending includes using said first blending value as part of a process to blend said first pixel with a corresponding pixel in said first image.

30. A method according to claim 24, wherein:

said steps of accessing and comparing include the steps of;

accessing a first pixel in said second image, said first pixel having a first color value,

accessing additional pixels nearby to said first pixel in said second image, said additional pixels having additional color values,

comparing a first color value for said first pixel to a color map, said color map stores color values for said set of colors and said blending values for said set of colors,

comparing said additional color values to said color map,

identifying a first blending value corresponding to said first color value based on said color map,

identifying additional blending values corresponding to said additional color values based on said color map,

calculating an average of said first blending value and said additional blending values, and

adjusting said average based on a weighting factor; and

said step of causing a blending includes using said adjusted average to blend said first pixel with a corresponding pixel in said first image.

31. A method according to claim 24, wherein:

said step of causing a blending includes performing a flicker filter.

32. A method according to claim 24, wherein:

said set of colors includes a first subset of colors and a second subset of colors;

said first subset of colors is associated with a first blending value; and

said second subset of colors represents a taper zone having blending values ranging from at or near said first blending value toward a second blending value, said second blending value represents no blending of said first image with said second image.

33. A method according to claim 24, wherein:

said set of colors includes multiple visibly distinct colors.

34. A method according to claim 24, wherein:

said step of accessing color data includes accessing color data for a first portion of said second image, said first portion of said second image corresponds in position to said first image; and

said step of causing a blending includes causing said first image to be blended with said first portion of said second image and not other portions of said second image.

35. A method according to claim 24, wherein:

said second image is a live video image of a live event; and

said steps of storing, receiving a first image, receiving a second image, accessing, comparing and causing are performed during said live event.

36. A method according claim 35, further including the step of:

updating said blending values for said set of colors during said live event.

37. An apparatus, comprising:

one or more processors;

an input device in communication with said one or more processors;

an output device in communication with said one or more processors; and

at least one storage device in communication with said one or more processors, said one or more processors perform a method comprising the steps of;

storing blending values for a set of colors,

receiving a first image,

receiving a second image,

accessing color data for said second image,

comparing said color data to said set of colors, and

causing a blending of said first image with said second image based on using one or more blending values for colors of said set of colors that match said color data for said second image.

38. An apparatus according to claim 37, wherein said method further includes the steps of:

receiving a selection of a portion of a third image, said portion of said third image includes a subset of said set of colors; and

receiving one or more blending values for said subset of said set of colors.

39. An apparatus according to claim 37, wherein:

said step of comparing includes the steps of;

comparing a first color value for a first pixel of said second image to a color map, said color map stores color values for said set of colors and said blending values for said set of colors, and

identifying a first blending value corresponding to said first color value based on said color map; and

said step of causing a blending includes using said first blending value as part of a process to blend said first pixel with a corresponding pixel in said first image.

40. An apparatus according to claim 37, wherein:

said steps of accessing and comparing include the steps of;

accessing a first pixel in said second image, said first pixel having a first color value,

accessing additional pixels nearby to said first pixel in said second image, said additional pixels having additional color values,

comparing a first color value for said first pixel to a color map, said color map stores color values for said set of colors and said blending values for said set of colors,

comparing said additional color values to said color map,

identifying a first blending value corresponding to said first color value based on said color map,

identifying additional blending values corresponding to said additional color values based on said color map,

calculating an average of said first blending value and said additional blending values, and

adjusting said average based on a weighting factor; and

said step of causing a blending includes using said adjusted average to blend said first pixel with a corresponding pixel in said first image.

41. An apparatus according to claim 37, wherein:

said step of causing a blending includes performing a flicker filter.

42. An apparatus according to claim 37, wherein:

said set of colors includes a first subset of colors and a second subset of colors;

said first subset of colors is associated with a first blending value; and

said second subset of colors represents a taper zone having blending values ranging from at or near said first blending value to a second blending value, said second blending value represents no or minimal blending of said first image with said second image.

43. An apparatus according to claim 37, wherein:

set of colors includes multiple visibly distinct colors.

44. An apparatus according to claim 37, wherein:

said step of accessing color data includes accessing color data for a first portion of said second image, said first portion of said second image corresponds in position to said first image; and

said step of causing a blending includes causing said first image to be blended with said first portion of said second image and not other portions of said second image.

45. An apparatus according to claim 37, wherein:

said second image is a live video image of a live event; and

said steps of storing, receiving a first image, receiving a second image, accessing, comparing and causing are performed during said live event.

46. An apparatus according claim 45, wherein said method further includes the step of:

updating said blending values for said set of colors during said live event.

47. An apparatus, comprising:

one or more processors;

a drawing device in communication with said one or more processors;

an output device in communication with said one or more processors; and

at least one storage device in communication with said one or more processors, said one or more processors perform a method comprising the steps of;

receiving video, at least a portion of said video depicts at least a portion of a surface at a live event and a set of one or more objects occluding said surface,

receiving a graphic manually created by a human operator during said live event, and

causing a blending of a graphic with said portion of said video without drawing said graphic over said objects, said steps of receiving video, receiving a graphic and causing are performed during said live event.

48. An apparatus according to claim 47, wherein said step of causing a blending includes the steps of:

accessing color data for said video;

using said color data to access stored blending values, said stored blending values are organized by colors; and

causing a blending of said graphic with said video based on using said blending values.

49. An apparatus according to claim 48, wherein said method further includes the steps of:

receiving a selection of a portion of an image, said portion of said image includes a set of colors;

receiving a first blending value for said set of colors; and

storing said first blending value.

50. An apparatus according to claim 49, wherein:

said steps of accessing and using include the steps of;

accessing a first pixel in said video,

comparing a first color value for said first pixel to a color map, said color map stores color values and corresponding blending values, and

identifying a first blending value corresponding to said first color value based on said color map; and

said step of causing a blending includes using said first blending value as part of a process to blend said first pixel with a corresponding pixel in said graphic.

51. An apparatus according to claim 47, wherein:

said step of receiving a graphic includes receiving two dimensional position information for said graphic created in relation to a two dimensional image; and

said method further includes the steps of;

converting a first set of one or more two dimensional positions to one or more three dimensional locations, said first set of one or more two dimensional positions correspond to said two dimensional position information for said graphic, and

converting said one or more three dimensional locations to a second set of one or more two dimensional positions, said step of blending includes blending said graphic with said video based on said second set of one or more two dimensional positions.

52. An apparatus according to claim 47, wherein said method further includes the steps of:

smoothing said graphic prior to said step of blending, said graphic is a drawing.

53. An apparatus according to claim 47, wherein said step of causing a blending includes:

adding said graphic to a first video image at a first position in said first video image;

adding said graphic to a second video image at a second position in said second video image;

said first position is different than said second position;

said first position corresponds to a position of an image of a first portion of said surface in said first video image; and

said second position corresponds to a position of said image of said first portion of said surface in said second video image.

54. An apparatus, comprising:

one or more processors;

a drawing device in communication with said one or more processors;

an output device in communication with said one or more processors; and

at least one storage device in communication with said one or more processors, said one or more processors perform a method comprising the steps of;

receiving video, said video depicts a surface at said live event,

receiving two dimensional position information for at least a portion of a graphic created in relation to a two dimensional image,

converting a first set of one or more two dimensional positions to one or more three dimensional locations, said first set of one or more two dimensional positions correspond to said two dimensional position information,

converting said one or more three dimensional locations to a second set of one or more two dimensional positions, and

causing a blending of said graphic with said video based on said second set of one or more two dimensional positions.

55. An apparatus according to claim 54, further comprising:

a first set of one or more camera view sensors, said method further includes the step of receiving camera sensor data for a first camera from said first set of one or more camera view sensors, said video being captured by said first camera, said step of converting said one or more three dimensional locations to a second set of one or more two dimensional positions is performed based on said camera sensor data for said first camera.

56. An apparatus according to claim 55, further comprising:

a second set of one or more camera view sensors, said method further including the step of receiving camera sensor data for a second camera from said second set of one or more camera view sensors, said two dimensional image being captured by said second camera, said step of converting said first set of one or more two dimensional positions to one or more three dimensional locations is performed based on said camera sensor data for said second camera.

57. An apparatus according to claim 54, further comprising:

a first set of one or more camera view sensors, said method further including the step of receiving camera sensor data for a first camera from said first set of one or more camera view sensors, said two dimensional image being captured by said first camera, said step of converting said first set of one or more two dimensional positions to one or more three dimensional locations is performed based on said camera sensor data for said first camera.

58. An apparatus according to claim 54, wherein:

said first set of one or more two dimensional positions pertains to only a new portion of said graphic; and

said one or more three dimensional locations pertain to said graphic in its entirety.

59. An apparatus according to claim 54, wherein said method further includes the step of:

smoothing said graphic prior to said step of blending, said graphic is a drawing.

60. A method for annotating video, comprising:

receiving video, said video depicts a surface at a live event;

manually adding a graphic to said video during said live event; and

maintaining said graphic in said video such that said graphic appears to be drawn on said surface, said manually adding and maintaining includes adding a pre-defined image into said video;

wherein said step of adding a pre-defined image includes determining an amount of time that a touch screen is engaged and adding said pre-defined image if said touch screen is engaged for less than a predetermined amount of time.

61. A method for annotating video, comprising:

receiving video, said video depicts a surface at a live event;

manually adding a graphic to said video during said live event; and

maintaining said graphic in said video such that said graphic appears to be drawn on said surface, said manually adding and maintaining includes adding a pre-defined image into said video;

wherein adding a pre-defined image includes;

receiving a first two dimensional position in said video corresponding to said pre-defined image,

converting said two dimensional position to a three dimensional location in relation to said live event,

converting said three dimensional location to a second two dimensional position in said video, and

blending said pre-defined image with said video at said second two dimensional position.

62. A method according to claim 61, further comprising the steps of:

receiving camera sensor data for a first camera, said video being captured by said first camera, said step of converting said three dimensional location is based on said camera sensor data for said first camera.

63. A method according to claim 61, wherein:

said pre-defined image is initially added to a first video field; and

said step of maintaining includes blending said pre-defined image with a second video field later in time than said first video field.

64. A method according to claim 61, wherein:

said pre-defined image is initially added to a first video field from a first camera; and

said step of maintaining includes blending said pre-defined image with a second video field from a second camera.

65. A method according to claim 61, further comprising the step of:

creating a model of at least a portion of said surface, said step of converting two dimensional position to a three dimensional locations is performed using said model.

66. An apparatus for annotating video, comprising:

a storage medium; and

one or more processors in communication with said storage medium, said one or more processor perform a method comprising;

receiving video, said video depicts a surface at a live event,

manually adding a graphic to said video during said live event, and

maintaining said graphic in said video such that said graphic appears to be drawn on said surface, said manually adding and maintaining include adding a pre-defined image into said video;

wherein prior to adding said graphic to said video, at least a portion of said video depicts at least a portion of said surface at said live event and a set of one or more objects occluding said surface; and

wherein said steps of adding and maintaining include adding said pre-defined image with said portion of said video without drawing said pre-defined image over said objects.

67. An apparatus for annotating video, comprising:

a storage medium; and

one or more processor in communication with said storage medium, said one or more processor perform a method comprising;

receiving video, said video depicts a surface at a live event,

manually adding a graphic to said video during said live event, and

maintaining said graphic in said video such that said graphic appears to be drawn on said surface, said manually adding and maintaining include adding a pre-defined image into said video;

wherein said steps of manually adding and maintaining are performed without using image recognition.

68. One or more processor readable storage devices for storing processor readable code, said processor readable code for programming one or more processors to perform a method comprising:

receiving video, said video depicts a surface at a live event;

providing for the manual addition of a graphic to said video during said live event; and

maintaining said graphic in said video such that said graphic appears to be drawn on said surface, said providing and maintaining include adding a pre-defined image into said video;

wherein prior to adding said graphic to said video, at least a portion of said video depicts at least a portion of said surface at said live event and a set of one or more objects occluding said surface; and

wherein said step of maintaining include adding said pre-defined image with said portion of said video without drawing said pre-defined image over said objects.