SYSTEM FOR ENHANCING VIDEO FROM A MOBILE CAMERA

• US 20130033598A1
• Filed: 07/18/2012
• Published: 02/07/2013
• Est. Priority Date: 08/05/2011
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportvision, Inc. (Sportsmedia Technology Corporation)
• Inventors: Milnes, Kenneth A., Honey, Stanley K., McGuffin, James O., Cavallaro, Richard H., Pendleton, Rand, Borton, John R.
• Granted Patent: US 9,215,383 B2
• Time in Patent Office: Days
• US Class Current: 348/144
• CPC Class Codes: G01C 21/203 Specially adapted for sailing ships
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

A system for enhancing video can insert graphics, in perspective, into the video where the positions of the graphics are dependent on sensor measurements of the locations and/or attitudes of objects and the movable camera capturing the video.

Claims

1. A method for enhancing video, comprising:

automatically sensing a location of a movable camera that can change locations;

receiving position data from a sensor for an object;

converting a location in world space to a position in a video image of the camera based on the sensed location of the camera, the location in world space is based on the position data; and

enhancing the video image based on the position.

2. The method of claim 1, wherein:

the sensing the location of the camera includes sensing the location of the camera while the camera is changing location.

3. The method of claim 1, wherein:

the location of the camera is sensed while the camera is unrestrained in a local space.

4. The method of claim 1, further comprising:

receiving orientation information for the object, the enhancing is partially based on the orientation information, the position data and orientation information are measured and received in real time while the object is moving.

5. The method of claim 4, wherein:

the object is in the field of view of the camera.

6. The method of claim 5, wherein:

the position data identifies the location in world space;

the camera is sensing video of an event that includes the video image;

the video image depicts the object;

the enhancing is performed during the event; and

the enhancing includes adding a graphic to the video image such that object occludes the graphic in the video image.

7. The method of claim 1, further comprising:

sensing an atmospheric condition using a sensor, the enhancing includes creating a graphic based on the sensed atmospheric condition and adding the graphic to the video image.

8. The method of claim 1, wherein:

the camera is sensing video of an event; and

the enhancing is performed during the event.

9. The method of claim 8, wherein:

the enhancing is performed in real time.

10. The method of claim 1, further comprising:

determining an orientation of the camera, the location in world space is converted to the position in the video image of the camera based on the sensed location of the camera and the determined orientation of the camera.

11. The method of claim 10, wherein the sensing the location of the camera comprises:

determining a location of a sensor; and

using the location of the sensor to determine a location of a camera base, the determined orientation of the camera is in relation to the camera base.

12. The method of claim 10, wherein the sensing the location of the camera comprises:

determining a location of a sensor;

using the location of the sensor to determine a location of an inertial measurement system, the inertial measurement system is used to determine the orientation of the camera; and

using the location of the inertial measurement system to determine a location of a camera base, the determined orientation of the camera is in relation to the camera base.

13. The method of claim 1, further comprising:

determining an orientation of the camera;

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark;

automatically determining lay lines based on sensed wind direction and wind speed; and

automatically determining isochrons for the lay lines;

wherein the converting the location in world space includes converting a set of locations representing the lay lines and isochrons to a set of positions in the video image based on the sensed position and determined orientation of the camera;

wherein the enhancing includes automatically creating graphics of the lay lines and isochrons based on the set of positions and adding the graphics of the lay lines and isochrons to the video image.

14. The method of claim 13, further comprising:

automatically determining location of two more moving boats, the enhancing comprises adding the graphics of the lay lines and isochrons to the video image without occluding images of the boats.

15. The method of claim 1, further comprising:

determining an orientation of the camera;

automatically determining locations of two or more moving boats;

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark;

automatically determining lay lines based on sensed wind direction and wind speed; and

automatically determining advantage lines based on the lay lines and the locations of the two or more moving boats;

wherein the converting the location in world space includes converting a set of locations representing the advantage lines to a set of positions in the video image based on the sensed position and sensed orientation of the camera;

wherein the enhancing includes automatically creating graphics of the advantage lines based on the set of positions and adding the graphics of the advantage lines to the video image.

16. The method of claim 1, further comprising:

determining an orientation of the camera;

automatically determining a location of a moving boats;

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark; and

automatically determining lay lines based on sensed wind direction and wind speed;

wherein the converting the location in world space includes converting at least a particular location representing a particular lay line to a particular position in the video image based on the sensed position and determined orientation of the camera;

wherein the enhancing includes automatically creating a graphic of the particular lay line and adding the graphic of the particular lay line to the video image based on the particular position.

17. An apparatus for enhancing video, comprising:

a first set of one or more sensors that sense location information for a movable camera that can change locations;

a second set of one or more sensors that sense position information for one or more objects; and

one or more processors in communication with the first set of one or more sensors and the second set of one or more sensors, the one or more processors obtain a location in world space based on the position information from the second set of one or more sensors, the one or more processors convert the location in world space to a position in a video image of the camera based on the sensed location of the camera and enhance the video image based on the position in the video image.

18. The apparatus of claim 17, wherein:

the one or more processors obtain the location in world space based on GPS data that identifies the location in world space.

19. The apparatus of claim 17, wherein:

the one or more processors obtain the location in world space by using GPS data that identifies a location that has a functional relationship with the location in world space.

20. The method of claim 17, further comprising:

a third set of one or more sensors that sense orientation information for the object, the one or more processors enhance the video image based on the orientation information, the position information and the orientation information are measured in real time while the object is moving.

21. The apparatus of claim 20, wherein:

the position information identifies the location in world space;

the video image depicts the object;

the one or more processors enhance the video image during the event; and

the one or more processors enhance the video image by adding a graphic to the video image such that object occludes the graphic in the video image.

22. The apparatus of claim 21, further comprising:

a fourth set of one or more sensors that sense an atmospheric condition, the one or more processors enhance the video image by creating a graphical image based on the sensed atmospheric condition and adding the graphical image to the video image.

23. The apparatus of claim 17, wherein:

the camera is sensing video of an event; and

the one or more processors enhance the video image during the event.

24. The apparatus of claim 17, wherein:

the one or more processors enhance the video image in real time.

25. The apparatus of claim 17, further comprising:

a set of one or more orientation sensors that provide information indicating an orientation of the camera, the location in real space is converted to the position in the video image of the camera based on the sensed location of the camera and the orientation of the camera.

26. The apparatus of claim 25, wherein:

the one or more processors determine a current location of at least one of the first set of one or more sensors and use the current location to determine a location of a camera base, the orientation of the camera is in relation to the camera base.

27. The apparatus of claim 17, further comprising:

an inertial measurement system that provides information indicating an orientation of the camera, the one or more processors determine a current location of at least one of the first set of one or more sensors and use the current location to determine a location of the inertial measurement system, the one or more processors use the location of the inertial measurement system to determine a location of a camera base, the orientation of the camera is in relation to the camera base.

28. An apparatus for enhancing video, comprising:

a first set of one or more sensors that sense location information for a movable camera that is unrestrained in a local space;

a second set of one or more sensors that sense orientation information for the camera, the first set of sensors and the second set of sensors are co-located with the camera on an aircraft;

a third set of one or more sensors that concurrently sense location information for multiple moving objects;

one or more communication stations in communication with the first set of one or more sensors, the second set of one or more sensors and the third set of one or more sensors; and

one or more processors in communication with the one or more communication stations, the one or more processors receive video from the camera, the one or more processors convert locations of the moving objects into positions in a video image from the camera based on the location information for the camera and the orientation information for the camera, the one or more processors create one or more graphics based on the positions in the video image and add the one or more graphics to the video image.