System for enhancing video from a mobile camera

• US 9,215,383 B2
• Filed: 07/18/2012
• Issued: 12/15/2015
• Est. Priority Date: 08/05/2011
• Status: Active Grant

• Current Assignee: Sportsmedia Technology Corporation
• Original Assignee: Sportsvision, Inc.
• Inventors: Milnes, Kenneth A., Honey, Stanley K., McGuffin, James O., Cavallaro, Richard H., Borton, John R., Pendleton, Rand
• Primary Examiner(s): Kostak, Victor
• Application Number:
  US13/551,997
• Publication Number:
  US 20130033598A1
• Time in Patent Office: 1,245 Days
• Field of Search: 348/144, 348/157, 340988-991, 340/995.1, 340/995.14, 340/995.24, 340/995.28, 340/539.13, 701/468, 701/482
• US Class Current: 1/1
• CPC Class Codes: G01C 21/203 Specially adapted for sailing ships
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G01S 19/53 Determining attitude
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control

Abstract

An automated system is configured to enhance live video in real time by adding virtual graphics to imagery obtained from a moving camera, where the added virtual graphics can represent real yet not visible attributes such as wind speed and direction and non-real attributes such as lines indicative of racing advantages. The displayed positions of the virtual graphics are dependent on sensor measurements of the locations and/or attitudes in a real world 3D coordinate system of objects and of the movable camera The displayed positions of the virtual graphics are functions of corresponding locations in the real world 3D coordinate system.

Claims

1. A method for enhancing video, comprising:

automatically sensing a location and orientation within a three-dimensional world space of a movable camera that can change locations, the camera generating camera-taken video including video imagery of an object visible in the camera-taken video and disposed within the world space;

receiving position data from a sensor indicative of a position within the world space of the visible object;

identifying one or more locations within the world space as being respectively related with at least one virtual graphic;

transforming one or more of the identified locations within the world space to a corresponding plurality of positions in a video coordinate system of a video image that includes at least part of imagery from the camera-taken video; and

producing an enhanced video image including the at least one virtual graphic disposed at one or more of the corresponding plural positions defined by said transforming and including said at least part of the imagery from the camera-taken video.

2. The method of claim 1, wherein:

the sensing of the location and orientation of the camera within the three-dimensional world space includes sensing the location and orientation of the camera in real time while the camera is changing location relative to a moving platform to which the camera is movably mounted.

3. The method of claim 1, wherein:

the location of the camera is sensed while the camera is unrestrained in a local space.

4. The method of claim 1, further comprising:

receiving orientation information for the object,

wherein the producing of the enhanced video image is partially based on the orientation information and the position data for the object, and

wherein the position data and the orientation information for the object are measured and received in real time while the object is moving.

5. The method of claim 4 wherein:

the position data identifies the location in the world space;

the camera is sensing video of an event that is depicted in the video image;

the video image depicts the object;

the producing of the enhanced video image is performed during the event; and

the producing of the enhanced video image enhancing includes adding a graphic to the video image such that object occludes the graphic in the video image.

6. The method of claim 1, further comprising:

sensing an atmospheric condition using a sensor,

wherein the producing of the enhanced video image includes creating a graphic based on the sensed atmospheric condition and adding the graphic to the video image.

7. The method of claim 1, wherein:

the camera is sensing video of an event; and

the producing of the enhanced video image is performed during the event.

8. The method of claim 1 wherein:

the producing of the enhanced video image is performed in real time.

9. The method of claim 1 wherein the sensing of the location of the camera comprises:

determining a location of a sensor; and

using the location of the sensor to determine a location of a camera base, the determined orientation of the camera is in relation to the camera base.

10. The method of claim 1 wherein the sensing of the location of the camera comprises:

determining a location of a sensor;

using the location of the sensor to determine a location of an inertial measurement system, the inertial measurement system is used to determine the orientation of the camera; and

using the location of the inertial measurement system to determine a location of a camera base, the determined orientation of the camera is in relation to the camera base.

11. The method of claim 1, further comprising:

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark;

automatically identifying one or more locations within the world space for lay lines based on the sensed wind direction and wind speed; and

automatically determining isochrons for the lay lines;

wherein the transforming of the one or more of the identified locations within the world space includes converting a set of locations representing the lay lines and isochrons to a set of positions in the video image based on the sensed position and determined orientation of the camera; and

wherein the producing of the enhanced video image includes automatically creating graphics of the lay lines and isochrons based on the set of positions and adding the graphics of the lay lines and isochrons to the video image.

12. The method of claim 11, further comprising:

automatically determining location of two more moving boats, the enhancing comprises adding the graphics of the lay lines and isochrons to the video image without occluding images of the boats.

13. The method of claim 1, further comprising:

automatically determining locations of two or more moving boats;

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark;

automatically determining lay lines based on sensed wind direction and wind speed; and

automatically determining advantage lines based on the lay lines and the locations of the two or more moving boats;

wherein the transforming of the one or more of the identified locations within the world space includes converting a set of locations representing the advantage lines to a set of positions in the video image based on the sensed position and sensed orientation of the camera;

wherein the producing of the enhanced video image includes automatically creating graphics of the advantage lines based on the set of positions and adding the graphics of the advantage lines to the video image.

14. The method of claim 1, further comprising:

automatically determining locations of one or more moving boats;

automatically sensing wind direction and wind speed;

automatically sensing a location of a mark; and

automatically determining lay lines based on sensed wind direction and wind speed;

wherein the transforming of the one or more of the identified locations within the world space includes converting at least a particular location representing a particular lay line to a particular position in the video image based on the sensed position and determined orientation of the camera;

wherein the producing of the enhanced video image includes automatically creating a graphic of the particular lay line and adding the graphic of the particular lay line to the video image based on the particular position.

15. An apparatus for enhancing video, comprising:

a first set of one or more sensors configured to sense location within a three-dimensional (3D) world space and to provide corresponding 3D location information for a movable camera that can change locations;

a second set of one or more sensors configured to sense position within the 3D world space and to provide corresponding 3D position information for one or more movable objects; and

one or more processors in communication with the first set of one or more sensors and the second set of one or more sensors, the one or more processors being configured to obtain a location in the 3D world space based on the position information from the second set of one or more sensors, the one or more processors being configured to convert the obtained location in the 3D world space to a corresponding position in a video image that includes video imagery obtained from the camera based on the sensed location of the camera and being configured to enhance the video image based on the position in the video image;

wherein the enhancing of the video image includes;

identifying one or more locations within the 3D world space as being respectively related with at least one virtual graphic;

transforming one or more of the identified locations within the 3D world space to a corresponding plurality of positions in a video coordinate system of the video image.

16. The apparatus of claim 15, wherein:

the one or more processors obtain the location in world space based on GPS data that identifies the location in world space.

17. The apparatus of claim 15, wherein:

the one or more processors obtain the location in world space by using GPS data that identifies a location that has a functional relationship with the location in world space.

18. The method of claim 15, further comprising:

a third set of one or more sensors that sense orientation information for the object, the one or more processors enhance the video image based on the orientation information, the position information and the orientation information are measured in real time while the object is moving.

19. The apparatus of claim 18, wherein:

the position information identifies the location in world space;

the video image depicts the object;

the one or more processors enhance the video image during the event; and

the one or more processors enhance the video image by adding a graphic to the video image such that object occludes the graphic in the video image.

20. The apparatus of claim 19, further comprising:

a fourth set of one or more sensors that sense an atmospheric condition, the one or more processors enhance the video image by creating a graphical image based on the sensed atmospheric condition and adding the graphical image to the video image.

21. The apparatus of claim 15, wherein:

the camera is sensing video of an event; and

the one or more processors enhance the video image during the event.

22. The apparatus of claim 15, wherein:

the one or more processors enhance the video image in real time.

23. The apparatus of claim 15, further comprising:

a set of one or more orientation sensors configured to provide information indicating an orientation of the camera within the 3D world space, wherein the provided orientation information is used for said transforming of the one or more of the identified locations within the 3D world space to a corresponding plurality of positions in a video coordinate system of the video image.

24. The apparatus of claim 23, wherein:

the one or more processors determine a current location of at least one of the first set of one or more sensors and use the current location to determine a location of a camera base, the orientation of the camera is in relation to the camera base.

25. The apparatus of claim 15, further comprising:

an inertial measurement system that provides information indicating an orientation of the camera, the one or more processors determine a current location of at least one of the first set of one or more sensors and use the current location to determine a location of the inertial measurement system, the one or more processors use the location of the inertial measurement system to determine a location of a camera base, the orientation of the camera is in relation to the camera base.

26. An apparatus for enhancing video, comprising:

a first set of one or more sensors configured to sense location within a three-dimensional (3D) world space and to provide corresponding 3D location information for a movable camera that is unrestrained in a local space;

a second set of one or more sensors configured to sense orientation within the 3D world space and to provide corresponding 3D orientation information for the camera, the first set of sensors and the second set of sensors are co-located with the camera on an aircraft;

a third set of one or more sensors configured to concurrently sense location within the 3D world space and to provide corresponding 3D object location information for multiple moving objects;

one or more communication stations in communication with the first set of one or more sensors, the second set of one or more sensors and the third set of one or more sensors; and

one or more processors in communication with the one or more communication stations, the one or more processors being configured to receive video from the camera, and to receive the 3D object location information for multiple moving objects;

the one or more processors being configured to convert the received 3D object location information of the moving objects into positions in a video image from the camera based on the location information for the camera and the orientation information for the camera, the one or more processors being configured to create one or more graphics based on the positions in the video image and add the one or more graphics to the video image;

wherein the creating of the graphics includes;

identifying one or more locations within the 3D world space as being respectively related to the graphics; and

transforming one or more of the identified locations within the 3D world space to a corresponding plurality of positions in a video coordinate system of the video image.