SYSTEMS AND METHODS FOR PROVIDING IMMERSIVE DISPLAYS OF VIDEO CAMERA INFORMATION FROM A PLURALITY OF CAMERAS

• US 20130010144A1
• Filed: 09/14/2012
• Published: 01/10/2013
• Est. Priority Date: 04/16/2008
• Status: Active Grant

• Current Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Original Assignee: Johnson Controls Technology Company (Johnson Controls International plc)
• Inventors: Park, Youngchoon
• Granted Patent: US 8,428,391 B2
• Time in Patent Office: Days
• US Class Current: 348/218.1
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/695 Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A system for providing stitched video from a first camera and a second camera to an electronic display system includes a processing circuit configured to associate a view a first camera with an approximate location. The processing circuit is further configured to build relationship data between the first camera and a second camera using the approximate location. The processing circuit is further configured to transform video from the first camera relative to video from the second camera, the transformation based on the relationship data. The processing circuit is further configured to use the transformed video to cause the stitched video to be provided to the electronic display system.

Claims

1. A system for providing stitched video from a first camera and a second camera to an electronic display system, comprising:

a processing circuit configured to associate a view a first camera with an approximate location and to build relationship data between the first camera and a second camera using the approximate location, wherein the processing circuit is further configured to transform video from the first camera relative to video from the second camera, the transformation based on the relationship data, the processing circuit further configured to use the transformed video to cause the stitched video to be provided to the electronic display system.

2. The system of claim 1, wherein the processing circuit is configured to calculate a geometric representation of a field-of-view for the first camera and the second camera using location information and camera property information associated with each camera, wherein the first camera is a movable camera.

3. The system of claim 2, wherein the processing circuit is configured to build the relationship data by comparing the geometric representations;

wherein building the relationship data comprises updating a data structure that relates at least one of an adjacent camera pair and an adjacent field-of-view pair.

4. The system of claim 3, wherein the processing circuit determines that a field-of-view is adjacent to another field of view or that a camera is adjacent another camera by determining whether the geometric representations of the fields-of-view intersect.

5. The system of claim 4, wherein the geometric representation of the field of view is at least one of a two-dimensional shape and a three-dimensional shape;

and

wherein the processing circuit is configured to build the relationship without manual input from a user.

6. A system for providing stitched video from a plurality of cameras to an electronic display system, comprising:

a processing circuit configured to detect corners in a first image from a first camera and to detect corners in a second image from a second camera;

wherein the processing circuit is further configured to correlate a plurality of detected corner points in the first image and the second image to obtain a set of most likely pairs;

wherein the processing circuit is further configured to calculate triangles formed by corner points relating to the set of most likely pairs;

wherein the processing circuit is further configured to correlate calculated triangles in the first image with calculated triangles in the second image;

wherein the processing circuit is further configured to compute a transformation function between the correlated triangles;

wherein the processing circuit is further configured to utilize the transformation function to transform video from at least one of the first camera and the second camera to provide the stitched video to the electronic display system.

7. The system of claim 6, further comprising:

communications electronics configured to receive the first image and the second image from the first camera and the second camera; and

wherein the processing circuit comprises a rendering module that utilizes the transformation function to transform a perspective view provided by the first camera to match a perspective view provided by the second camera.

8. The system of claim 7, wherein the transformation function describes at least a scale transformation, a translation transformation, and a rotation transformation of the video from the camera.

9. The system of claim 8, wherein the processing circuit further comprises:

a corner detection module configured to detect the corners in the first image and the second image.

10. The system of claim 9, wherein the processing circuit further comprises:

a corner correlation module configured to correlate the plurality of detected corner points in the first image and the second image, wherein the corner correlation module is configured to sort correlations of corner points in the first and second images and to select at least one of a top number of corner pairs and a top percentage of corner pairs to obtain the set of most likely pairs.

11. The system of claim 10, wherein the processing circuit further comprises a camera relationship module configured to build relationship data between the plurality of cameras and the at least one camera using approximate location information and camera parameter information from each camera.

12. The system of claim 11, wherein the processing circuit utilizes the adjacency relationship data to determine which cameras of the plurality of cameras to process for corner correlations.

13. The system of claim 12, wherein the processing circuit further comprises a triangle transformation module configured to compute a transformation function between all triangles of the first image and the second image having a high correlation.

14. The system of claim 13, wherein the processing circuit further comprises an image transformation module configured to calculate an average of the computed transformation functions and to exclude computed functions deviating from the average to provide a final transformation function between the first camera and the second camera.

15. A computer program product, comprising a computer usable medium having computer readable program code embodied therein, the computer readable program code adapted to be executed to implement steps comprising:

detecting corners in a first image from a first camera;

detecting corners in a second image from a second camera;

correlating the detected corners in the first image and the second image to obtain a set of most likely pairs;

calculating triangles formed by corner points of the set of most likely pairs;

computing a transformation function between correlated triangles; and

utilizing the transformation function to transform a perspective view provided by the first camera to match a perspective view provided by the second camera.

16. The computer program product of claim 15, wherein the steps further comprise:

providing a system comprising distinct software modules;

wherein the distinct software modules comprise a corner detection module, a corner correlation module, a triangle calculation module, a triangle transformation module, an image transformation module, and a rendering module.

17. The computer program product of claim 16, wherein the corner detection module is configured to detect the corners in the first image and the second image and wherein the corner correlation module is configured to correlate a pair of points.

18. The computer program product of claim 17, wherein the triangle calculation module is configured to calculate the triangles formed by the corner points of the set of most likely pairs;

and

wherein the triangle transformation module is configured to compute a transformation function between all triangle matches between the first image and the second image having a high correlation.

19. The computer program product of claim 18, wherein the image transformation module is configured to calculate an average of the computed transformation functions and to exclude computed transformation functions deviating from the average to provide a final transformation function between the first camera and the second camera;

and

wherein the rendering module is configured to utilize the final transformation function to complete the transformation from the perspective view provided by the first camera to match the perspective view provided by the second camera.

20. The computer program product of claim 19, wherein the distinct software modules further comprise a camera relationship module configured to build adjacency relationship data between the plurality of cameras and the at least one camera using approximate location information and camera parameter information from each camera, and wherein the camera relationship module is further configured to utilize the adjacency relationship data to determine which cameras of the plurality of the cameras to process for corner correlations by the corner detection module.