Method and device for generating wide image sequences

• US 20060181610A1
• Filed: 07/08/2004
• Published: 08/17/2006
• Est. Priority Date: 07/14/2003
• Status: Active Grant

• Current Assignee: ChyronHego AB (Vector CH Holdings (Cayman) LP)
• Original Assignee: Cogeye AB
• Inventors: Carlsson, Stefan, Sullivan, Josephine, Hayman, Eric
• Granted Patent: US 7,864,215 B2
• Time in Patent Office: Days
• US Class Current: 348/159
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

The invention relates to a video recording apparatus comprising: a microprocessor (130), a memory means (120) for storing program for generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded; said memory means (120) also storing program for recording of wide image video sequences; read and write memory means (140) for storing data relating to recorded video sequences from at least two video cameras; input means (300) for input of manual input of parameters, input of recorded video sequences, output means (300) for output of a wide image video sequence. The invention also relates to a method for generating a wide image video sequence, said method comprising the steps of generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded; recording synchronously video sequences using each of said at least two video cameras, and generating a wide image video sequence from each of said synchronously recorded video sequences.

Claims

1. A method for generating a wide image video sequence, said method comprising the steps of:

a. generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded;

b. recording synchronously video sequences using each of said at least two video cameras, and

c. generating a wide image video sequence from each of said synchronously recorded video sequences.

2. A method according to claim 1 in which the synchronously recorded video sequences are stored in a memory means.

3. A method according to claim 1 in which the synchronously recorded video sequences are used concurrently for generating the wide image video sequence.

4. A method according to claim 3 in which the wide image video sequence is transmitted live.

5. A method according to claim 3 in which the wide image video sequence is stored on a memory means.

6. A method according to claim 1 in which the generation of calibration parameters comprises the following steps:

a. Start of calibration process;

b. Synchronize the sequences from each camera, which means that at least a video sequence has to be recorded by all cameras;

c. Compute inter-image projective transformations;

d. Use the transformations to refer each image to a common reference frame;

e. Choose a real or virtual reference camera such that certain lines on the pitch and/or stadium are essentially horizontal and parallel in the wide image;

f. Select a rectangular region of interest within the wide image. This region contains e.g. the entire pitch and as much of the stadium as is required or visible; and

g. Record all computed values resulting from the calibration process to be used as the calibration parameters.

7. A method according to claim 6 in which the steps of finding the lens distortion parameter(s) for each camera, and correcting radial distortion in each image produced are comprised.

8. A method according to claim 6 in which the step of selecting non-linear distortion parameters to reduce perspective distortion of the wide image is comprised.

9. Method according to claim 1 in which step b is performed manually by identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

10. Method according to claim 1 in which step b is performed automatically by an algorithm for identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

11. Method according to claim 1 which comprises the following steps:

a. Apply the computed and registered calibration parameters.

For each pixel in the wide image, compute and store parameters describing

1. Which pixels from which image(s) contributes to this pixel in the wide image.

2. How much these pixels each contribute to the wide image;

b. Repeat until the end of the sequence is reached;

c. Obtain one new image from each camera;

d. If required, update the parameters needed to transform intensities (colours/brightness) in one or more cameras to eliminate visible seams;

e. If necessary, adjust the intensities (colours/brightness) in the images from one or more cameras;

f. Create the current seamless, wide image from the current images from each camera;

g. Output the wide image to a display or to a memory means; and

h. End of sequence. Return to step b until end of generation of the wide image video sequence.

12. Method according to claim 11 wherein the new images from each camera are read from live sources, each such source comprising a video camera.

13. Method according to claim 11 wherein the new images from each video camera are read from a memory means.

22. In a device according to claim 9, the method which comprises the following steps:

a. Apply the computed and registered calibration parameters.

For each pixel in the wide image, compute and store parameters describing

1. Which pixels from which image(s) contributes to this pixel in the wide image.

2. How much these pixels each contribute to the wide image;

b. Repeat until the end of the sequence is reached;

c. Obtain one new image from each camera;

d. If required, update the parameters needed to transform intensities (colours/brightness) in one or more cameras to eliminate visible seams;

e. If necessary, adjust the intensities (colours/brightness) in the images from one or more cameras;

f. Create the current seamless, wide image from the current images from each camera;

g. Output the wide image to a display or to a memory means; and

h. End of sequence. Return to step b until end of generation of the wide image video sequence.

23. In a device according to claim 22, the method wherein the new images from each camera are read from live sources, each such source comprising a video camera.

24. In a device according to claim 22, the method wherein the new images from each video camera are read from a memory means.

14. In a device having a processor means, which executes instructions stored in at least one memory means, a method for generating video sequences comprising the steps of:

a. generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded;

b. recording synchronously video sequences using each of said at least two video cameras, and

c. generating a wide image video sequence by generating from each of said synchronously recorded video sequences.

15. In a device according to claim 14, the method in which the synchronously recorded video sequences are stored in a memory means.

16. In a device according to claim 14, the method in which the synchronously recorded video sequences are used concurrently for generating the wide image video sequence.

17. In a device according to claim 14, the method in which the generation of calibration parameters comprises the following steps:

a. Start of calibration process;

b. Synchronize the sequences from each camera, which means that at least a video sequence has to be recorded by all cameras;

c. Compute inter-image projective transformations;

d. Use the transformations to refer each image to a common reference frame;

e. Choose a real or virtual reference view such that certain lines on the pitch and/or stadium are essentially horizontal and parallel in the wide image;

f. Select a rectangular region of interest within the wide image. This region contains the entire pitch and as much of the stadium as is required or visible; and

g. Record all computed values resulting from the calibration process to be used as the calibration parameters.

18. In a device according to claim 14, the method in which the generation of calibration parameters the following steps of finding the lens distortion parameter(s) for each camera, and correcting radial distortion in each image produced are comprised.

19. In a device according to claim 14, the method in which the generation of calibration parameters the following step of selecting non-linear distortion parameters to reduce perspective distortion of the wide image is comprised.

20. In a device according to claim 14, the method in which step b is performed manually by identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

21. In a device according to claim 14, the method in which step b is performed automatically by an algorithm for identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

26. A memory means storing a program according to claim 17, in which the synchronously recorded video sequences are stored in a memory means.

27. A memory means storing a program according to claim 17, in which the synchronously recorded video sequences are used concurrently for generating the wide image video sequence.

28. A memory means storing a program according to claim 17, in which the generation of calibration parameters comprises the following steps:

a. Start of calibration process;

b. Synchronize the sequences from each camera, which means that at least a video sequence has to be recorded by all cameras;

c. Compute inter-image projective transformations;

d. Use the transformations to refer each image to a common reference frame;

e. Choose a real or virtual reference view such that certain lines on the pitch and/or stadium are essentially horizontal and parallel in the wide image;

f. Select a rectangular region of interest within the wide image. This region contains the entire pitch and as much of the stadium as is required or visible; and

g. Record all computed values resulting from the calibration process to be used as the calibration parameters.

29. A memory means storing a program according to claim 28, in which the steps of finding the lens distortion parameter(s) for each camera, and correcting radial distortion in each image produced are comprised.

30. A memory means storing a program according to claim 28, the step of selecting non-linear distortion parameters to reduce perspective distortion of the wide image is comprised.

31. A memory means storing a program according to claim 28, in which step b is performed manually by identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

32. A memory means storing a program according to claim 28, in which step b is performed automatically by and algorithm for identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer means.

33. A memory means storing a program according to claim 28, which comprises the following steps:

a. Apply the computed and registered calibration parameters.

For each pixel in the wide image, compute and store parameters describing

1. Which pixels from which image(s) contributes to this pixel in the wide image.

2. How much these pixels each contribute to the wide image;

b. Repeat until the end of the sequence is reached;

c. Obtain one new image from each camera;

d. If required, update the parameters needed to transform intensities (colours/brightness) in one or more cameras to eliminate visible seams;

e. If necessary, adjust the intensities (colours/brightness) in the images from one or more cameras;

f. Create the current seamless, wide image from the current images from each camera;

g. Output the wide image to a display or to a memory means; and

h. End of sequence. Return to step b until end of generation of the wide image video sequence.

34. A memory means according to claim 28, wherein the new images from each camera are read from live sources, each such source comprising a video camera.

35. A memory means storing a program according to claim 28, wherein the new images from each video camera are read from a memory means.

25. A computer readable memory means storing a program which executes the steps of:

a. generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded;

b. recording synchronously video sequences using each of said at least two video cameras, and

c. generating a wide image video sequence by generating from each of said synchronously recorded video sequences.

36. A video recording apparatus comprising:

a microprocessor(130), a memory means (120) for storing program for generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded;

said memory means (120) also storing program for recording of wide image video sequences;

read and write memory means (140) for storing data relating to recorded video sequences from at least two video cameras;

input means (300) for input of manual input of parameters, input of recorded video sequences, and output means (300) for output of a wide image video sequence.