Method and device for generating wide image sequences

• US 7,864,215 B2
• Filed: 07/08/2004
• Issued: 01/04/2011
• Est. Priority Date: 07/14/2003
• Status: Active Grant

• Current Assignee: ChyronHego AB (Vector CH Holdings (Cayman) LP)
• Original Assignee: Cogeye AB
• Inventors: Carlsson, Stefan, Sullivan, Josephine, Hayman, Eric
• Primary Examiner(s): Ho; Tuan
• Assistant Examiner(s): WANG, KENT F
• Application Number:
  US10/564,286
• Publication Number:
  US 20060181610A1
• Time in Patent Office: 2,371 Days
• Field of Search: 348/159, 348/36, 348/38, 348/39, 348/143, 348/153, 34820799-376, 382276-325
• US Class Current: 348/211.11
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/85 Stereo camera calibration
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

The invention relates to a video recording apparatus comprising: a microprocessor (130), a memory means (120) for storing program for generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded; said memory means (120) also storing program for recording of wide image video sequences; read and write memory means (140) for storing data relating to recorded video sequences from at least two video cameras; input means (300) for input of manual input of parameters, input of recorded video sequences, output means (300) for output of a wide image video sequence. The invention also relates to a method for generating a wide image video sequence, said method comprising the steps of generating a set of calibration parameters related to a device having at least two video cameras which are arranged in a predetermined relationship to each other, said parameters being unique for the at least two cameras and their current location as related to the object being recorded; recording synchronously video sequences using each of said at least two video cameras, and generating a wide image video sequence from each of said synchronously recorded video sequences.

Claims

1. A method for generating a wide image video sequence of a scene being recorded using a device having at least two video cameras substantially co-located in a predetermined relationship to each other such that there will be an overlap between images from the respective cameras, said method comprising the steps of:

a. identifying corresponding lines in overlapping parts of the images from the respective cameras;

b. calculating a projective transform for pixels of the images from at least one of said cameras based on slopes and positions of said corresponding lines, which said projective transform compensates for the relative rotation between the at least two video cameras;

c. synchronously recording video sequences using each of said at least two video cameras; and

d. generating a wide image video sequence by combining said synchronously recorded video sequences using said projective transform.

2. A method according to claim 1, wherein the synchronously recorded video sequences are stored in a memory storage area.

3. A method according to claim 1, wherein the wide image video sequence is transmitted live.

4. A method according to claim 1, wherein the wide image video sequence is stored on a memory storage area.

5. A method according to claim 1 further comprising the following steps:

a. starting a calibration process;

b. synchronizing the sequences from each of the at least two cameras;

c. computing inter-image projective transformations;

d. using the transformations to refer each image to a common reference frame;

e. selecting a real or virtual reference camera such that certain lines on the pitch or stadium are substantially horizontal and substantially parallel in the wide image;

f. selecting a rectangular region of interest within the wide image; and

g. storing the computed values resulting from the calibration process to be used as calibration parameters.

6. A method according to claim 5 further comprising the step of determining lens distortion parameters for each camera, and correcting radial distortion in each image.

7. A method according to claim 5 further comprising the step of selecting non-linear distortion parameters to reduce perspective distortion of the wide image.

8. A method according to claim 1 in which step c is performed automatically by an algorithm for identification of corresponding features in concurrent video images and the coordinates for the corresponding features are input via a computer means.

9. A method according to claim 1 further comprising performing the following steps until completing the generation of the wide image video sequence:

a. applying calculated calibration parameters;

b. until the end of the sequence is reached, for each pixel in the wide image, computing and storing parameters describing (1) which pixels from which images contribute to the respective pixels in the wide image and (2) how much each of the respective pixels contributes to the wide image;

c. retrieving one new image from each camera;

d. selectively updating the parameters needed to transform intensities in one or more of the cameras to eliminate visible seams;

e. selectively adjusting intensities in the images from one or more of the cameras;

f. creating the current seamless, wide image from the current images from each camera; and

g. outputting the wide image to a display or to a memory storage area.

10. A method according to claim 9, wherein the new images from each camera are read from live sources, each such source comprising a video camera.

11. A method according to claim 9 wherein the new images from each video camera are read from a memory storage area.

12. A method according to claim 1, wherein said corresponding lines depict a line naturally occurring in the scene being recorded.

13. A method according to claim 12, wherein the scene being recorded is a sports field, and wherein said line naturally occurring in the scene being recorded is a sports field mark line.

14. A method according to claim 1, wherein the step of identifying said corresponding lines is performed manually.

15. A method according to claim 1, wherein the step of identifying said corresponding lines is performed automatically.

16. A device having at least two video cameras substantially co-located and arranged in a predetermined relationship to each other such that there will be an overlap between images from the respective cameras, a processor, at least one memory storage area, wherein the processor is configured to:

a. identify corresponding lines in overlapping parts of the images from the respective cameras;

b. calculate a projective transform for pixels of the images from at least one of said cameras based on slopes and positions of said corresponding lines, which said projective transform compensates for the relative rotation between the at least two video cameras;

c. synchronously record video sequences using each of said at least two video cameras; and

d. generate a wide image video sequence of a scene being recorded by combining said synchronously recorded video sequences using said projective transform.

17. The device of claim 16, wherein the synchronously recorded video sequences are stored in a memory storage area.

18. The device of claim 16, wherein to calculate calibration parameters, the processor is further configured to:

a. start a calibration process;

b. synchronize the sequences from each of the at least two cameras;

c. compute inter-image projective transformations;

d. use the transformations to refer each image to a common reference frame;

e. select a real or virtual reference camera such that certain lines on the pitch or stadium are substantially horizontal and substantially parallel in the wide image;

f. select a rectangular region of interest within the wide image; and

g. store the computed values resulting from the calibration process to be used as the calibration parameters.

19. The device of claim 18, wherein to calculate the calibration parameters, the processor is further configured to determine lens distortion parameters for each camera, and correct radial distortion in each image produced.

20. The device of claim 18, wherein to calculate the calibration parameters, the processor is further configured to select non-linear distortion parameters to reduce perspective distortion of the wide image.

21. The device of claim 16, wherein step c is performed automatically by an algorithm for identification of corresponding features in concurrent video images and the coordinates for these corresponding features are input to a computer storage area.

22. The device of claim 16, wherein the processor is further configured to:

a. apply calculated calibration parameters;

b. until the end of the sequence is reached, for each pixel in the wide image, compute and store parameters describing (1) which pixels from which images contribute to the respective pixels in the wide image and (2) how much each of the respective pixels contributes to the wide image;

c. retrieve one new image from each camera;

d. selectively update the parameters needed to transform intensities in one or more of the cameras to eliminate visible seams;

e. selectively adjust intensities in the images from one or more of the cameras;

f. create the current seamless, wide image from the current images from each camera; and

g. output the wide image to a display or to a memory storage area.

23. The device of claim 22, wherein the new images from each camera are read from live sources, each such source comprising a video camera.

24. The device of claim 22, wherein the new images from each video camera are read from a memory storage area.

25. The device of claim 16, wherein said corresponding lines depict a line naturally occurring in the scene being recorded.

26. The device of claim 16, wherein the scene being recorded is a sports field, and wherein said projective transform refers each image to a common reference frame, and the processor is further configured to select said reference frame such that certain lines on the sports field become essentially horizontal and parallel in the wide image video sequence.

27. A computer-readable medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising executable portions that cause a device having at least two video cameras substantially co-located in a predetermined relationship to each other such that there will be an overlap between images from the respective cameras to:

a. identify corresponding lines in overlapping parts of the images from the respective cameras;

b. calculate a projective transform for pixels of the images from at least one of said cameras based on slopes and positions of said corresponding lines, which said projective transform compensates for the relative rotation between the at least two video cameras;

c. synchronously record video sequences using each of said at least two video cameras; and

d. generate a wide image video sequence of a scene being recorded by combining said synchronously recorded video sequences using said projective transform.

28. The computer-readable medium of claim 27, wherein the synchronously recorded video sequences are stored in a memory storage area.

29. The computer-readable medium of claim 27, wherein to calculate calibration parameters, the computer-readable medium further comprises executable portions that cause the device to:

a. start a calibration process;

b. synchronize the sequences from each of the at least two cameras;

c. compute inter-image projective transformations;

d. use the transformations to refer each image to a common reference frame;

e. select a real or virtual reference camera such that certain lines on the pitch or stadium are substantially horizontal and substantially parallel in the wide image;

f. select a rectangular region of interest within the wide image; and

g. store the computed values resulting from the calibration process to be used as the calibration parameters.

30. The computer-readable medium of claim 29, further comprising executable portions that cause the device to determine lens distortion parameters for each camera, and correct radial distortion in each image produced.

31. The computer-readable medium of claim 29, further comprising executable portions that cause the device to select non-linear distortion parameters to reduce perspective distortion of the wide image.

32. The computer-readable medium of claim 29, wherein step c is performed automatically by an algorithm for identification of corresponding features in concurrent video images and the coordinates for the corresponding features are input to a computer means.

33. The computer-readable medium of claim 29, further comprising executable portions that causes the device to:

a. apply the calculated calibration parameters;

b. until the end of the sequence is reached, for each pixel in the wide image, compute and store parameters describing (1) which pixels from which images contribute to the respective pixels in the wide image and (2) how much each of the respective pixels contributes to the wide image;

c. retrieve one new image from each camera;

d. selectively update the parameters needed to transform intensities in one or more of the cameras to eliminate visible seams;

e. selectively adjust intensities in the images from one or more of the cameras;

f. create the current seamless, wide image from the current images from each camera; and

g. output the wide image to a display or to a memory storage area.

34. The computer-readable medium of claim 29, wherein the new images from each camera are read from live sources, each such source comprising a video camera.

35. The computer-readable medium of claim 29, wherein the new images from each video camera are read from a memory storage area.

36. The computer-readable medium of claim 27, wherein said corresponding lines depict a line naturally occurring in the scene being recorded.

37. The computer-readable medium of claim 27, wherein the scene being recorded is a sports field, and wherein said projective transform refers each image to a common reference frame, and the computer-readable medium further comprises executable portions that cause the device to select said reference frame such that certain lines on the sports field become essentially horizontal and parallel in the wide image video sequence.

38. A video recording apparatus having at least two video cameras substantially co-located in a predetermined relationship to each other such that there will be an overlap between images from the respective cameras, the video recording apparatus comprising:

a microprocessor (130);

a memory storage area (120) storing a program for;

a. identifying corresponding lines in overlapping parts of the images from the respective cameras,

b. calculating a projective transform for pixels of the images from at least one of said cameras based on slopes and positions of said corresponding lines, which said projective transform compensates for the relative rotation between the at least two video cameras,

c. synchronously recording video sequences using each of said at least two video cameras, and

d. generating a wide image video sequence of a scene being recorded by combining said synchronously recorded video sequences using said projective transform,

a read and write memory storage area (140) for storing data relating to recorded video sequences from the at least two video cameras;

an input component (300) for receiving input of parameters, and input of recorded video sequences; and

an output component (300) for outputting of a wide image video sequence.

39. The apparatus of claim 38, wherein said corresponding lines depict a line naturally occurring in the scene being recorded.

40. The apparatus of claim 38, wherein the scene being recorded is a sports field, and wherein said projective transform refers each image to a common reference frame, and the program is further for selecting said reference frame such that certain lines on the sports field become essentially horizontal and parallel in the wide image video sequence.