Method and device for hybrid robotic/virtual pan-tilt-zoom cameras for autonomous event recording

• US 10,165,157 B2
• Filed: 09/30/2013
• Issued: 12/25/2018
• Est. Priority Date: 02/19/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Carr, G. Peter K., Mistry, Michael Peter K., Matthews, Iain, Sheikh, Yaser
• Primary Examiner(s): Navas, Jr., Edemio
• Application Number:
  US14/042,265
• Publication Number:
  US 20140232818A1
• Time in Patent Office: 1,912 Days
• Field of Search: None
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications

Abstract

A system and method generates a broadcast image. The method includes receiving a first image captured by a first camera having a first configuration incorporating a first center of projection. The method includes determining a first mapping from a first image plane of the first camera onto a sphere. The method includes determining a second mapping from the sphere to a second image plane of a second virtual camera having a second configuration incorporating the first center of projection of the first configuration. The method includes generating a second image based upon the first image and a concatenation of the first and second mappings.

Claims

1. A method, comprising:

receiving a first image captured by a first camera having a first configuration including a first optical axis;

determining a first mapping from a first image plane of the first camera onto a sphere, wherein the sphere is centered at the first camera;

determining a second mapping from the sphere to a second image plane of a second virtual camera having a second configuration; and

generating a second image based on a resampling of the first image, the second mapping from the sphere to the second image plane and applying a three-dimensional rotation to the first optical axis to generate a second optical axis that corresponds to the second image plane.

2. The method of claim 1, wherein the first camera remains stationary, and the first configuration is fixed.

3. The method of claim 1, wherein the first camera is a Pan-Tilt-Zoom (PTZ) camera, and the first configuration is determined by an internal state of the PTZ camera.

4. The method of claim 3, wherein the PTZ camera is steered by a control loop based on a discrepancy between a desired first configuration and an actual first configuration.

5. The method of claim 4, wherein at least one of the desired first configuration and a desired second configuration is determined by an analysis of sensor data.

6. The method of claim 5, wherein the at least one of the desired first configuration and the desired second configuration is determined by a human operator.

7. The method of claim 1, wherein the determination of the second configuration is delayed relative to the determination of the first configuration.

8. The method of claim 1, wherein the second configuration is restricted from deviating from the first configuration beyond a predetermined value.

9. The method of claim 4, wherein the control loop is based on a discrepancy between a desired first configuration, an actual first configuration, a desired second configuration, and an actual second configuration.

10. The method of claim 9, wherein the desired second configuration and actual second configuration are delayed relative to the desired first configuration and actual first configuration.

11. A device, comprising:

a processor coupled to a memory, wherein the processor is programmed to generate a broadcast image by;

receiving a first image captured by a first camera having a first configuration including a first optical axis;

determining a first mapping from a first image plane of the first camera onto a sphere, wherein the sphere is centered at the first camera;

determining a second mapping from the sphere to a second image plane of a second virtual camera having a second configuration; and

generating a second image based on a resampling of the first image, the second mapping from the sphere to the second image plane and applying a three-dimensional rotation to the first optical axis to generate a second optical axis that corresponds to the second image plane.

12. The device of claim 11, wherein the first camera remains stationary, and the first configuration is fixed.

13. The device of claim 11, wherein the first camera is a Pan-Tilt-Zoom (PTZ) camera, and the first configuration is determined by an internal state of the PTZ camera.

14. The device of claim 13, wherein the PTZ camera is steered by a control loop based on a discrepancy between a desired first configuration and an actual first configuration.

15. The device of claim 14, wherein at least one of the desired first configuration and a desired second configuration is determined by an analysis of sensor data.

16. The device of claim 15, wherein the at least one of the desired first configuration and the desired second configuration is determined by a human operator.

17. The device of claim 11, wherein the determination of the second configuration is delayed relative to the determination of the first configuration.

18. The device of claim 11, wherein the second configuration is restricted from deviating from the first configuration beyond a predetermined value.

19. The device of claim 14, wherein the control loop is based on a discrepancy between a desired first configuration, an actual first configuration, a desired second configuration, and an actual second configuration.

20. A non-transitory computer readable storage medium including a set of instructions executable by a processor, the set of instructions operable to:

receive a first image captured by a first camera having a first configuration including a first optical axis;

determine a first mapping from a first image plane of the first camera onto a sphere, wherein the sphere is centered at the first camera;

determine a second mapping from the sphere to a second image plane of a second virtual camera having a second configuration; and

generate a second image based on a resampling of the first image, the second mapping from the sphere to the second image plane and applying a three-dimensional rotation to the first optical axis to generate a second optical axis that corresponds to the second image plane.