Camera array removing lens distortion

• US 10,368,011 B2
• Filed: 07/28/2015
• Issued: 07/30/2019
• Est. Priority Date: 07/25/2014
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: Annau, Thomas M., van Hoff, Arthur, Christensen, Jens
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Smith, Stephen R
• Application Number:
  US14/810,479
• Publication Number:
  US 20160191815A1
• Time in Patent Office: 1,463 Days
• Field of Search: 348 36, 348 38, 348 42, 348 48, 348 51, 348515, 34821111, 3482221, 348153, 382100, 381 92, 396419, 386224
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20228 Disparity calculation for image-based rendering
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The disclosure includes a camera array comprising camera modules, the camera modules comprising a master camera that includes a processor, a memory, a sensor, a lens, a status indicator, and a switch, the switch configured to instruct each of the camera modules to initiate a start operation to start recording video data using the lens and the sensor in the other camera modules and the switch configured to instruct each of the camera modules to initiate a stop operation to stop recording, the status indicator configured to indicate a status of at least one of the camera modules. Lens distortion effects may be removed from the frames described by the video data. The camera modules of the camera array are configured to provide a 3× field of view overlap.

Claims

1. A camera system comprising:

a camera array comprising camera modules, the camera modules comprising a master camera that includes a processor, a memory, a sensor, a lens and a switch, the switch configured to instruct each of the camera modules to initiate a start operation to start recording video data using the lens and the sensor in each of the camera modules and the switch configured to instruct the camera modules to initiate a stop operation to stop recording; and

an aggregation system communicatively coupled to the camera array to receive the video data from the camera array, the aggregation system including code and routines stored on a non-transitory memory and configured to provide lens distortion correction for the video data when executed by the processor based on a forward model to map a three-dimensional real-world point onto an undistorted two-dimensional image by capturing the three-dimensional real-world point (V), determining a length of V by calculating a Euclidean norm, determining a normalized three-dimensional point (V_N) by normalizing V, and determining an angle of V_N;

wherein the camera modules of the camera array are configured to provide at least a 3×

field of view overlap; and

wherein the camera array comprises equator camera modules that are arranged around an equator of the camera array so that the equator camera modules are adjacent to one another and configured to capture images having a same orientation relative to one another, wherein the same orientation is a portrait orientation or a landscape orientation of an optical sensor.

2. The system of claim 1, wherein the camera array comprises twenty four camera modules and the equator camera modules includes at least sixteen camera modules.

3. The system of claim 2, wherein:

the equator camera modules are configured to capture images having the portrait orientation;

four of the twenty four camera modules are positioned above the equator of the camera array and are configured to capture images having the landscape orientation; and

four of the twenty four camera modules are positioned below the equator of the camera array and are configured to capture images having the landscape orientation.

4. The system of claim 3, wherein:

the four camera modules placed above the equator of the camera array are positioned to have an upward pitch of substantially positive sixty-five degrees relative to the sixteen camera modules positioned around the equator of the camera array; and

the four camera modules placed below the equator of the camera array are positioned to have a downward pitch of substantially negative sixty-five degrees relative to the sixteen camera modules positioned around the equator of the camera array.

5. The system of claim 1, wherein the camera modules form a daisy chain with the master camera being coupled to a first camera module that is coupled to an n camera module that is coupled to the master camera.

6. The system of claim 1, wherein the lens distortion correction corrects one or more of a barrel effect or a pincushion effect.

7. The system of claim 1, wherein the camera system includes an overall status indicator configured to indicate a faulty status of a fault occurring in any of the camera modules and wherein the camera modules further include individual status indicators configured to indicate the faulty status of the fault occurring in one of the camera modules.

8. The system of claim 1, wherein each camera module includes a soundfield microphone so that the camera array includes an array of soundfield microphones and wherein the array of soundfield microphones is configured to capture audio for enabling reconstruction of sound from any arbitrary direction.

9. The system of claim 1, wherein the aggregation system is further configured to generate a stream of three-dimensional video and audio data for displaying panoramic images.

10. The system of claim 9, further comprising a viewing system configured to decode and render the three-dimensional video and play the audio data on a virtual reality display and a surround sound system.

11. The system of claim 1, wherein the camera modules are synchronized through a daisy chain to capture corresponding video data in different directions simultaneously.

12. The system of claim 11, wherein the camera modules pass control and status messages to one another via the daisy chain.

13. The system of claim 1, further comprising a connection hub coupled to the camera modules and configured to transfer the video data from each of the camera modules to a client device.

14. The system of claim 13, wherein the connection hub includes one or more universal serial bus (USB) 2.0 ports or one or more USB 3.0 ports and the system further comprises an expansion port including hardware configured to enable one or more of the following to be communicatively coupled to provide one or more inputs to the system:

a wired remote;

a wireless dongle; and

a touch panel display.

15. The system of claim 1, wherein the camera modules each further comprise an image signal processor for embedding a device identifier associated with a corresponding camera module into the video data.

16. The system of claim 1, wherein each camera module includes a heat sink.

17. The system of claim 1, wherein the camera array includes one or more 3-axis accelerometers configure to detect the orientation of the camera modules.

18. The camera system of claim 1, wherein each pixel recorded by the camera array is recorded by three different camera modules of the equator camera modules.

19. A computer-implemented method comprising:

identifying, with one or more processors, a device identifier and a position of each camera module in a camera array, the camera modules including a master camera and the camera modules configured to provide at least a 3×

field of view overlap wherein the camera array comprises at least sixteen camera modules that are arranged around an equator of the camera array so that the at least sixteen camera modules are adjacent to one another and configured to capture images having a same orientation relative to one another, wherein the same orientation is a portrait orientation or a landscape orientation of an optical sensor;

confirming an absence of faults in the camera module;

initiating a start operation in the master camera, the master camera instructing the other camera modules to start recording;

receiving video data comprising image frames from the camera modules;

removing one or more lens distortion effects present in the image frames based on a forward model to map a three-dimensional real-world point onto an undistorted two-dimensional image by determining an angle of a normalized three-dimensional point, determining an undistorted distance to an optical center of a lens of one of the camera modules in the camera array, and determining (x, y) coordinates of a two-dimensional point of the undistorted two-dimensional image;

stitching the image frames together based on the video data;

generating three-dimensional video;

synthesizing audio data; and

generating a stream of the three-dimensional video and the audio data for displaying panoramic images.

20. The method of claim 19, wherein the image frames are stitched together based on a relative position of each camera module and each pixel of the image has three candidate sets of video data to select from when stitching the image frames based on the at least 3×

field of view overlap configuration of the camera modules.

21. A special purpose computing device comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on the special purpose computing device causes the special purpose computing device to:

identify a device identifier and a position of each camera module in a camera array, the camera modules including a master camera and the camera modules configured to provide at least a 3×

field of view overlap of the camera modules wherein the camera array comprises at least sixteen camera modules that are arranged around an equator of the camera array so that the at least sixteen camera modules are adjacent to one another and configured to capture images having a same orientation relative to one another, wherein the same orientation is a portrait orientation or a landscape orientation of an optical sensor;

confirm an absence of faults in the camera modules;

initiate a start operation in the master camera, the master camera instructing the other camera modules to start recording;

receive video data comprising image frames from the camera modules;

remove one or more lens distortion effects from the image frames based on a forward model to map a three-dimensional real-world point onto an undistorted two-dimensional image by determining a length of the three-dimensional real-world point, determining a normalized three-dimensional point, determining an angle of the normalized three-dimensional point, determining an undistorted distance to an optical center of a lens of one of the camera modules in the camera array, and determining (x, y) coordinates of a two-dimensional point of the undistorted two-dimensional image;

stitch the image frames together based on the video data, wherein the image frames are stitched together based on a relative position of each camera module and each pixel of the image has three candidate sets of video data to select from when stitching the image frames based on the at least 3×

field of view overlap;

generate three-dimensional video;

synthesize audio data; and

generate a stream of the three-dimensional video and the audio data for displaying panoramic images.