Camera array including camera modules

• US 10,186,301 B1
• Filed: 08/31/2015
• Issued: 01/22/2019
• Est. Priority Date: 07/28/2014
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: van Hoff, Arthur, Annau, Thomas M., Christensen, Jens, Gardiner, Koji
• Primary Examiner(s): Tran, Thai Q
• Assistant Examiner(s): Smith, Stephen R
• Application Number:
  US14/841,452
• Time in Patent Office: 1,240 Days
• Field of Search: 348 36, 348 38, 348 42, 348 48, 348 51, 348515, 34821111, 3482221, 348153, 382100, 381 92, 396419, 386224, 356124
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 2013/0088 Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image
• CPC Class Codes: H04N 2013/0096 Synchronisation or controlling aspects
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

The disclosure includes a camera array comprising camera modules, the camera modules comprising a master camera that includes a processor, a memory, a sensor, a lens, a status indicator, and a switch, the switch configured to instruct each of the camera modules to initiate a start operation to start recording video data using the lens and the sensor in the other camera modules and the switch configured to instruct each of the camera modules to initiate a stop operation to stop recording, the status indicator configured to indicate a status of at least one of the camera modules. Lens distortion effects may be removed from the frames described by the video data. The camera modules of the camera array are configured to provide a 3× field of view overlap.

Claims

1. A camera system comprising:

a camera array comprising;

equator camera modules arranged around an equator of the camera array, wherein the equator camera modules capture a first set of images;

below the equator camera modules arranged below the equator of the camera array, wherein the below the equator camera modules capture a second set of images; and

above the equator camera modules arranged above the equator of the camera array, wherein the above the equator camera modules capture a third set of images;

wherein the equator camera modules, the below the equator camera modules, and the above the equator camera modules are synchronized to capture the first set of images, the second set of images, and the third set of images, respectively, at substantially a same time;

wherein the equator camera modules are arranged around the equator of the camera array so that each pixel in each image included in the first set of images is captured by at least three adjacent camera modules selected from the equator camera modules to provide at least a 3×

field of view overlap; and

an aggregation system comprising one or more processors and a non-transitory memory storing computer code which, when executed by the one or more processors causes the one or more processors to;

receive video data that describes the first set of images, the second set of images, and the third set of images;

identify a device identifier and a position of each corresponding camera module;

stitch the first set of images, the second set of images, and the third set of images together based on a relative position of each corresponding camera module to generate three-dimensional (3D) video; and

correct lens distortion that occurs in the 3D video based on a forward model by mapping a 3D real world point onto an undistorted two-dimensional (2D) world point.

2. The system of claim 1, wherein correcting the lens distortion is further based on forward and backward mapping.

3. The system of claim 1, wherein correcting the lens distortion is further based on an inverse model that is used to map a 3D real world point from an undistorted 2D image to a 3D image given a particular distance.

4. The system of claim 1, wherein the first set of images, the second set of images, and the third set of images are stitched together by identifying, for each image, the equator camera modules, the below the equator camera modules, or the above the equator camera modules that captured the image and a time when the image was captured so that, for a given time frame, at least sixteen images are identified as having been captured at substantially the same time and the at least sixteen images are stitched together in an order corresponding to arrangement of the equator camera modules around the equator of the camera array, the below the equator camera modules arranged below the equator of the camera array, and the above the equator camera modules arranged above the equator of the camera array so that stitched stereoscopic images form the three hundred and sixty degree view of an environment.

5. The system of claim 1, wherein the 3D video includes a stitched set of stereoscopic images that are synchronized for each frame based on synchronization of the equator camera modules, the below the equator camera modules, and the above the equator camera modules.

6. The system of claim 1, wherein the equator camera modules form a daisy chain with a master camera being coupled to a first camera module that is coupled to an n camera module.

7. The system of claim 1 further comprising a tray to hold the equator camera modules, the below the equator camera modules, and the above the equator camera modules from a bottom of the camera array.

8. The system of claim 1 further comprising a status indicator configured to indicate a status of at least one of the equator camera modules, the below the equator camera modules, and the above the equator camera modules.

9. The system of claim 8, wherein:

the status of one of the equator camera modules, the below the equator camera modules, and the above the equator camera modules includes a faulty status; and

the status indicator is configured to indicate the faulty status responsive to a fault occurring in one of the equator camera modules, the below the equator camera modules, and the above the equator camera modules.

10. The system of claim 8, wherein the status indicator is an overall status indicator configured to indicate a faulty status of a fault occurring in any of the equator camera modules, the below the equator camera modules, and the above the equator camera modules and wherein the equator camera modules, the below the equator camera modules, and the above the equator camera modules further include individual status indicators configured to indicate the faulty status of the fault occurring in one of the equator camera modules, the below the equator camera modules, and the above the equator camera modules.

11. The system of claim 1, wherein each of the equator camera modules includes a soundfield microphone so that the camera array includes an array of soundfield microphones and wherein the array of soundfield microphones is configured to capture audio for enabling reconstruction of sound from any arbitrary direction.

12. The system of claim 1, wherein the computer code is further configured to, responsive to confirming an absence of faults in two equator camera modules unless the two equator camera modules are next to each other, instruct the equator camera modules, the below the equator camera modules, and the above the equator camera modules to start recording the video data.

13. The system of claim 1, wherein the computer code is further configured to generate a stream of 3D video and audio data for displaying panoramic images based on corrected 3D video and wherein the stream of 3D video is captured by two or more of the equator camera modules, the below the equator camera modules, and the above the equator camera modules.

14. The system of claim 1, further comprising a viewing system configured to decode and render the 3D video and play corresponding audio data on a virtual reality display and a surround sound system.

15. The system of claim 1, wherein the equator camera modules, the below the equator camera modules, and the above the equator camera modules are synchronized through a daisy chain to capture corresponding video data in different directions simultaneously.

16. The system of claim 15, wherein the equator camera modules, the below the equator camera modules, and the above the equator camera modules are configured to pass control and status messages to one another via the daisy chain.

17. The system of claim 1, further comprising a connection hub coupled to the equator camera modules, the below the equator camera modules, and the above the equator camera modules and configured to transfer the video data from each of the equator camera modules, the below the equator camera modules, and the above the equator camera modules to a client device, wherein the connection hub includes one or more universal serial bus ports.

18. The system of claim 1, wherein one or more of the equator camera modules, the below the equator camera modules, and the above the equator camera modules further comprises an image signal processor for embedding the device identifier associated with the one or more of the equator camera modules, the below the equator camera modules, and the above the equator camera modules into the video data.

19. The system of claim 1, wherein a first equator camera module is configured to share camera settings with a remaining number of equator camera modules, the below the equator camera modules, and the above the equator camera modules.

20. The system of claim 1, wherein the equator camera modules, the below the equator camera modules, and the above the equator camera modules are configured so that the equator camera modules, the below the equator camera modules, and the above the equator camera modules are synchronized through a daisy chain to capture corresponding video data in different directions simultaneously.

21. A camera system comprising:

a camera array comprising camera modules arranged around an equator of the camera array so that each pixel of an image frame captured by one of the camera modules is captured by at least two other adjacent camera modules to provide at least a 3×

field of view overlap, wherein each of the camera modules captures image frames and the camera modules are synchronized to capture the image frames at substantially the same time; and

an aggregation system comprising one or more processors and a non-transitory memory storing computer code which, when executed by the one or more processors causes the one or more processors to;

receive video data that describes the image frames captured by the camera modules;

identify a device identifier and a position of each corresponding camera module;

stitch the image frames together based on a relative position of each corresponding camera module to generate three-dimensional (3D) video; and

correct lens distortion that occurs in the 3D video based on a forward model by mapping a 3D real world point onto an undistorted two-dimensional (2D) world point.

22. A camera system comprising:

a camera array comprising camera modules arranged around an equator of the camera array, wherein;

the camera modules are synchronized to capture a set of images of an environment where the camera array is located at substantially the same time;

the camera modules are arranged around the equator of the camera array so that each pixel in each image included in the set of images is captured by at least three adjacent camera modules selected from the camera modules to provide at least a 3×

field of view overlap;

the set of images is described by video data; and

the set of images captures substantially a three hundred and sixty degree view of the environment where the camera array is located; and

an aggregation system comprising one or more processors and a non-transitory memory storing computer code which, when executed by the one or more processors causes the one or more processors to;

receive video data that describes the set of images captured by the camera modules;

identify a device identifier and a position of each corresponding camera module;

stitch the set of images together based on a relative position of each corresponding camera module to generate three-dimensional (3D) video; and

correct lens distortion that occurs in the 3D video based on an inverse model by mapping a 3D real world point from an undistorted 2D image to a 3D image given a particular distance.

23. A camera system comprising:

a camera array comprising camera modules, wherein;

the camera modules include equator camera modules, below the equator camera modules, and above the equator camera modules;

the camera modules are synchronized to capture a set of images of an environment where the camera array is located at substantially the same time;

the camera modules are arranged in the camera array so that each pixel in each image included in the set of images is captured by at least three camera modules to provide at least a 3×

field of view overlap;

the set of images is described by video data; and

the set of images captures substantially a three hundred and sixty degree view of the environment where the camera array is located; and

an aggregation system comprising one or more processors and a non-transitory memory storing computer code which, when executed by the one or more processors causes the one or more processors to;

receive video data that describes the set of images captured by the camera modules;

identify a device identifier and a position of each corresponding camera module;

stitch the set of images together based on a relative position of each corresponding camera module to generate three-dimensional (3D) video; and

correct lens distortion that occurs in the 3D video based on a forward model by mapping a 3D real world point onto an undistorted two-dimensional (2D) world point.