Camera array including camera modules

• US 9,911,454 B2
• Filed: 05/29/2015
• Issued: 03/06/2018
• Est. Priority Date: 05/29/2014
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Jaunt Inc. (Verizon Communications Inc.)
• Inventors: van Hoff, Arthur, Annau, Thomas M., Christensen, Jens, Gardiner, Koji
• Primary Examiner(s): Adams, Eileen
• Application Number:
  US14/726,118
• Publication Number:
  US 20150348580A1
• Time in Patent Office: 1,012 Days
• Field of Search: None
• CPC Class Codes: G06Q 30/0263 based upon Internet or website rating
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G11B 19/20 Driving; Starting; Stopping; Control thereof
• CPC Class Codes: H04N 13/106 Processing image signals for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

The disclosure includes a camera array comprising camera modules, the camera modules comprising a master camera that includes a processor, a memory, a sensor, a lens, a status indicator, and a switch, the switch configured to instruct each of the camera modules to initiate a start operation to start recording video data using the lens and the sensor in the other camera modules and the switch configured to instruct each of the camera modules to initiate a stop operation to stop recording, the status indicator configured to indicate a status of at least one of the camera modules. The camera modules of the camera array are configured to provide a 3× field of view overlap.

Claims

1. A camera array system comprising:

a plurality of camera modules, the plurality of camera modules comprising a master camera module and a plurality of slave camera modules, such that the master camera module instructs initiation of an operation in the plurality of slave camera modules after initiation of the operation in the master camera module, the master camera module further comprising;

a processor;

a memory;

a sensor;

a lens;

a status indicator, configured to indicate a status of at least one of the plurality of camera modules or the camera array system; and

a switch, configured to instruct each of the plurality of camera modules to initiate a start operation to start recording video data using the lens and the sensor in each of the plurality of camera modules and also to instruct the plurality of camera modules to initiate a stop operation to stop recording;

wherein the plurality of camera modules are configured to provide a 3×

field of view overlap.

2. The system of claim 1, wherein the plurality of camera modules include twenty four camera modules.

3. The system of claim 2, wherein:

sixteen of the twenty four camera modules are arranged around an equator of the camera array and configured to capture images having portrait orientation;

four of the twenty four camera modules positioned above the equator of the camera array and configured to capture images having landscape orientation; and

four of the twenty four camera modules are positioned below the equator of the camera array and configured to capture images having landscape orientation.

4. The system of claim 3, wherein:

the four camera modules placed above the equator of the camera array are positioned to have an upward pitch of substantially positive sixty-five degrees relative to the camera modules positioned around the equator of the camera array; and

the four camera modules placed below the equator of the camera array are positioned to have a downward pitch of substantially negative sixty-five degrees relative to the camera modules positioned around the equator of the camera array.

5. The system of claim 1, wherein the plurality of camera modules form a daisy chain with the master camera being coupled to a first camera module that is coupled to an n camera module that is coupled to the master camera.

6. The system of claim 1, wherein:

the status of one of the plurality of camera module includes a faulty status; and

the status indicator is configured to indicate the faulty status responsive to a fault occurring in one of the plurality of camera modules.

7. The system of claim 1, wherein the status indicator is an overall status indicator configured to indicate a faulty status of a fault occurring in any of the plurality of camera modules and wherein the camera modules further include individual status indicators configured to indicate the faulty status of the fault occurring in any of the plurality of camera modules.

8. The system of claim 1, wherein each camera module of the plurality of camera modules includes a soundfield microphone so that the camera array includes an array of soundfield microphones and wherein the array of soundfield microphones is configured to capture audio for enabling reconstruction of sound from any arbitrary direction.

9. The system of claim 1, further comprising an aggregation system for generating a stream of three-dimensional video and audio data for displaying panoramic images.

10. The system of claim 9, further comprising a viewing system configured to decode and render the three-dimensional video and play the audio data on a virtual reality display and a surround sound system.

11. The system of claim 1, wherein the plurality of camera modules are synchronized through a daisy chain to capture corresponding video data in different directions simultaneously.

12. The system of claim 11, wherein the plurality of camera modules pass control and status messages to one another via the daisy chain.

13. The system of claim 1, further comprising a connection hub coupled to the camera modules and configured to transfer the video data from each of the plurality of camera modules to a client device.

14. The system of claim 13, wherein the connection hub includes one or more universal serial bus (USB) 2.0 ports or one or more USB 3.0 ports and the system further comprises an expansion port including hardware configured to enable one or more of the following to be communicatively coupled to provide one or more inputs to the system:

a wired remote;

a wireless dongle; and

a touch panel display.

15. The system of claim 1, wherein a camera module of the plurality of camera modules further comprises an image signal processor for embedding a device identifier associated with the camera module into the video data.

16. The system of claim 1, wherein each camera module of the plurality of camera modules include a heat dissipation element.

17. The system of claim 1, wherein the camera array includes one or more 3-axis accelerometers configure to detect an orientation of one or more of the plurality of camera modules.

18. A computer-implemented method comprising:

identifying, with one or more processors, a device identifier and a position of each of a plurality of camera modules in a camera array, the camera array including a master camera module and a plurality of slave camera modules, and the plurality of camera modules configured to provide a 3×

field of view overlap;

confirming an absence of faults in the plurality of camera modules;

initiating a start operation in the master camera module, the master camera module instructing the plurality of slave camera modules to start recording;

receiving video data comprising image frames from the plurality of camera modules;

stitching the image frames together based on the video data;

generating three-dimensional video;

synthesizing audio data; and

generating a stream of the three-dimensional video and the audio data for displaying panoramic images.

19. The method of claim 18, wherein the image frames are stitched together based on a relative position of each camera module of the plurality of camera modules and each pixel of an image has three candidate sets of video data to select from when stitching the image frames based on the 3×

field of view overlap configuration of the camera modules.

20. A special purpose computing device comprising a non-transitory computer-usable medium including a computer-readable program, wherein the computer-readable program when executed on the special purpose computing device causes the special purpose computing device to:

identify a device identifier and a position of each camera module of camera modules in a camera array, wherein one of the camera modules includes a master camera and the camera modules are configured to provide a 3×

field of view overlap of the camera modules;

confirm an absence of faults in the camera modules;

initiate a start operation in the master camera, the master camera instructing other camera modules to start recording;

receive video data comprising image frames from the camera modules;

stitch the image frames together based on the video data, wherein the image frames are stitched together based on a relative position of each camera module and each pixel of an image has three candidate sets of video data to select from when stitching the image frames based on the 3×

field of view overlap;

generate three-dimensional video;

synthesize audio data; and

generate a stream of the three-dimensional video and the audio data for displaying panoramic images.

21. A camera system comprising:

a camera array comprising sixteen camera modules arranged around an equator of the camera array such that the sixteen camera modules include a master camera module and a plurality of slave camera modules, wherein the master camera module instructs initiation of an operation in the slave camera modules after initiation of the operation in the master camera module;

wherein the sixteen camera modules are synchronized to capture a set of stereoscopic images of an environment at substantially the same time;

wherein the set of stereoscopic images is described by image data;

wherein the set of stereoscopic images captures a three hundred and sixty degree view of the environment where the camera array is located;

wherein the image data is configured so that, for each stereoscopic image, a camera module from the sixteen camera modules that captured the stereoscopic image is identifiable; and

wherein the image data includes, for each stereoscopic image, a timestamp indicating a time when the stereoscopic image was captured.

22. The system of claim 21, further comprising an aggregation system configured to receive the image data and generate three-dimensional video based on the image data.

23. The system of claim 22, wherein the aggregation system generates the three-dimensional video by stitching the set of stereoscopic images together to generate three-dimensional video that depicts the environment.

24. The system of claim 23, wherein the set of stereoscopic images are stitched together by identifying, for each stereoscopic image, corresponding camera modules which captured the stereoscopic image and the time when the stereoscopic image was captured so that, for a given time frame, at least sixteen stereoscopic images are identified as having been captured at substantially the same time and the at least sixteen stereoscopic images are stitched together in an order corresponding to arrangement of the corresponding camera modules around the equator of the camera array so that the stitched stereoscopic images form the three hundred and sixty degree view of the environment.