Omnidirectional sensor array system

• US 9,876,953 B2
• Filed: 10/31/2011
• Issued: 01/23/2018
• Est. Priority Date: 10/29/2010
• Status: Expired due to Fees

• Current Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Original Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Inventors: Bagnato, Luigi, Jacques, Laurent, Vandergheynst, Pierre, Afshari, Hossein, Schmid, Alexandre, Leblebici, Yusuf
• Primary Examiner(s): Tran, Thai
• Assistant Examiner(s): Braniff, Christopher T
• Application Number:
  US13/881,463
• Publication Number:
  US 20140146132A1
• Time in Patent Office: 2,276 Days
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An omnidirectional sensor array system, for example a panoptic camera, comprising a plurality of sensors arranged on a support of predetermined shape to acquire data, wherein said sensors are directional and wherein each sensor is attached to a processing node which comprises integrated electronics that carries out at least a portion of the signal processing algorithms locally in order to reduce the computational load of a central hardware unit.

Claims

1. A sensing device comprising:

a plurality of sensors to capture light intensity values and light direction information of incoming light, each sensor comprising a plurality of sensor pixels, each sensor pixel having a sensor pixel field of view;

the plurality of sensors being arranged on a support of predetermined shape to acquire light from directions observable by the plurality of sensors;

an image constructor configured to generate an image comprising a plurality of image pixels, and configured to generate an image pixel observation direction for each image pixel, the image pixel observation direction being a viewing direction from the image pixel for which an image pixel intensity value of the image pixel is determined;

the image constructor being further configured to receive image pixel intensity values form and image pixel intensity generator and to construct said image using the received image pixel intensity values;

a sensor pixel position generator configured to determine, for each image pixel observation direction, at least one sensor having said image pixel observation direction in or closet to its sensor pixel field of view; and

an image pixel intensity generator configured to determine, for each image pixel observation direction, an image pixel intensity value based on the at least one sensor pixel intensity value of said at least one sensor pixel.

2. The sensing device as defined in claim 1, wherein the plurality of sensors and their processing node are placed in a network or an interconnected network arrangement in order to guarantee a target data routing and throughput quality of service, and dynamic load balancing of tasks among neighboring processing nodes.

3. The sensing device as defined in claim 1, wherein the sensors are cameras with different frame rates and triggers and are used to reconstruct images or videos at frame rates higher than an individual frame rate of any one of the plurality of sensors used.

4. The sensing device as defined in claim 1, wherein the plurality of signal processing algorithms causes the sensing device to use light field interpolation to render a stereo pair of omnidirectional images.

5. The sensing device as defined in claim 1, wherein the plurality of signal processing algorithms causes the sensing device to compute an omnidirectional depth map.

6. A system comprising a sensing device as defined in claim 1, a computer, and a rendering means for rendering information sensed by the sensing device.

7. A system as defined in claim 1, wherein the rendering means is a display and the information is image related information.

8. The sensing device according to claim 1, wherein the image reconstruction is omnidirectional image reconstruction and the device further comprises dedicated hardware to carry out the omnidirectional image reconstruction based on the captured light intensity values and light direction information of the incoming light.

9. The omnidirectional sensing device according to claim 1, wherein each sensor of said plurality of sensors has a different focal plane compared to the focal plane of each other sensor of said plurality of sensors.

10. The omnidirectional sensing device according to claim 1, wherein the plurality of sensors is arranged on the support such that neighboring sensors have overlapping fields of view.

11. The omnidirectional sensing device according to claim 9, wherein the focal plane of each of the plurality of sensors is offset from the focal plane of every other sensor by rotation along at least two perpendicular axes of rotation.

12. The sensing device as defined in claim 1, wherein the image constructor is configured to define an image pixel position of each image pixel relative to the support of the sensing device;

and to define for each image pixel, an image pixel observation line given by the image pixel position and the image pixel observation direction.

13. The sensing device as defined in claim 12, wherein the sensor pixel position generator is configured to determine, for an image pixel observation point on each image pixel observation line, at least one sensor pixel having said point in or closest to its sensor pixel field of view, said image pixel observation point having a specific distance from the sensing device.

14. The sensing device as defined in claim 12, wherein the image pixel intensity generator is configured to determine said image pixel intensity value, for each image pixel, using the sensor pixel intensity value of the sensor pixel in the sensor having the smallest distance from the image pixel observation line.

15. The sensing device as defined in claim 1, wherein the image pixel intensity generator is configured to determine said image pixel intensity value, for each image pixel, using an interpolation of sensor pixel intensity values from a plurality of sensor pixels.

16. The sensing device as defined in claim 1, wherein the image constructor is configured to generate image pixel observation directions to implement pan, tilt or zoom functions.

17. The sensing device as defined in claim 1, comprising a video generator for rendering the image generated by the image constructor.

18. The sensing device as defined in claim 1, comprising an observation direction distance generator configured to determine, for a plurality of observation directions, using a disparity algorithm on image pairs from sensor pairs with overlapping fields of view, a distance to observable surroundings of the sensing device.

19. The sensing device as defined in claim 13, comprising two image constructors configured to generate two aligned but spatially separated images to provide stereoscopic vision.

20. The sensing device as defined in claim 1, where each sensor includes a processing node configured to handle sensor specific processing tasks, thereby offloading a central processing unit configured to handle processing tasks common to all sensors.