Omnidirectional sensor array system

• US 10,362,225 B2
• Filed: 01/23/2018
• Issued: 07/23/2019
• Est. Priority Date: 10/29/2010
• Status: Active Grant

• Current Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Original Assignee: Ecole polytechnique fãdãrale de lausanne epfl (Schweizerische Eidgenossenschaft)
• Inventors: Bagnato, Luigi, Jacques, Laurent, Vandergheynst, Pierre, Afshari, Hossein, Schmid, Alexandre, Leblebici, Yusuf
• Primary Examiner(s): Braniff, Christopher
• Application Number:
  US15/877,781
• Publication Number:
  US 20180220070A1
• Time in Patent Office: 546 Days
• Field of Search: None
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

An omnidirectional sensor array system, for example a panoptic camera, comprising a plurality of sensors arranged on a support of predetermined shape to acquire data, wherein said sensors are directional and wherein each sensor is attached to a processing node which comprises integrated electronics that carries out at least a portion of the signal processing algorithms locally in order to reduce the computational load of a central hardware unit.

Claims

1. A sensing device comprising:

a plurality of sensors having distinct focal planes to capture light intensity values and light direction information of incoming light, each sensor comprising a plurality of sensor pixels, each sensor pixel having a sensor pixel field of view, the plurality of sensors being arranged on a support of predetermined shape to acquire light from directions observable by the plurality of sensors; and

;

an image constructor configured to generate, for each of a plurality of image pixels, an image pixel observation direction, the image pixel observation direction being a viewing direction from the image pixel for which an image pixel intensity value of the image pixel is determined;

a sensor pixel position generator configured to determine, for each image pixel observation direction, at least one sensor pixel having the image pixel observation direction in or closest to its sensor pixel field of view, and further configured to determine a corresponding sensor pixel intensity value of the at least one sensor pixel; and

an image pixel intensity generator configured to determine the image pixel intensity value for the image pixel observation direction of each image pixel based on the corresponding sensor pixel intensity value of the at least one sensor pixel.

2. The sensing device of claim 1, wherein each sensor includes a processing node configured to handle a first portion of a plurality of processing algorithms that are sensor-specific, thereby offloading a central processing unit configured to handle a second portion of a plurality of processing algorithms that are common to all sensors.

3. The sensing device of claim 2, wherein the plurality of sensors and their processing node are placed in a network or an interconnected network arrangement in order to guarantee a target data routing and throughput quality of service, and dynamic load balancing of tasks among neighboring processing nodes.

4. The sensing device of claim 1, wherein the sensors are cameras with different frame rates and triggers and are used to reconstruct images or videos at frame rates higher than an individual frame rate of any one of the plurality of sensors used.

5. The sensing device of claim 2, wherein the plurality of signal processing algorithms causes the sensing device to use light field interpolation to render a stereo pair of omnidirectional images.

6. The sensing device of claim 2, wherein the plurality of signal processing algorithms causes the sensing device to compute an omnidirectional depth map.

7. A system comprising a sensing device as defined in claim 1, a computer, and a rendering means for rendering information sensed by the sensing device.

8. A system as defined in claim 7, wherein the rendering means is a display and the information is image-related information.

9. The sensing device of claim 1, further comprising dedicated hardware to carry out omnidirectional image reconstruction based on the captured light intensity values and light direction information of the incoming light.

10. The sensing device of claim 1, wherein the plurality of sensors is arranged on the support such that neighboring sensors have overlapping fields of view.

11. The sensing device of claim 1, wherein the focal plane of each of the plurality of sensors is offset from the focal plane of every other sensor by rotation along at least two perpendicular axes of rotation.

12. The sensing device of claim 1, wherein the image constructor is configured to define an image pixel position of each image pixel relative to the support of the sensing device;

and to define for each image pixel, an image pixel observation line given by the image pixel position and the image pixel observation direction.

13. The sensing device of claim 12, wherein the sensor pixel position generator is configured to determine, for an image pixel observation point on each image pixel observation line, at least one sensor pixel having said point in or closest to its sensor pixel field of view, said image pixel observation point having a specific distance from the sensing device.

14. The sensing device of claim 12, wherein the at least one sensor pixel is in the sensor having the smallest distance from the image pixel observation line.

15. The sensing device of claim 1, wherein the image pixel intensity generator is configured to determine the image pixel intensity value, for the image pixel observation direction of each image pixel, using an interpolation of sensor pixel intensity values from a plurality of sensor pixels.

16. The sensing device of claim 1, wherein the image constructor is configured to generate image pixel observation directions to implement pan, tilt or zoom functions.

17. The sensing device of claim 1, comprising a video generator for rendering an image generated by the image constructor.

18. The sensing device of claim 1, comprising an observation direction distance generator configured to determine, for a plurality of observation directions and using a disparity algorithm on image pairs from sensor pairs with overlapping fields of view, a distance to observable surroundings of the sensing device.

19. The sensing device of claim 13, comprising two image constructors configured to generate two aligned but spatially separated images to provide stereoscopic vision.

20. The sensing device of claim 1, wherein the image constructor is further configured to construct an image comprising the plurality of image pixels.