Panoramic camera having multiple sensor systems and methods of collecting data of an environment using the same

• US 10,681,329 B2
• Filed: 05/01/2017
• Issued: 06/09/2020
• Est. Priority Date: 05/06/2016
• Status: Active Grant

• Current Assignee: NCTech, Ltd.
• Original Assignee: NCTech, Ltd.
• Inventors: Tocher, Neil, Ure, Cameron, Ahmadian, Mansour
• Primary Examiner(s): Li, Tracy Y.
• Application Number:
  US15/583,393
• Publication Number:
  US 20170324942A1
• Time in Patent Office: 1,135 Days
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

A camera and associated method of operation, the camera comprising a plurality of sensor systems, each sensor system comprising at least one spatial sensor and at least one image sensor, wherein at least part of a field of view of one or more or each of the sensor systems differs to at least part of the field of view of at least one or each other of the sensor systems.

Claims

1. A panoramic camera comprising:

a plurality of sensor systems, each sensor system comprising at least one spatial sensor and at least one image sensor, and

a mount for mounting to a support for supporting the camera such that at least part of the camera is rotatable relative to the mount or support on which the camera is mounted around a rotational axis;

wherein;

at least part of a field of view of each of the sensor systems differs to at least part of the field of view of each other of the sensor systems;

the sensor systems are positioned such that at least part of the fields of view of each of the plurality of sensor systems, that respectively comprise fields of view of the at least one spatial sensor and the at least one imaging sensor of the respective sensor system, are distributed over a vertical direction, in use, such that a total field of view of all of the sensor systems extends above the camera; and

the image sensors of the sensor systems are positioned along or on the rotational axis or a surface or axis substantially parallel, co-linear or co-directional to the rotational axis.

2. The camera of claim 1, wherein at least one or each of the sensor systems is oriented differently to at least one or each other of the sensor systems.

3. The camera of claim 1, wherein the spatial sensor comprises at least one of a depth, distance, angle, or range sensor configured to determine at least one of distances or angles between the spatial sensor and locations on surfaces or objects within the field of view of the sensor and/or in the environment to be imaged or modelled.

4. The camera of claim 1, wherein the spatial sensor comprises a stereo sensor.

5. The camera according to claim 1, wherein at least one of the spatial sensors comprises at least two receivers and at least one transmitter.

6. The camera according to claim 1, wherein at least one or each spatial sensor comprises an infra-red sensor and/or the at least one image sensor is configured to take visual and/or optical images.

7. The camera according to claim 1, comprising at least three sensor systems.

8. The camera according to claim 1, wherein part of the field of view of at least one or more or each sensor system overlaps part, but not all, of the field of one or more adjacent sensor system.

9. The camera according to claim 1, wherein at least one or each sensor system is an integral or unitary sensor system in which the spatial sensor(s) and the imaging sensor(s) for a given sensor system are combined together into a single integral unit.

10. The camera according to claim 1, wherein the sensor systems or the respective optical or central axes of the field of view of at least two or more or each of the sensor systems are distributed or spaced apart over at least one direction or over at least one axis or surface.

11. The camera according to claim 10, wherein the surface is a curved axis or surface or a convex surface.

12. The camera according to claim 1, wherein the fields of view or optical axes of respective sensor systems are mutually divergent.

13. The camera according to claim 1, wherein the camera is operable in at least one of a mounted or a rotation mode in which the camera is configured to collect images and/or spatial data whilst the camera is mounted on the support, wherein, in the mounted or rotation mode, the camera is configured to collect at least one of images or spatial data:

(a) whilst at least one or all of the sensor systems are being rotated around the rotational axis between rotations of the at least one or all of the sensor systems around the rotational axis, or (c) at one or more different rotational positions around the rotational axis.

14. The camera according to claim 1, wherein the camera is operable in a roaming mode in which the camera is manually at least one of movable, positionable, or orientable.

15. The camera according to claim 1, wherein the camera comprises or is configured to communicate with a processing system that is configured to create a model, composited image or virtual environment from the spatial data and/or the images collected from at least one or more or each of the sensor systems by combining or compositing the images collected by the image sensors with spatial data collected by the spatial sensors.

16. The camera according to claim 15, wherein:

the processing system is configured to form a 3D skeleton, framework or construct from at least the spatial data collected using at least one or more or each of the spatial sensors, the 3D skeleton, framework or construct comprising or defining one or more of;

positions, spatial extents, shapes and orientations of one or more surfaces and/or objects; and

the processing system is configured to wrap, skin or overlay the 3D skeleton, framework or construct with at least one or more images collected using at least one or more or each of the imaging sensors or a combined image formed therefrom to form the model, composited image or virtual environment.

17. The camera according to claim 16, wherein the processing system is configured to alter or configure or reorient the one or more or each image to conform to the 3D skeleton, framework or construct.

18. The camera according to claim 15, wherein the camera is configured to collect images with the imaging sensor(s) and/or spatial data with the spatial sensor(s) using a plurality or range of settings of at least one or each of the imaging sensors and/or spatial sensors and the processing system being configured to combine a plurality of at least partially or wholly overlapping images and/or spatial data collected with different settings in order to form at least part of the model, composited image or virtual environment.

19. The camera according to claim 18, wherein the camera is configured to provide high dynamic range (HDR) imaging by producing HDR images from the plurality of images collected with different exposures.

20. The camera according to claim 15, wherein the processing system is configured to determine which part of the model, composite image or virtual environment the image or spatial data is associated with or belongs to.

21. A method of collecting modelling and/or imaging data of an environment to be imaged or modelled, the method comprising the step of:

Collecting spatial data and images of the environment using the panoramic camera according to claim 1,

wherein the spatial data and images are collected using the plurality of sensor systems of the camera arranged such that at least part of the fields of view of at least two of the plurality of sensor systems, that respectively comprise fields of view of the at least one spatial sensor and the at least one imaging sensor of the respective sensor system, are distributed over a vertical direction, in use, such that a total field of view of all of the sensor systems extends above the camera.

22. The method of claim 21, comprising the step of collecting at least one of images or spatial data whilst the camera is mounted on the support and whilst at least one or all of the sensor systems are being rotated around the rotational axis or between rotations around the rotational axis, or at one or more different rotational positions around the rotational axis.

23. The method of claim 21 comprising the step of manually at least one of moving, repositioning, or re-orienting the camera and collecting images and spatial data at different positions and/or orientations of the camera.

24. A non-transitory carrier medium comprising a computer program product that, when loaded onto a processing and/or control apparatus and/or a camera comprising a processor, causes the processing and/or control apparatus and/or the camera to implement the method of claim 21.

25. A processing system for creating a model, composited image or virtual environment, the processing system comprising:

at least one processor for creating the model, composited image or virtual environment,

data storage for storing one or more images and spatial data of an environment to be imaged or modelled, and

a communications or interface module for receiving the images and/or spatial data,

wherein;

the one or more images and spatial data are collected using the panoramic camera according to claim 1, and

the one or more images and spatial data being indicative of fields of view of at least two of the plurality of sensor systems, that respectively comprise fields of view of the at least one spatial sensor and the at least one imaging sensor of the respective sensor system, that are distributed over a vertical direction such that a total field of view of all of the sensor systems extends above the camera.

26. A method for creating a model, composited image or virtual environment from one or more images and spatial data of an environment to be imaged or modelled, wherein at least one of the images and/or at least some of the spatial data relate to a different field of view to at least one other of the images or at least some other spatial data;

and the one or more images and spatial data are collected using the panoramic camera according to claim 1, the one or more images and spatial data being indicative of fields of view of at least two of the plurality of sensor systems, that respectively comprise fields of view of the at least one spatial sensor and the at least one imaging sensor of the respective sensor system, that are distributed over a vertical direction such that a total field of view of all of the sensor systems extends above the camera.