Imaging system, imaging apparatus, and system

• US 10,681,268 B2
• Filed: 11/13/2015
• Issued: 06/09/2020
• Est. Priority Date: 05/15/2014
• Status: Active Grant

• Current Assignee: Ricoh Company Limited
• Original Assignee: Ricoh Company Limited
• Inventors: Yamamoto, Hideaki
• Primary Examiner(s): Ustaris, Joseph G
• Assistant Examiner(s): Lee, Jimmy S
• Application Number:
  US14/941,152
• Publication Number:
  US 20160073024A1
• Time in Patent Office: 1,670 Days
• Field of Search: 348 36
• CPC Class Codes: G03B 37/00 Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

An imaging system includes an imaging unit, a detecting unit, a recording unit, a correcting unit, and a video generating unit. The imaging unit captures images of a plurality frames. The detecting unit detects a tilt of the imaging unit with respect to a reference direction. The recording unit records time-series data of the tilt detected by the detecting unit. The correcting unit applies tilt correction to an image of each of the frames captured by the imaging unit, based on the time-series data of the tilt recorded by the recording unit and additional information unique to the imaging unit. The video generating unit generates video data based on the image of each of the frames corrected by the correcting unit.

Claims

1. An imaging system, comprising:

an omnidirectional camera including a plurality of lenses, each with a different image capturing direction, and first processing circuitry configured to

capture a plurality of associated images through the plurality of lenses over a period of time to capture a sequence of the plurality of associated images;

detect, while capturing the sequence of the plurality of associated images, a tilt angle of the omnidirectional camera with respect to a direction of gravity and generate tilt angle data; and

record a time series of the tilt angle data corresponding to the sequence of the plurality of associated images into a memory; and

an image processing apparatus including second processing circuitry configured to

access the sequence of the plurality of associated images and the time series of the tilt angle data corresponding to the plurality of associated images; and

apply, based on the time series of the tilt angle data, a tilt correction with respect to the direction of gravity to each of the images of the sequence of the plurality of associated images to generate omnidirectional video data including a plurality of frames of omnidirectional images with corrected tilt angles, so that a zenith direction of the omnidirectional images of the plurality of frames that are corrected coincide with the direction of gravity,

wherein;

the first processing circuitry is further configured to generate fisheye images, and the second processing circuitry is further configured to convert the fisheye images into the omnidirectional images by using a conversion table, the conversion table including design data of lens optical systems of the omnidirectional camera, and

the conversion table is modified to reflect the zenith direction when a direct upward direction of the omnidirectional camera does not coincide with a vertical line.

2. The imaging system according to claim 1, wherein

the first processing circuitry is further configured to

compress the sequence of the plurality of associated images, and

output the sequence of the plurality of associated images which has been compressed to the image processing apparatus,

the second processing circuitry is further configured to receive the sequence of the plurality of associated images which has been compressed and output by the first processing circuitry, and

in applying the tilt correction, the second processing circuitry is configured to

decompress the sequence of the plurality of associated images which has been compressed,

apply the tilt correction to the sequence of the plurality of associated images which has been decompressed, and

generate the omnidirectional images with corrected tilt angles from the sequence of the plurality of associated images which has been decompressed.

3. The imaging system according to claim 1, wherein

the first processing circuitry is configured to output the sequence of the plurality of associated images to the image processing apparatus, and

the second processing circuitry is configured to receive the sequence of the plurality of associated images output by the first processing circuitry.

4. The imaging system according to claim 1, wherein

the omnidirectional camera includes a first wide-angle imaging optical system including one of the plurality of lenses and a second wide-angle imaging optical system including another of the plurality of lenses,

the first processing circuitry captures the sequence of the plurality of associated images of the plurality of frames by controlling the first wide-angle imaging optical system to capture first partial omnidirectional images and by controlling the second wide-angle imaging optical system to capture second partial omnidirectional images,

the second processing circuitry is configured to

decompress the sequence of the plurality of associated images into the first and second partial omnidirectional images,

modify, according to the time series of the tilt angle data, conversion data for converting the first partial omnidirectional images and the second partial omnidirectional images into data in a spherical coordinate system,

convert, using the modified conversion data, the first partial omnidirectional images and the second partial omnidirectional images into data in the spherical coordinate system,

generate adjusted omnidirectional images from the first partial omnidirectional images and the second partial omnidirectional images, and

generate adjusted video data based on the adjusted omnidirectional images.

5. The imaging system according to claim 1, wherein the second processing circuitry is configured to load into a second memory, to apply the tilt correction, some of the images of the sequence of the plurality of associated images that are based on a starting frame.

6. The imaging system according to claim 1, wherein

the omnidirectional camera includes an acceleration sensor, and

the first processing circuitry detects the tilt angle of the omnidirectional camera according to an output of the acceleration sensor.

16. The imaging system according to claim 1, wherein the second processing circuitry is configured to, in applying the tilt correction,

apply the tilt correction to an image conversion table unique to the omnidirectional camera based on the time series of the tilt angle data, the image conversion table for adjusting the omnidirectional images, and

generate the omnidirectional video data based on the images of the sequence of the plurality of associated images and the image conversion table corrected by the applied tilt correction.

17. The imaging system according to claim 1, wherein

the first processing circuitry is configured to wirelessly transmit the images of the sequence of the plurality of associated images and the time series of the tilt angle data to the image processing apparatus, and

the second processing circuitry is configured to receive the images of the sequence of the plurality of associated images and the time series of the tilt angle data from the omnidirectional camera.

19. The imaging system according to claim 1, wherein the first processing circuitry is further configured to

generate signals of acceleration components of a sensor based on the detected tilt angle, and

record the plurality of pieces of tilt angle data corresponding to the sequence of the plurality of associated images into the memory by sampling the signals of the acceleration components in synchronization with the plurality of frames.

7. An imaging apparatus, comprising:

processing circuitry configured to

capture a plurality of associated images through a plurality of lenses over a period of time to capture a sequence of the plurality of associated images of a subject;

detect a tilt of the imaging apparatus during capture of the sequence of the plurality of associated images with respect to a direction of gravity;

generate a time series of tilt angle data of the imaging apparatus according to the detected tilt;

record the sequence of the plurality of associated images; and

record, into a memory, the time series of the tilt angle data corresponding to the sequence of the plurality of associated images, wherein

an image processing apparatus accesses the sequence of the plurality of associated images and the time series of the tilt angle data corresponding to the sequence of the plurality of associated images and applies, based on corresponding tilt angle data, a tilt correction with respect to the direction of gravity to the sequence of the plurality of associated images to generate omnidirectional video data including adjusted omnidirectional images in a spherical coordinate system and having corrected tilt so that a zenith direction of the adjusted omnidirectional images coincide with the direction of gravity,

wherein;

the capture of the plurality of associated images captures fisheye images,

the processing circuitry further configured to convert the fisheye images into the omnidirectional images by using a conversion table, the conversion table including design data of lens optical systems of an omnidirectional camera that captures the fisheye images, and

the processing circuitry to modify the conversion table to reflect the zenith direction when a direct upward direction of the omnidirectional camera does not coincide with a vertical line.

8. The imaging apparatus according to claim 7, wherein the processing circuitry is configured to associate each of the sequence of the plurality of associated images with corresponding tilt angle data based on the tilt of the imaging apparatus during capture of the sequence of the plurality of associated images.

9. The imaging apparatus according to claim 7, further comprising:

a plurality of cameras, wherein

the processing circuitry is configured to generate the omnidirectional video data based on the sequence of the plurality of associated images output from the plurality of cameras, and

the processing circuitry is configured to output the omnidirectional video data.

10. The imaging apparatus according to claim 7, further comprising:

a first wide-angle imaging optical system; and

a second wide-angle imaging optical system, wherein

the processing circuitry is configured to capture the sequence of the plurality of associated images by controlling the first wide-angle imaging optical system to capture first partial omnidirectional images and by controlling the second wide-angle imaging optical system to capture second partial omnidirectional images.

11. The imaging apparatus according to claim 7, further comprising:

an acceleration sensor, wherein

the processing circuitry is configured to detect the tilt of the imaging apparatus according to an output of the acceleration sensor.

18. The imaging apparatus according to claim 7, wherein the processing circuitry is configured to wirelessly transmit the images of the sequence of the plurality of associated images and the time series of the tilt angle data to the image processing apparatus.

12. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by processing circuitry, cause an image processing apparatus to:

receive a sequence of a plurality of associated images which are fisheye images and which has been captured through a plurality of lenses over a period of time, and a time series of tilt angle data corresponding to the sequence of the plurality of associated images, the time series of tilt angle data representing a tilt angle of the imaging apparatus with respect to a direction of gravity during capture of the sequence of a plurality of associated images; and

apply, based on corresponding tilt angle data of the time series of tilt angle data, a tilt correction with respect to the direction of gravity to the sequence of the plurality of associated images to generate omnidirectional video data including omnidirectional images with corrected tilt angles using a conversion table, so that a zenith direction of the omnidirectional images that are corrected coincide with the direction of gravity, the conversion table including design data of lens optical systems of an omnidirectional camera

wherein the conversion table is modified to reflect the zenith direction when a direct upward direction of the omnidirectional camera does not coincide with a vertical line.

13. The non-transitory computer-readable medium according to claim 12, wherein

the sequence of the plurality of associated images are captured by an omnidirectional camera that comprises a first wide-angle imaging optical system and a second wide-angle imaging optical system,

the omnidirectional camera captures the omnidirectional images by controlling the first wide-angle imaging optical system which includes a lens of the plurality of lenses to capture first partial omnidirectional images and by controlling the second wide-angle imaging optical system which includes another lens of the plurality of lenses to capture second partial omnidirectional images,

the image processing apparatus is further caused to:

decompress the sequence of the plurality of associated images into the first and second partial omnidirectional images,

modify, according to the tilt angle of the imaging apparatus, conversion data for converting the first partial omnidirectional images and the second partial omnidirectional images into data in a spherical coordinate system;

convert, using the modified conversion data, the first partial omnidirectional images and the second partial omnidirectional images into data in the spherical coordinate system;

generate adjusted omnidirectional images from the first partial omnidirectional images and the second partial omnidirectional images; and

generate adjusted video data based on the adjusted omnidirectional images.

14. The non-transitory computer-readable medium according to claim 12, wherein the image processing apparatus is further caused to load, to apply the tilt correction, some of the omnidirectional images based on a starting image.

15. The non-transitory computer-readable medium according to claim 12, wherein

a device which captures the sequence of the plurality of associated images includes an acceleration sensor, and

the device which captures the sequence of the plurality of associated images detects the tilt of the imaging apparatus according to an output of the acceleration sensor.

20. An imaging system, comprising:

an omnidirectional camera including first processing circuitry configured to

capture omnidirectional images of a plurality of frames;

detect a tilt angle of the omnidirectional camera with respect to a direction of gravity;

record the omnidirectional images of the plurality of frames; and

record a plurality of pieces of tilt angle data corresponding to the plurality of frames into a memory, each piece of tilt angle data corresponding to a respective frame of the plurality of frames and including the detected tilt angle of the omnidirectional camera with respect to the direction of gravity; and

an image processing apparatus including second processing circuitry configured to

access the omnidirectional images of the plurality of frames and the plurality of pieces of tilt angle data corresponding to the plurality of frames; and

apply, based on a corresponding piece of tilt angle data of the plurality of pieces of tilt angle data, a tilt correction with respect to the direction of gravity to each of the omnidirectional images of the plurality of frames to generate omnidirectional video data including omnidirectional images with corrected tilt angles, so that a zenith direction of the omnidirectional images of the plurality of frames that are corrected coincide with the direction of gravity,

wherein the first processing circuitry is further configured to generate fisheye images, and the second processing circuitry is further configured to convert the fisheye images into the omnidirectional images by using a conversion table, the conversion table including design data of lens optical systems of the omnidirectional camera, and

wherein the conversion table is modified to reflect the zenith direction when a direct upward direction of the omnidirectional camera does not coincide with a vertical line.