Imaging device and method for generating an undistorted wide view image

• US 9,819,865 B2
• Filed: 10/27/2016
• Issued: 11/14/2017
• Est. Priority Date: 10/30/2015
• Status: Expired due to Fees

• Current Assignee: Essential Products, Inc.
• Original Assignee: Essential Products, Inc.
• Inventors: Evans, V, David John, Jiang, Xinrui, Rubin, Andrew E., Hershenson, Matthew, Miao, Xiaoyu, Tate, Joseph Anthony, Gagne-Keats, Jason Sean
• Primary Examiner(s): Aghevli, Reza
• Application Number:
  US15/336,588
• Publication Number:
  US 20170126972A1
• Time in Patent Office: 383 Days
• Field of Search: 348 38
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 21/602 Providing cryptographic facilities or services
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/12 Panospheric to cylindrical image transformations
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Certain aspects of the technology disclosed herein involve combining images to generate a wide view image of a surrounding environment. Images can be recorded using an stand-alone imaging device having wide angle lenses and/or normal lenses. Images from the imaging device can be combined using methods described herein. In an embodiment, a pixel correspondence between a first image and a second image can be determined, based on a corresponding overlap area associated with the first image and the second image. Corresponding pixels in the corresponding overlap area associated with the first image and the second image can be merged based on a weight assigned to each of the corresponding pixels.

Claims

1. A method to generate an undistorted 360°

image of a surrounding environment, the method comprising;

receiving a first plurality of images corresponding to a first plurality of cameras, and a second plurality of images corresponding to a second plurality of cameras from a 360°

imaging device, wherein at least one camera of the first plurality of cameras and the second plurality of cameras has a wide angle lens;

dewarping a distorted image of the first plurality of images or the second plurality of images, wherein the at least one camera having the wide angle lens recorded the distorted image;

determining a pixel correspondence between a first image in the first plurality of images and a second image in the second plurality of images, based on a corresponding overlap area associated with the first image and the second image; and

combining the first plurality of images and the second plurality of images by merging corresponding pixels in the corresponding overlap area, said merging comprising;

for each pixel in the first image associated with the corresponding overlap area, assigning a first weight, and wherein the first weight is associated with a determined distortion of a pixel in the first image; and

for each corresponding pixel in the second images associated with the corresponding overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;

wherein the weight dictates a proportional contribution of each pixel to values of a merged pixel; and

tiling one or more non-overlapping areas and the corresponding overlap area to obtain a 360°

image.

2. The method of claim 1, further comprising:

obtaining a plurality of mapping functions corresponding to the at least one camera having the wide angle lens, wherein a mapping function in the plurality of mapping functions comprises a transformation, wherein when the transformation is applied to the distorted image associated with the wide angle lens, the transformation dewarping the distorted image and producing an undistorted image.

3. The method of claim 2, wherein said obtaining the plurality of mapping functions comprises:

recording a first image using the wide angle lens;

recording a second image using a normal lens, wherein the normal lens is disposed the same way as the wide angle lens;

calculating the mapping function; and

storing the mapping function in a memory device.

4. The method of claim 2, wherein said obtaining the plurality of mapping functions comprises retrieving the plurality of mapping functions from a memory device.

5. The method of claim 1, wherein the said determining the pixel correspondence comprises:

identifying a plurality of overlap areas associated with the first image in the first plurality of images and the second image in the second plurality of images;

calculating a plurality of first match scores corresponding to the plurality of overlap areas;

identifying an overlap area of the plurality of overlap areas having a highest first match score of the plurality of first match scores as the corresponding overlap area;

identifying one or more first set of pixels in the corresponding overlap area of the first image and one or more second set of pixels in the corresponding overlap area of the second image;

calculating a plurality of second match scores corresponding to the one or more first set of pixels and the one or more second set of pixels; and

identifying the one or more first sets of pixels and the one or more second sets of pixels having a highest second match score of the plurality of second match scores as corresponding pixels.

6. The method of claim 1, further comprising:

compressing the undistorted 360°

image by storing image data of an identified object in a database and inserting a reference to the stored image data of the identified object in the database; and

encrypting the inserted reference to the stored image data of the identified object.

7. The method of claim 1, further comprising:

identifying an object of a class in an image of the first plurality of images or the second plurality of images.

8. The method of claim 7, further comprising:

performing any of;

removing the identified object from the image by merging pixels of the identified object with corresponding pixels of a corresponding overlap area, wherein the pixels of the identified object are assigned a weight of 0;

or

locally dewarp a region surround the identified object by mapping corresponding pixels of the identified object in the first image and the second image to determine a distortion function and utilizing the distortion function to normalize the region surrounding the identified object.

9. The method of claim 1, further comprising:

identifying an object of a class in an image of the first plurality of images or the second plurality of images recorded by a camera of the first plurality of cameras or the second plurality of cameras;

determining a location of the identified object in another image recorded by another camera; and

focusing on the determined location of the identified object in the another image recorded by the another camera.

10. The method of claim 1, wherein merging corresponding pixels comprises evaluating weighted scores associated with a residual distortion of the corresponding pixels to determine a proportional contribution of each of the corresponding pixels.

11. An imaging device capable of recording an undistorted full view of a surrounding environment, the imaging device comprising:

a first plurality of cameras comprising a plurality of wide angle lenses, wherein the first plurality of cameras is configured to record a first plurality of images associated with the surrounding environment,

a second plurality of cameras comprising a plurality of normal lenses, wherein the second plurality of cameras is configured to record a second plurality of images associated with the surrounding environment; and

a processor configured to perform a method comprising;

obtain the first plurality of images and the second plurality of images;

dewarp the first plurality of images;

determine a pixel correspondence between a first image of the first plurality of images and a second image of the second plurality of images, based on a corresponding overlap area associated with the first image and the second image; and

combine the first plurality of images and the second plurality of images by merging corresponding pixels in the corresponding overlap area, said merging comprising;

for each pixel in the first image associated with the corresponding overlap area, assigning a first weight, and wherein the first weight is associated with a determined distortion of a pixel in the first image; and

for each corresponding pixel in the second images associated with the corresponding overlap area, weighing the pixel by a second weight, wherein the second weight increases as the first weight decreases;

wherein the weight dictates a proportional contribution of each pixel to values of a merged pixel; and

tiling one or more non-overlapping areas and the corresponding overlap area to obtain an undistorted full view image associated with the surrounding environment.

12. The imaging device of claim 11, wherein the processor comprises any of:

a processor within the imaging device; and

a processor connected to the imaging device.

13. The imaging device of claim 11, wherein determining the pixel correspondence comprises:

identifying a plurality of overlap areas associated with the first image in the first plurality of images and the second image in the second plurality of images;

calculating a plurality of first match scores corresponding to the plurality of overlap areas;

identifying an overlap area of the plurality of overlap areas having a highest first match score of the plurality of first match scores as the corresponding overlap area;

identifying one or more first set of pixels in the corresponding overlap area of the first image and one or more second set of pixels in the corresponding overlap area of the second image;

calculating a plurality of second match scores corresponding to the one or more first set of pixels and the one or more second set of pixels; and

identifying the one or more first sets of pixels and the one or more second sets of pixels having a highest second match score of the plurality of second match scores as corresponding pixels.

14. The imaging device of claim 11, wherein each image in the first plurality of images comprises a first image periphery distorted by a wide angle lens in the plurality of wide angle lenses, and a first image center undistorted by the wide angle lens;

and

wherein each image in the second plurality of images comprises a second image center undistorted by a camera in the second plurality of cameras, wherein the second plurality of image centers associated with the second plurality of images overlaps the first plurality of image peripheries associated with the first plurality of images.

15. The imaging device of claim 11, wherein merging corresponding pixels comprises evaluating weighted scores associated with a residual distortion of the corresponding pixels to determine a proportional contribution of each of the corresponding pixels.