IMAGE STITCHING FOR THREE-DIMENSIONAL VIDEO

• US 20160088287A1
• Filed: 09/16/2015
• Published: 03/24/2016
• Est. Priority Date: 09/22/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Sadi, Sajid, Perdices-Gonzalez, Sergio, Budhiraja, Rahul, Lee, Brian Dongwoo, Khwaja, Ayesha Mudassir, Mistry, Pranav
• Granted Patent: US 10,313,656 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 23/957 : Light-field or plenoptic cameras or camera modules
• CPC Class Codes: H04N 23/959 : by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
• CPC Class Codes: H04N 5/265 : Mixing

Abstract

In one embodiment, a method includes receiving multiple images from multiple cameras, where the multiple cameras include a first camera having a first orientation and a second camera having a second orientation, and the multiple images include a first image from the first camera and a second image from the second camera. The method also includes detecting multiple feature points within the first and second images. The method further includes determining one or more pairs of corresponding feature points located within a region of overlap between the first and second images, where the pairs of corresponding feature points include a respective one of the feature points from each of the first and second images. The method also includes spatially adjusting the first or second image based on a calculated offset between each pair of corresponding feature points.

Claims

1. A method comprising:

receiving a plurality of images from a plurality of cameras, wherein the plurality of cameras comprise a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;

detecting a plurality of feature points within the first and second images;

determining one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;

spatially adjusting the first or second image based on a calculated offset between each pair of corresponding feature points; and

combining the first and second images into a merged image based on the spatial adjustment.

2. The method of claim 1, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

3. The method of claim 1, wherein the detection comprises:

detecting one or more features of the first and second images based on a local contrast gradient; and

extracting one or more feature points of the detected features.

4. The method of claim 3, wherein the extraction comprises using a scale-invariant feature transform (SIFT), speeded up robust features (SURF), or oriented FAST and Rotated BRIEF (ORB) feature detection algorithm.

5. The method of claim 1, wherein the determination comprises:

partitioning the first and second images into a pre-determined number of regions in a pre-determined grid;

comparing feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and

identifying the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

6. The method of claim 5, wherein the comparison within each region along the edge is performed on a pre-determined area around each feature point, wherein the pre-determined area is based on the number of pixels of the region of overlap.

7. The method of claim 6, further comprising determining whether a sum of square of differences in the pre-determined area is less than a pre-determined threshold value.

8. The method of claim 1, further comprising receiving a third image from a third camera, wherein the third image is captured during a same time period as the first and second images, wherein the plurality of cameras further comprises the third camera having the first orientation, and wherein the second camera is located between the first and third cameras.

9. The method of claim 8, further comprising:

detecting a plurality of feature points within the third image;

determining one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and

determining one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.

10. The method of claim 8, wherein the first and third camera comprise a stereoscopic pair of cameras.

11. The method of claim 1, further comprising receiving a third image from a third camera, wherein the plurality of cameras further comprises a third camera having an orientation that is approximately orthogonal to a plane defined by the first and second cameras.

12. The method of claim 11, further comprising:

determining one or more pairs of corresponding feature points located within a region of overlap between the merged image and the third image, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the merged image and the third image; and

applying a radial warp around a center of the third image, such the center of the third image is fixed.

13. The method of claim 1, wherein the calculation comprises calculating a cost function that is proportional to an absolute value of a spatial offset between each pair of corresponding feature points.

14. The method of claim 1, wherein the calculation comprises calculating a cost function that is proportional to a rotational offset between each pair of corresponding feature points along one or more axes of rotation.

15. The method of claim 1, wherein the second orientation is different from the first orientation, and wherein each image is captured during a same period of time.

16. The method of claim 1, wherein combining the first and second images comprises:

detecting an edge in the region of overlap; and

applying a gradient on the edge.

17. The method of claim 1, wherein the adjustment comprises a rotation or translation of the first or second image.

18. One or more non-transitory computer-readable storage media embodying instructions that are operable when executed to:

receive a plurality of images from a plurality of cameras, wherein the plurality of cameras comprise a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;

detect a plurality of feature points within the first and second images;

determine one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;

spatially adjust the first or second image based on a calculated offset between each pair of corresponding feature points; and

combine the first and second images into a merged image based on the spatial adjustment.

19. The media of claim 18, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

20. The media of claim 18, wherein the detect the plurality of feature points within the first and second images comprises:

detect one or more features of the first and second images based on a local contrast gradient; and

extract one or more feature points of the detected features.

21. The media of claim 18, wherein the determine the one or more pairs of corresponding feature points located within the region of overlap between the first and second images comprises:

partition the first and second images into a pre-determined number of regions in a pre-determined grid;

compare feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and

identify the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

22. The media of claim 18, wherein the instructions are further operable when executed to receive a third image from a third camera, wherein the third image is captured during a same time period as the first and second images, wherein the plurality of cameras further comprises the third camera having the first orientation, and wherein the second camera is located between the first and third cameras.

23. The media of claim 22, wherein the instructions are further operable when executed to:

detect a plurality of feature points within the third image;

determine one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and

determine one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.

24. The media of claim 18, wherein the calculated offset between each pair of corresponding feature points comprises a cost function that is proportional to an absolute value of a spatial offset between each pair of corresponding feature points.

25. An apparatus comprising:

one or more non-transitory computer-readable storage media embodying instructions; and

one or more processors coupled to the storage media and configured to execute the instructions to;

receive a plurality of images from a plurality of cameras, wherein the plurality of cameras comprise a first camera having a first orientation and a second camera having a second orientation, wherein the plurality of images comprises a first image from the first camera and a second image from the second camera;

detect a plurality of feature points within the first and second images;

determine one or more pairs of corresponding feature points located within a region of overlap between the first and second images, wherein the pairs of corresponding feature points comprise a respective one of the feature points from each of the first and second images;

spatially adjust the first or second image based on a calculated offset between each pair of corresponding feature points; and

combine the first and second images into a merged image based on the spatial adjustment.

26. The apparatus of claim 25, wherein the region of overlap between the first and second images is proportional to an overlap in a respective field of view of the first and second cameras.

27. The apparatus of claim 25, wherein the detect the plurality of feature points within the first and second images comprises:

detect one or more features of the first and second images based on a local contrast gradient; and

extract one or more feature points of the detected features.

28. The apparatus of claim 25, wherein the determine the one or more pairs of corresponding feature points located within the region of overlap between the first and second images comprises:

partition the first and second images into a pre-determined number of regions in a pre-determined grid;

compare feature points located within each region along an edge of the first image with a corresponding region along an opposite edge of the second image; and

identify the corresponding pairs of feature points based on a pattern of feature points within each region of the edges.

29. The apparatus of claim 25, wherein the processors are further configured to execute the instructions to receive a third image from a third camera, wherein the third image is captured during a same time period as the first and second images, wherein the plurality of cameras further comprises the third camera having the first orientation, and wherein the second camera is located between the first and third cameras.

30. The apparatus of claim 29, wherein the processors are further configured to execute the instructions to:

detect a plurality of feature points within the third image;

determine one or more pairs of corresponding feature points located within the third image and the region of overlap between the first and third images, wherein the pairs of corresponding feature points of the first and third images comprise a respective one of the feature points from each of the first and third images; and

determine one or more pairs of corresponding feature points located within a region of overlap between the second and third images, wherein the pairs of corresponding feature points of the second and third images comprise a respective one of the feature points from each of the second and third images.

31. The apparatus of claim 25, wherein the calculated offset between each pair of corresponding feature points comprises a cost function that is proportional to an absolute value of a spatial offset between each pair of corresponding feature points.