Reconstruction of three-dimensional video

• US 10,257,494 B2
• Filed: 09/16/2015
• Issued: 04/09/2019
• Est. Priority Date: 09/22/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Sadi, Sajid, Perdices-Gonzalez, Sergio, Budhiraja, Rahul, Lee, Brian Dongwoo, Khwaja, Ayesha Mudassir, Mistry, Pranav
• Primary Examiner(s): Suh, Joseph
• Application Number:
  US14/856,504
• Publication Number:
  US 20160088285A1
• Time in Patent Office: 1,301 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G03B 35/08 by simultaneous recording
• CPC Class Codes: G03B 37/04 with cameras or projectors providing touching or overlapping fields of view
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/54 using feature points or meshes
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/265 Mixing

Abstract

In one embodiment, a method includes accessing multiple images representing a 360° view of a scene, each image representing a portion of the 360° scene and including a first overlap area and a second overlap area, the first overlap area corresponding to an overlap with a first adjacent image, and the second overlap area corresponding to an overlap with a second adjacent image. The method also includes, calculating, for each overlap area, a set of homographies based on one or more feature points in the overlap area. The method further includes creating a set of left images from the multiple images, the set of left images representing a first 360° view of the scene. The method also includes creating a set of right images from the multiple images, the set of right images representing a second 360° view of the scene.

Claims

1. A method comprising:

accessing a plurality of images representing a 360°

view of a scene, each image representing a portion of the 360°

view of the scene and comprising a first overlap area and a second overlap area, the first overlap area corresponding to an overlap with a first adjacent image, and the second overlap area corresponding to an overlap with a second adjacent image, wherein the plurality of accessed images without modification do not produce a three-dimensional view of the scene;

calculating, for each of the overlap areas, a set of homographies based on one or more feature points in the overlap area;

creating a set of left images from the plurality of images, the set of left images representing a first 360°

view of the scene;

creating a set of right images from the plurality of images, the set of right images representing a second 360°

view of the scene, wherein the set of right images is distinct from the set of left images;

stitching the set of left images together using the calculated homographies to produce a stitched 360°

left image;

stitching the set of right images together using the calculated homographies to produce a stitched 360°

right image that is distinct from the stitched 360°

left image;

providing for display to a left eye of a user at least a portion of the stitched 360°

left image; and

providing for display to a right eye of the user at least a portion of the stitched 360°

right image.

2. The method of claim 1, wherein the plurality of images are captured by a plurality of cameras, wherein each camera of the plurality of cameras has greater than or equal to 50% angular overlap with an adjacent camera.

3. The method of claim 2, wherein each camera has a different orientation.

4. The method of claim 1, wherein:

the first overlap area of an image of the plurality of images comprises greater than or equal to 50% of an area of the image; and

the second overlap area of the image comprises greater than or equal to 50% of the area of the image.

5. The method of claim 1, wherein:

the set of left images is formed from the first overlap area of each of the plurality of images; and

the set of right images is formed from the second overlap area of each of the plurality of images.

6. The method of claim 1, further comprising, stitching the plurality of images together to form a stitched image, the stitching based at least in part on the feature points in overlap areas of adjacent images.

7. The method of claim 1, further comprising:

displaying to the left eye of the user at least a portion of the stitched 360°

left image; and

displaying to the right eye of the user at least a portion of the stitched 360°

right image.

8. The method of claim 1, wherein:

providing for display to the left eye of the user at least the portion of the stitched 360°

left image comprises sending, to a client computing device of the user, at least the portion of the stitched 360°

left image; and

providing for display to the right eye of the user at least the portion of the stitched 360°

right image comprises sending, to the client computing device of the user, at least the portion of the stitched 360°

right image.

9. The method of claim 1, wherein one or both of the set of left images and the set of right images is created using one or more masks.

10. The method of claim 1, wherein one or both of the set of left images and the set of right images is created using direct image segmentation.

11. One or more non-transitory computer-readable storage media comprising instructions that are operable when executed to:

access a plurality of images representing a 360°

view of a scene, each image representing a portion of the 360°

view of the scene and comprising a first overlap area and a second overlap area, the first overlap area corresponding to an overlap with a first adjacent image, and the second overlap area corresponding to an overlap with a second adjacent image, wherein the plurality of accessed images without modification do not produce a three-dimensional view of the scene;

calculate, for each of the overlap areas, a set of homographies based on one or more feature points in the overlap area;

create a set of left images from the plurality of images, the set of left images representing a first 360°

view of the scene;

create a set of right images from the plurality of images, the set of right images representing a second 360°

view of the scene, wherein the set of right images is distinct from the set of left images;

stitch the set of left images together using the calculated homographies to produce a stitched 360°

left image;

stitch the set of right images together using the calculated homographies to produce a stitched 360°

right image that is distinct from the stitched 360°

left image;

provide for display to a left eye of a user at least a portion of the stitched 360°

left image; and

provide for display to a right eye of the user at least a portion of the stitched 360°

right image.

12. The media of claim 11, wherein the plurality of images are captured by a plurality of cameras, wherein each camera of the plurality of cameras has greater than or equal to 50% angular overlap with an adjacent camera.

13. The media of claim 12, wherein each camera has a different orientation.

14. The media of claim 11, wherein:

the first overlap area of an image of the plurality of images comprises greater than or equal to 50% of an area of the image; and

the second overlap area of the image comprises greater than or equal to 50% of the area of the image.

15. The media of claim 11, wherein:

the set of left images is formed from the first overlap area of each of the plurality of images; and

the set of right images is formed from the second overlap area of each of the plurality of images.

16. The media of claim 11, wherein the instructions are further operable when executed to stitch the plurality of images together to form a stitched image, the stitching based at least in part on the feature points in overlap areas of adjacent images.

17. The media of claim 11, wherein the instructions are further operable when executed to:

display to the left eye of the user at least the portion of the stitched 360°

left image; and

display to the right eye of the user at least the portion of the stitched 360°

right image.

18. The media of claim 11, wherein:

providing for display to the left eye of the user at least the portion of the stitched 360°

left image comprises sending, to a client computing device of the user, at least the portion of the stitched 360°

left image; and

providing for display to the right eye of the user at least the portion of the stitched 360°

right image comprises sending, to the client computing device of the user, at least the portion of the stitched 360°

right image.

19. An apparatus comprising:

one or more non-transitory computer-readable storage media embodying instructions; and

one or more processors coupled to the storage media and configured to execute the instructions to;

access a plurality of images representing a 360°

view of a scene, each image representing a portion of the 360°

view of the scene and comprising a first overlap area and a second overlap area, the first overlap area corresponding to an overlap with a first adjacent image, and the second overlap area corresponding to an overlap with a second adjacent image, wherein the plurality of accessed images without modification do not produce a three-dimensional view of the scene;

calculate, for each of the overlap areas, a set of homographies based on one or more feature points in the overlap area;

create a set of left images from the plurality of images, the set of left images representing a first 360°

view of the scene;

create a set of right images from the plurality of images, the set of right images representing a second 360°

view of the scene, wherein the set of right images is distinct from the set of left images;

stitch the set of left images together using the calculated homographies to produce a stitched 360°

left image;

stitch the set of right images together using the calculated homographies to produce a stitched 360°

right image that is distinct from the stitched 360°

left image;

provide for display to a left eye of a user at least a portion of the stitched 360°

left image; and

provide for display to a right eye of the user at least a portion of the stitched 360°

right image.

20. The apparatus of claim 19, wherein the plurality of images are captured by a plurality of cameras, wherein each camera of the plurality of cameras has greater than or equal to 50% angular overlap with an adjacent camera.

21. The apparatus of claim 20, wherein each camera has a different orientation.

22. The apparatus of claim 19, wherein:

the first overlap area of an image of the plurality of images comprises greater than or equal to 50% of an area of the image; and

the second overlap area of the image comprises greater than or equal to 50% of the area of the image.

23. The apparatus of claim 19, wherein:

the set of left images is formed from the first overlap area of each of the plurality of images; and

the set of right images is formed from the second overlap area of each of the plurality of images.

24. The apparatus of claim 19, wherein the processors are further configured to execute the instructions to stitch the plurality of images together to form a stitched image, the stitching based at least in part on the feature points in overlap areas of adjacent images.

25. The apparatus of claim 19, wherein the processors are further configured to execute the instructions to:

display to the left eye of the user at least the portion of the stitched 360°

left image; and

display to the right eye of the user at least the portion of the stitched 360°

right image.

26. The apparatus of claim 19, wherein:

providing for display to the left eye of the user at least the portion of the stitched 360°

left image comprises sending, to a client computing device of the user, at least the portion of the stitched 360°

left image; and

providing for display to the right eye of the user at least the portion of the stitched 360°

right image comprises sending, to the client computing device of the user, at least the portion of the stitched 360°

right image.