Method and system for constructing view from multiple video streams

• US 10,616,551 B2
• Filed: 01/26/2018
• Issued: 04/07/2020
• Est. Priority Date: 01/27/2017
• Status: Active Grant

• Current Assignee: Orbviu Inc.
• Original Assignee: Orbviu Inc.
• Inventors: Watson, Brian Michael Christopher, Mao, Crusoe Xiaodong, Shen, Jiandong, Umminger, III, Frederick William
• Primary Examiner(s): Le, Peter D
• Application Number:
  US15/881,608
• Publication Number:
  US 20180220120A1
• Time in Patent Office: 802 Days
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video

Abstract

A method and system for constructing view from multiple video streams is provided. The method includes receiving a view independent stream. The method further includes selecting a first view dependent stream, wherein the view independent stream and the first view dependent stream has at least one different geometry. The method also includes generating end user views corresponding to the view independent stream and the first view dependent stream. Further, the method includes blending the end user views to generate a view for display.

Claims

1. A method for constructing view from multiple video streams at a client device of a user, the method comprising:

receiving a view independent stream, the view independent stream representing a background video stream;

determining a view point of the user based on orientation and position of the client device;

selecting a first view dependent stream, representing a foreground video stream, based on orientation metadata associated with the determined view point, wherein the view independent stream and the first view dependent stream have at least one different geometry, and wherein the view independent stream has a higher field of view than the first view dependent stream;

generating end user views corresponding to the view independent stream and the first view dependent stream, wherein generating the end user views corresponding to the view independent stream and the first view dependent stream includes decoding the view independent stream and the first view dependent stream; and

blending the end user views to generate a 360-degree oriented view for display.

2. The method as claimed in claim 1, wherein selecting the first view dependent stream comprises:

accessing a metadata file, the metadata file comprising information of multiple view orientations and corresponding view for each view orientation; and

selecting the view corresponding to the view orientation, which is closest to the determined view point of the user.

3. The method as claimed in claim 1 and further comprising:

synchronizing the view independent stream and the first view dependent stream.

4. The method as claimed in claim 1 and further comprising:

displaying the 360-degree oriented view.

5. The method as claimed in claim 1 and further comprising:

selecting a second view dependent stream, wherein the view independent stream and the second view dependent stream have at least one different geometry;

generating end user views corresponding to the view independent stream and the second view dependent stream; and

blending the end user views to generate a view for display.

6. A system for constructing view from multiple video streams at a client device of a user, the system comprising:

a receiver to receive a view independent stream, the view independent stream representing a background video stream;

a selector to determine a view point of the user based on orientation and position of the client device, and select a first view dependent stream, representing a foreground video stream, based on orientation metadata associated with the determined view point, wherein the view independent stream and the first view dependent stream have at least one different geometry, and wherein the view independent stream has a higher field of view than the first view dependent stream;

a view generator to generate end user views corresponding to the view independent stream and the first view dependent stream, wherein to generate the end user views, the view generator decodes the view independent stream and the first view dependent stream; and

a blender to blend the end user views to generate a 360-degree oriented view for display.

7. The system as claimed in claim 6, wherein the selector:

accesses a metadata file, the metadata file comprising information of multiple view orientations and corresponding view for each view orientation; and

selects the view corresponding to the view orientation which is closest to the view point of the user.

8. The system as claimed in claim 6 and further comprising:

a synchronizer to synchronize the view independent stream and the first view dependent stream.

9. The system as claimed in claim 6 and further comprising:

a display for displaying the 360-degree oriented view.

10. The system as claimed in claim 6, wherein:

the selector further selects a second view dependent stream, wherein the view independent stream and the second view dependent stream have at least one different geometry;

the view generator further generates end user views corresponding to the view independent stream and the second view dependent stream; and

the blender further blends the end user views to generate a view for display.

11. A method for generating multiple video streams at a server, the method comprising:

receiving an input video stream;

generating a view independent stream, representing a background video stream, from the input video stream;

generating a first view dependent stream, representing a foreground video stream, using the input video stream and the view independent stream for a first viewpoint, wherein the view independent stream and the first view dependent stream have at least one different geometry, and wherein the view independent stream has a higher field of view than the first view dependent stream;

generating a metadata file, the metadata file comprising information of the first viewpoint and corresponding view of the first view dependent stream; and

storing the view independent stream, the first view dependent stream, and the metadata file.

12. The method as claimed in claim 11, wherein generating the view independent stream comprises:

converting equirectangular input video stream into cubemap stream; and

encoding the cubemap stream to generate the view independent stream.

13. The method as claimed in claim 11, wherein generating the first view dependent stream comprises:

performing view adaptive rotation on the input video stream;

converting rotated input video stream into a first trapezoid stream;

performing view adaptive rotation on the view independent stream;

converting rotated view independent stream into a second trapezoid stream;

processing the first trapezoid stream and the second trapezoid stream using a function; and

encoding the processed stream to generate the view dependent stream.

14. The method as claimed in claim 11 and further comprising:

generating a second view dependent stream using the input video stream and the view independent stream for a second viewpoint, wherein the independent view stream and the second view dependent stream have at least one different geometry; and

updating the metadata file with the second view point and corresponding view of the second view dependent stream.

15. A system for generating multiple video streams, the system comprising:

a receiver to receive an input video stream;

an independent view generator to generate a view independent stream, representing a background video stream, from the input video stream;

a dependent view generator to;

generate a first view dependent stream, representing a foreground video stream, using the input video stream and the view independent stream for a first viewpoint, wherein the view independent stream and the first view dependent stream have at least one different geometry, and wherein the view independent stream has a higher field of view than the first view dependent stream, and

generate a metadata file, the metadata file comprising information of first viewpoint and corresponding view of the first view dependent stream; and

a storage device to store the view independent stream, the first view dependent stream, and the metadata file.

16. The system as claimed in claim 15, wherein the independent view generator:

converts equirectangular input video stream into cubemap stream; and

encodes the cubemap stream to generate the view independent stream.

17. The system as claimed in claim 15, wherein the dependent view generator:

performs view adaptive rotation on the input video stream;

converts rotated input video stream into a first trapezoid stream;

performs view adaptive rotation on the view independent stream;

converts rotated view independent stream into a second trapezoid stream; and

processes the first trapezoid stream and the second trapezoid stream using a function; and

encodes the processed stream to generate the view dependent stream.

18. The system as claimed in claim 17, wherein the dependent view generator further:

generates a second view dependent stream using the input video stream and the view independent stream for a second viewpoint, wherein the independent view stream and the second view dependent stream have at least one different geometry; and

updating the metadata file with the second viewpoint and corresponding view of the second view dependent stream.