METHOD AND SYSTEM FOR CREATING A 3D VIDEO FROM A MONOSCOPIC 2D VIDEO AND CORRESPONDING DEPTH INFORMATION

• US 20120050481A1
• Filed: 03/31/2011
• Published: 03/01/2012
• Est. Priority Date: 08/27/2010
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Chen, Xuemin, Seshadri, Nambi, Karaoguz, Jeyhan, Boross, Chris
• Granted Patent: US 8,994,792 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/33 in the spatial domain
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

Image sensors and a depth sensor of a monoscopic video sensing device are utilized to capture a 2D video and corresponding depth information. Regions of interest (ROIs) for the captured 2D video are selected based on the captured corresponding depth information. The monoscopic video sensing device selectively processes the captured 2D video and the captured corresponding depth information based on the selected ROIs. A 3D video is composed from the processed 2D video for display. The captured depth information that is synchronized to the captured 2D video is stored as metadata, and may be interpolated to match video resolution of the captured 2D video. The captured 2D video and the captured corresponding depth information are enhanced through scalable video coding. With 3D video rendering, relevant image/video components of the captured 2D video are selected based on the selected ROIs to compose the 3D video together with the corresponding depth information.

Claims

1. A method, comprising:

capturing a two-dimensional video via one or more image sensors of a monoscopic video sensing device;

capturing corresponding depth information for said captured two-dimensional video via a depth sensor of said monoscopic video sensing device;

selecting one or more regions of interest for said captured two-dimensional video based on said captured corresponding depth information;

selectively processing said captured two-dimensional video and said captured corresponding depth information based on said selected one or more regions of interest; and

composing a three-dimensional video from said processed two-dimensional video and said processed corresponding depth information.

2. The method of claim 1, comprising synchronizing said captured corresponding depth information to said captured two-dimensional video.

3. The method according to claim 2, comprising storing said synchronized depth information as metadata for said captured two-dimensional video.

4. The method according to claim 3, comprising matching resolution of said stored depth information to video resolution of said captured two-dimensional video.

5. The method according to claim 4, comprising interpolating said stored depth information between pixels and/or between video frames to match said video resolution of said captured two-dimensional video.

6. The method according to claim 3, comprising scalable video coding said captured two-dimensional video and said stored depth information into a base layer video and an enhancement layer video, respectively.

7. The method according to claim 6, comprising selectively compressing said captured two-dimensional video and said stored depth information based on said selected one or more regions of interest during said scalable video coding.

8. The method according to claim 7, comprising decompressing said compressed two-dimensional video and said compressed depth information;

and determining relevance of video components within said decompressed two-dimensional video based on said selected one or more regions of interest.

9. The method according to claim 8, comprising composing said three-dimensional video from said decompressed two-dimensional video and said decompressed depth information based on said determined relevance.

10. The method according to claim 9, comprising rendering said composed three-dimensional video.

11. A system for processing signals, the system comprising:

one or more processors and/or circuits for use in a monoscopic video sensing device, said one or more processors and/or circuits comprising one or more image sensors and a depth sensor, wherein said one or more processors and/or circuits are operable to;

capture a two-dimensional video via said one or more image sensors;

capture corresponding depth information for said captured two-dimensional video via said depth sensor;

select one or more regions of interest for said captured two-dimensional video based on said captured corresponding depth information;

selectively process said captured two-dimensional video and said captured corresponding depth information based on said selected one or more regions of interest; and

compose a three-dimensional video from said processed two-dimensional video and said processed corresponding depth information.

12. The system according to claim 11, wherein said one or more circuits are operable to synchronize said captured corresponding depth information to said captured two-dimensional video.

13. The system according to claim 12, wherein said one or more circuits are operable to store said synchronized depth information as metadata for said captured two-dimensional video.

14. The system according to claim 13, wherein said one or more circuits are operable to match resolution of said stored depth information to video resolution of said captured two-dimensional video.

15. The system according to claim 14, wherein said one or more circuits are operable to interpolate said stored depth information between pixels and/or between video frames to match said video resolution of said captured two-dimensional video.

16. The system according to claim 13, wherein said one or more circuits are operable to scalable video code said captured two-dimensional video and said stored depth information into a base layer video and an enhancement layer video, respectively.

17. The system according to claim 16, wherein said one or more circuits are operable to selectively compress said captured two-dimensional video and said stored depth information based on said selected one or more regions of interest during said scalable video coding.

18. The system according to claim 17, wherein said one or more circuits are operable to decompress said compressed two-dimensional video and said compressed depth information;

and determining relevance of video components within said decompressed two-dimensional video based on said selected one or more regions of interest.

19. The system according to claim 18, wherein said one or more circuits are operable to compose said three-dimensional video from said decompressed two-dimensional video and said decompressed depth information based on said determined relevance.

20. The system according to claim 19, wherein said one or more circuits are operable to render said composed three-dimensional video.