Multi-view scene segmentation and propagation

• US 10,275,892 B2
• Filed: 03/17/2017
• Issued: 04/30/2019
• Est. Priority Date: 06/09/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Bleibel, Francois, Du, Tingfang, Nonn, Thomas, Tan, Jie
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US15/462,752
• Publication Number:
  US 20170358092A1
• Time in Patent Office: 774 Days
• Field of Search: None
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10052 Images from lightfield camera
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A depth-based effect may be applied to a multi-view video stream to generate a modified multi-view video stream. User input may designate a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream. Based on the user input, a reference mask may be generated to indicate the foreground region and the background region. The reference mask may be used to generate one or more other masks that indicate the foreground and background regions for one or more different images, from different frames and/or different views from the reference image. The reference mask and other mask(s) may be used to apply the effect to the multi-view video stream to generate the modified multi-view video stream.

Claims

1. A method for applying an effect to a multi-view video stream, the method comprising:

in a data store, receiving the video stream;

in an input device, receiving first user input designating a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream;

in a processor, based on the first user input, generating a reference mask that indicates the foreground region and the background region;

in the processor, using the reference mask to generate a first non-reference mask that indicates the foreground region and the background region for a first non-reference image, different from the reference image, of the video stream;

in the processor, generating a modified video stream by using the reference mask to apply the effect to the reference image and using the first non-reference mask to apply the effect to the first non-reference image; and

in a display device, displaying the modified video stream as a virtual reality or augmented reality experience.

2. The method of claim 1, further comprising, in a camera, capturing the video stream prior to receiving the video stream in the data store.

3. The method of claim 1, wherein:

the reference image is from a reference frame of the video stream; and

the first non-reference image is from a first frame, different from the reference frame, of the video stream.

4. The method of claim 3, further comprising, in the processor, using the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second frame, different from the reference frame and the first frame, of the video stream;

wherein generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

5. The method of claim 1, wherein:

the reference image is from a reference view of a reference frame of the video stream; and

the first non-reference image is from a first view, different from the reference view, of the reference frame of the video stream.

6. The method of claim 5, further comprising, in the processor, using the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second view, different from the reference view and the first view, of the reference frame of the video stream;

wherein generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

7. The method of claim 1, further comprising, in the input device, prior to receiving the first user input, receiving second user input designating the reference image from among a plurality of images of the video stream.

8. The method of claim 1, wherein generating the reference mask comprises:

using the boundary to compute an initial binary segmentation designating the foreground region and the background region of the reference image;

refining the boundary to delineate a refined boundary that more accurately designates the foreground region and the background region; and

using the refined boundary to generate the reference mask.

9. The method of claim 8, wherein using the refined boundary to generate the reference mask comprises using the refined boundary to generate a trimap that further indicates an unknown region between the foreground region and the background region, the unknown region containing pixels that may belong to the foreground region or the background region.

10. The method of claim 9, further comprising, in the processor, applying a matting algorithm to the trimap to obtain an alpha matte comprising alpha values for the pixels in the foreground region, the background region, and the unknown region.

11. The method of claim 10, wherein the video stream comprises light-field video captured by one or more light-field cameras.

12. The method of claim 11, further comprising back-projecting the alpha matte and a foreground image of the foreground region to the light-field video.

13. The method of claim 9, wherein using the reference mask to generate the first non-reference mask comprises:

designating a first non-reference boundary between a first non-reference segment and a second non-reference segment of the first non-reference image;

using the first non-reference boundary to compute an initial non-reference binary segmentation designating the first non-reference segment and the second non-reference segment of the first non-reference image;

refining the first non-reference boundary to delineate a refined non-reference boundary that more accurately designates the first non-reference segment and the second non-reference segment; and

using the refined non-reference boundary to generate a non-reference trimap that further indicates an unknown non-reference segment between the first non-reference segment and the second non-reference segment, the unknown non-reference segment containing non-reference pixels that may belong to the first non-reference segment or the second non-reference segment.

14. The method of claim 13, further comprising, in the processor, applying a matting algorithm to the non-reference trimap to obtain a non-reference alpha matte comprising non-reference alpha values for the non-reference pixels in the unknown non-reference segment, to generate a non-reference foreground image of the foreground region.

15. A non-transitory computer-readable medium for applying an effect to a multi-view video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a data store to receive the video stream;

causing an input device to receive first user input designating a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream;

based on the first user input, generating a reference mask that indicates the foreground region and the background region;

using the reference mask to generate a first non-reference mask that indicates the foreground region and the background region for a first non-reference image, different from the reference image, of the video stream;

generating a modified video stream by using the reference mask to apply the effect to the reference image and using the first non-reference mask to apply the effect to the first non-reference image; and

causing a display device to display the modified video stream as a virtual reality or augmented reality experience.

16. The non-transitory computer-readable medium of claim 15, further comprising instructions stored thereon, that when executed by a processor, cause a camera to capture the video stream prior to receiving the video stream in the data store.

17. The non-transitory computer-readable medium of claim 15, wherein:

the reference image is from a reference frame of the video stream;

the first non-reference image is from a first frame, different from the reference frame, of the video stream;

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, uses the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second frame, different from the reference frame and the first frame, of the video stream; and

generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

18. The non-transitory computer-readable medium of claim 15, wherein:

the video stream comprises a multi-view video stream;

the reference image is from a reference view of a reference frame of the video stream;

the first non-reference image is from a first view, different from the reference view, of the reference frame of the video stream;

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, use the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second view, different from the reference view and the first view, of the reference frame of the video stream; and

generating the modified video stream further comprises using the second non-reference mask to apply the effect to the second non-reference image.

19. The non-transitory computer-readable medium of claim 15, further comprising instructions stored thereon, that when executed by a processor, cause the input device, prior to receiving the first user input, to receive second user input designating the reference image from among a plurality of images of the video stream.

20. The non-transitory computer-readable medium of claim 15, wherein generating the reference mask comprises:

using the boundary to compute an initial binary segmentation designating the foreground region and the background region of the reference image;

refining the boundary to delineate a refined boundary that more accurately designates the foreground region and the background region; and

using the refined boundary to generate the reference mask;

wherein using the refined boundary to generate the reference mask comprises using the refined boundary to generate a trimap that further indicates an unknown region between the foreground region and the background region, the unknown region containing pixels that may belong to the foreground region or the background region.

21. The non-transitory computer-readable medium of claim 20, further comprising instructions stored thereon, that when executed by a processor, apply a matting algorithm to the trimap to obtain an alpha matte comprising alpha values for the pixels in the foreground region, the background region, and the unknown region.

22. The non-transitory computer-readable medium of claim 21, wherein:

the video stream comprises light-field video captured by one or more light-field cameras; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, back-project the alpha matte and a foreground image of the foreground region to the light-field video.

23. The non-transitory computer-readable medium of claim 20, wherein using the reference mask to generate the first non-reference mask comprises:

designating a first non-reference boundary between a first non-reference segment and a second non-reference segment of the first non-reference image;

using the first non-reference boundary to compute an initial non-reference binary segmentation designating the first non-reference segment and the second non-reference segment of the first non-reference image;

refining the first non-reference boundary to delineate a refined non-reference boundary that more accurately designates the first non-reference segment and the second non-reference segment; and

using the refined non-reference boundary to generate a non-reference trimap that further indicates an unknown non-reference segment between the first non-reference segment and the second non-reference segment, the unknown non-reference segment containing non-reference pixels that may belong to the first non-reference segment or the second non-reference segment.

24. The non-transitory computer-readable medium of claim 23, further comprising instructions stored thereon, that when executed by a processor, apply a matting algorithm to the non-reference trimap to obtain a non-reference alpha matte comprising non-reference alpha values for the non-reference pixels in the unknown non-reference segment, to generate a non-reference foreground image of the foreground region.

25. A system for applying an effect to a multi-view video stream, the system comprising:

a data store configured to receive the video stream;

an input device configured to receive first user input designating a boundary between a foreground region and a background region, at a different depth from the foreground region, of a reference image of the video stream;

a processor, communicatively coupled to the data store and the input device, configured to;

based on the first user input, generate a reference mask that indicates the foreground region and the background region;

use the reference mask to generate a first non-reference mask that indicates the foreground region and the background region for a first non-reference image, different from the reference image, of the video stream; and

generate a modified video stream by using the reference mask to apply the effect to the reference image and using the first non-reference mask to apply the effect to the first non-reference image; and

a display device configured to display the modified video stream as a virtual reality or augmented reality experience.

26. The system of claim 25, further comprising a camera configured to capture the video stream prior to receiving the video stream in the data store.

27. The system of claim 25, wherein:

the reference image is from a reference frame of the video stream;

the first non-reference image is from a first frame, different from the reference frame, of the video stream;

the processor is further configured to use the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second frame, different from the reference frame and the first frame, of the video stream; and

the processor is further configured to generate the modified video stream by using the second non-reference mask to apply the effect to the second non-reference image.

28. The system of claim 25, wherein:

the reference image is from a reference view of a reference frame of the video stream;

the first non-reference image is from a first view, different from the reference view, of the reference frame of the video stream;

the processor is further configured to use the first non-reference mask to generate a second non-reference mask that indicates the foreground region and the background region for a second non-reference image, from a second view, different from the reference view and the first view, of the reference frame of the video stream; and

the processor is further configured to generate the modified video stream by using the second non-reference mask to apply the effect to the second non-reference image.

29. The system of claim 25, wherein the input device is further configured to, prior to receiving the first user input, receive second user input designating the reference image from among a plurality of images of the video stream.

30. The system of claim 25, wherein:

the processor is further configured to generate the reference mask by;

using the boundary to compute an initial binary segmentation designating the foreground region and the background region of the reference image;

refining the boundary to delineate a refined boundary that more accurately designates the foreground region and the background region; and

using the refined boundary to generate the reference mask; and

the processor is further configured to use the refined boundary to generate the reference mask by using the refined boundary to generate a trimap that further indicates an unknown region between the foreground region and the background region, the unknown region containing pixels that may belong to the foreground region or the background region.

31. The system of claim 30, wherein the processor is further configured to apply a matting algorithm to the trimap to obtain an alpha matte comprising alpha values for the pixels in the foreground region, the background region, and the unknown region.

32. The system of claim 31, wherein:

the video stream comprises light-field video captured by one or more light-field cameras; and

the processor is further configured to back-project the alpha matte and a foreground image of the foreground region to the light-field video.

33. The system of claim 30, wherein the processor is further configured to use the reference mask to generate the first non-reference mask by:

designating a first non-reference boundary between a first non-reference segment and a second non-reference segment of the first non-reference image;

using the first non-reference boundary to compute an initial non-reference binary segmentation designating the first non-reference segment and the second non-reference segment of the first non-reference image;

refining the first non-reference boundary to delineate a refined non-reference boundary that more accurately designates the first non-reference segment and the second non-reference segment; and

using the refined non-reference boundary to generate a non-reference trimap that further indicates an unknown non-reference segment between the first non-reference segment and the second non-reference segment, the unknown non-reference segment containing non-reference pixels that may belong to the first non-reference segment or the second non-reference segment.

34. The system of claim 33, wherein the processor is further configured to apply a matting algorithm to the non-reference trimap to obtain a non-reference alpha matte comprising non-reference alpha values for the non-reference pixels in the unknown non-reference segment to generate a non-reference foreground image of the foreground region.