Multi-view rotoscope contour propagation

• US 10,679,361 B2
• Filed: 12/05/2017
• Issued: 06/09/2020
• Est. Priority Date: 12/05/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Karnad, Nikhil
• Primary Examiner(s): Le, Sarah
• Application Number:
  US15/831,770
• Publication Number:
  US 20180158198A1
• Time in Patent Office: 917 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/251 involving models
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: H04N 19/157 Assigned coding mode, i.e. the coding mode being predefined or preselected to be further used for selection of another element or parameter

Abstract

A video stream may be captured, and may have a plurality of frames including at least a first frame and a second frame. Each of the frames may have a plurality of views obtained from viewpoints that are offset from each other. A source contour, associated with a source view of the first frame, may be retrieved. Camera parameters, associated with the image capture device used to capture the video stream, may also be retrieved. The camera parameters may include a first offset between the source view and a destination view of the first frame. At least the first offset may be used to project the source contour to the destination view to generate a destination contour associated with the destination view.

Claims

1. A method for processing a video stream, the method comprising:

at an image capture device, capturing a video stream such that the video stream comprises a plurality of frames comprising at least a first frame and a second frame, wherein each of the frames comprises a plurality of views obtained from viewpoints that are offset from each other;

from a data store, retrieving a source contour representing a contour drawn in a source view of the first frame;

from the data store, retrieving camera parameters associated with the image capture device, the camera parameters comprising a first offset between the source view and a destination view of the first frame; and

at a processor, using at least the first offset to project the source contour to the destination view to generate a destination contour associated with the destination view.

2. The method of claim 1, wherein:

the source contour comprises a first closed shape defining a first interior within the first closed shape and a first exterior outside the first closed shape; and

the destination contour comprises a second closed shape defining a second interior within the second closed shape and a second exterior out-side the second closed shape;

and wherein the method further comprises;

applying an image processing technique to the source view such that the image processing technique is applied differently to the first interior than to the first exterior; and

applying the image processing technique to the destination view such that the image processing technique is applied differently to the second interior than to the second exterior.

3. The method of claim 1, further comprising, at an input device, receiving first user input designating the source contour.

4. The method of claim 3, further comprising, at the input device, receiving second user input modifying the destination contour.

5. The method of claim 1, wherein:

the camera parameters further comprise a plurality of additional offsets be-tween the source view and a plurality of additional destination views of the first frame; and

the method further comprises, at the processor, using at least the additional offsets to project the source contour to each of the additional destination views to generate a plurality of additional destination contours, each of which is associated with one of the additional destination views.

6. The method of claim 1, wherein using at least the first offset to project the source contour to the destination view comprises, for each source point of a plurality of source points on the source contour:

designating a source window of the source view containing the source point;

identifying one or more source features of the source view within the source window;

using the first offset to designate a destination window of the destination view that is believed to contain a destination point on the destination contour, the destination point corresponding to the source point;

identifying one or more destination features, corresponding to the source features, within the destination window; and

using the destination features to locate the destination point within the destination window.

7. The method of claim 6, wherein identifying the source features and identifying the destination features comprises applying at least one image analysis technique selected from the group consisting of:

photometric matching;

gradient magnitude analysis;

foreground/background modeling; and

shape consistency/connectedness analysis.

8. The method of claim 6, wherein identifying the source features and identifying the destination features comprises assessing one or more metrics selected from the group consisting of:

a photometric similarity score;

an object segmentation score; and

a snap-to-edge score.

9. The method of claim 8, wherein:

assessing one or more of the metrics comprises assessing a plurality of the metrics; and

identifying the source features and identifying the destination features further comprises;

computing a weighted average of the metrics; and

using the weighted average of the metrics to determine that the destination features are similar to the source features.

10. The method of claim 6, wherein:

capturing the video stream comprises;

capturing source depth information indicative of depths of objects in the source view;

capturing destination depth information indicative of depths of objects in the destination view; and

using at least the first offset to project the source contour to the destination view comprises further using the source depth information and the destination depth information to project the source contour to the destination view.

11. The method of claim 10, wherein using the source depth information and the destination depth information to project the source contour to the destination view comprises:

using the source depth information for the source point to un-project the source point from 2D space to 3D space; and

using the camera parameters to re-project the source point back to the destination view as the destination point.

12. The method of claim 1, wherein:

the image capture device comprises a tiled camera array comprising a plurality of cameras, each of which captures one of the views of the first frame; and

the method further comprises calibrating the image capture device to obtain the camera parameters.

13. The method of claim 12, wherein the camera parameters comprise a plurality of 3D transformations usable to transform a first position and/or orientation of each of the cameras to a second position and/or orientation of any other of the cameras.

14. The method of claim 13, wherein the 3D transformations comprise an epipolar line that provides a 1D line across the destination view, corresponding to the source view.

15. The method of claim 1, wherein:

the method further comprises, from the data store, retrieving one or more additional source contours, each of which is associated with one of one or more additional source views of the first frame;

the camera parameters further comprise one or more additional offsets between the one or more additional source views and the destination view; and

using at least the first offset to project the source contour to the destination view further comprises using the one or more additional offsets to generate the destination contour.

16. A non-transitory computer-readable medium for processing a video stream, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

from a data store, retrieving at least part of a video stream captured by an image capture device, the video stream comprising a plurality of frames comprising at least a first frame and a second frame, wherein each of the frames comprises a plurality of views obtained from viewpoints that are offset from each other;

from the data store, retrieving a source contour representing a contour drawn in a source view of the first frame;

from the data store, retrieving camera parameters associated with the image capture device, the camera parameters comprising a first offset between the source view and a destination view of the first frame; and

using at least the first offset to project the source contour to the destination view to generate a destination contour associated with the destination view.

17. The non-transitory computer-readable medium of claim 16, wherein:

the source contour comprises a first closed shape defining a first interior within the first closed shape and a first exterior outside the first closed shape; and

the destination contour comprises a second closed shape defining a second interior within the second closed shape and a second exterior outside the second closed shape;

and wherein the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, perform the steps of;

applying an image processing technique to the source view such that the image processing technique is applied differently to the first interior than to the first exterior; and

applying the image processing technique to the destination view such that the image processing technique is applied differently to the second interior than to the second exterior.

18. The non-transitory computer-readable medium of claim 16, further comprising instructions stored thereon, that when executed by a processor, cause an input device to receive first user input designating the source contour.

19. The non-transitory computer-readable medium of claim 18, further comprising instructions stored thereon, that when executed by a processor, cause the input device to receive second user input modifying the destination contour.

20. The non-transitory computer-readable medium of claim 16, wherein:

the camera parameters further comprise a plurality of additional offsets between the source view and a plurality of additional destination views of the first frame; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, use at least the additional offsets to project the source contour to each of the additional destination views to generate a plurality of additional destination contours, each of which is associated with one of the additional destination views.

21. The non-transitory computer-readable medium of claim 16, wherein using at least the first offset to project the source contour to the destination view comprises, for each source point of a plurality of source points on the source contour:

designating a source window of the source view containing the source point;

identifying one or more source features of the source view within the source window;

using the first offset to designate a destination window of the destination view that is believed to contain a destination point on the destination contour, the destination point corresponding to the source point;

identifying one or more destination features, corresponding to the source features, within the destination window; and

using the destination features to locate the destination point within the destination window.

22. The non-transitory computer-readable medium of claim 21, wherein:

capturing the video stream comprises;

capturing source depth information indicative of depths of objects in the source view;

capturing destination depth information indicative of depths of objects in the destination view; and

using at least the first offset to project the source contour to the destination view comprises further using the source depth information and the destination depth information to project the source contour to the destination view by;

using the source depth information for the source point to un-project the source point from 2D space to 3D space; and

using the camera parameters to re-project the source point back to the destination view as the destination point.

23. The non-transitory computer-readable medium of claim 16, wherein:

the image capture device comprises a tiled camera array comprising a plurality of cameras, each of which captures one of the views of the first frame; and

the non-transitory computer-readable medium further comprises instructions stored thereon, that when executed by a processor, calibrate the image capture device to obtain the camera parameters; and

the camera parameters comprise a plurality of 3D transformations usable to transform a first position and/or orientation of each of the cameras to a second position and/or orientation of any other of the cameras.

24. A system for processing a video stream, the system comprising:

a data store configured to;

store at least part of a video stream captured by an image capture device, the video stream comprising a plurality of frames comprising at least a first frame and a second frame, wherein each of the frames comprises a plurality of views obtained from viewpoints that are offset from each other;

retrieve a source contour representing a contour drawn in a source view of the first frame; and

retrieve camera parameters associated with the image capture device, the camera parameters comprising a first offset between the source view and a destination view of the first frame; and

a processor configured to use at least the first offset to project the source contour to the destination view to generate a destination contour associated with the destination view.

25. The system of claim 24, wherein:

the source contour comprises a first closed shape defining a first interior within the first closed shape and a first exterior outside the first closed shape; and

the destination contour comprises a second closed shape defining a second interior within the second closed shape and a second exterior out-side the second closed shape;

and wherein the processor is further configured to;

apply an image processing technique to the source view such that the image processing technique is applied differently to the first interior than to the first exterior; and

apply the image processing technique to the destination view such that the image processing technique is applied differently to the second interior than to the second exterior.

26. The system of claim 24, further comprising an input device configured to receive first user input designating the source contour.

27. The system of claim 26, wherein the input device is further configured to receive second user input modifying the destination contour.

28. The system of claim 24, wherein:

the camera parameters further comprise a plurality of additional offsets be-tween the source view and a plurality of additional destination views of the first frame; and

the processor is further configured to use at least the additional offsets to project the source contour to each of the additional destination views to generate a plurality of additional destination contours, each of which is associated with one of the additional destination views.

29. The system of claim 24, wherein the processor is further configured to use at least the first offset to project the source contour to the destination view by, for each source point of a plurality of source points on the source contour:

designating a source window of the source view containing the source point;

identifying one or more source features of the source view within the source window;

using the first offset to designate a destination window of the destination view that is believed to contain a destination point on the destination contour, the destination point corresponding to the source point;

identifying one or more destination features, corresponding to the source features, within the destination window; and

using the destination features to locate the destination point within the destination window.

30. The system of claim 29, further comprising the image capture device, wherein the image capture device is configured to capture the video stream.

31. The system of claim 30, wherein:

the image capture device is further configured to capture the video stream by;

capturing source depth information indicative of depths of objects in the source view;

capturing destination depth information indicative of depths of objects in the destination view; and

using at least the first offset to project the source contour to the destination view comprises further using the source depth information and the destination depth information to project the source contour to the destination view; and

the processor is further configured to use the source depth information and the destination depth information to project the source contour to the destination view by;

using the source depth information for the source point to un-project the source point from 2D space to 3D space; and

using the camera parameters to re-project the source point back to the destination view as the destination point.

32. The system of claim 30, wherein:

the image capture device comprises a tiled camera array comprising a plurality of cameras, each of which captures one of the views of the first frame;

the processor is further configured to calibrate the image capture device to obtain the camera parameters; and

the camera parameters comprise a plurality of 3D transformations usable to transform a first position and/or orientation of each of the cameras to a second position and/or orientation of any other of the cameras.