Detecting foreground regions in video frames
First Claim
1. A method of detecting foreground regions in a panoramic video, the method comprising:
- receiving, by a computing device, panoramic video data comprising a plurality of panoramic frames and video frame sequence data indicating a video frame sequence for the plurality of panoramic frames, wherein a first panoramic frame of the plurality of panoramic frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values on a two-dimensional grid described by an X-axis and a Y-axis;
selecting, by the computing device and from the first panoramic frame, a plurality of scene point locations positioned on the two-dimensional grid, wherein a first scene point location of the plurality of scene point locations depicts a first scene point from a scene shown by the panoramic video depicted by the panoramic video data, and wherein the first scene point location is at a first position on the two-dimensional grid corresponding to a location of a first pixel value of the first plurality of pixel values;
identifying, by the computing device, a first prior location depicting the first scene point at a second panoramic frame before the first panoramic frame in the video frame sequence;
identifying, by the computing device, a second prior location depicting the first scene point at a third panoramic frame before the second panoramic frame in the panoramic video;
determining, by the computing device, a first trajectory for the first scene point, wherein the first trajectory is a vector comprising a concatenation of a first coordinate set for the first scene point location, a second coordinate set for the first prior location and a third coordinate set for the second prior location;
determining, by the computing device a plurality of columns from the first panoramic frame, wherein the first scene point location is in a first column of the plurality of columns;
randomly selecting, by the computing device, a second scene point location in the first column, a third scene point location in the first column, and a fourth scene point location in the first column;
generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point depicted at the second scene point location, a third trajectory of a third scene point depicted at the third scene point location, and a fourth trajectory of a fourth scene point depicted at the fourth scene point are basis vectors of the first vector subspace;
determining, by the computing device, a first projection error for the first scene point to the first vector subspace;
determining, by the computing device, that the first projection error is greater than a projection error threshold; and
writing, by the computing device, an indication of the first pixel value to a listing of foreground pixel values.
1 Assignment
0 Petitions
Accused Products
Abstract
Various examples are directed to systems and methods for determining foreground regions in video frames. A computing device may select from a first frame of a video, a plurality of scene point locations and divide the first frame into a plurality of sections. For a first section of the first frame, the computing device may generate a first vector subspace, basis vectors of the first vector subspace are trajectories of scene point locations in the first section. The computing device may determine that a projection error for a first scene point location in the first section is greater than a projection error threshold and write an indication of the first pixel value to a listing of foreground pixel values.
-
Citations
20 Claims
-
1. A method of detecting foreground regions in a panoramic video, the method comprising:
-
receiving, by a computing device, panoramic video data comprising a plurality of panoramic frames and video frame sequence data indicating a video frame sequence for the plurality of panoramic frames, wherein a first panoramic frame of the plurality of panoramic frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values on a two-dimensional grid described by an X-axis and a Y-axis; selecting, by the computing device and from the first panoramic frame, a plurality of scene point locations positioned on the two-dimensional grid, wherein a first scene point location of the plurality of scene point locations depicts a first scene point from a scene shown by the panoramic video depicted by the panoramic video data, and wherein the first scene point location is at a first position on the two-dimensional grid corresponding to a location of a first pixel value of the first plurality of pixel values; identifying, by the computing device, a first prior location depicting the first scene point at a second panoramic frame before the first panoramic frame in the video frame sequence; identifying, by the computing device, a second prior location depicting the first scene point at a third panoramic frame before the second panoramic frame in the panoramic video; determining, by the computing device, a first trajectory for the first scene point, wherein the first trajectory is a vector comprising a concatenation of a first coordinate set for the first scene point location, a second coordinate set for the first prior location and a third coordinate set for the second prior location; determining, by the computing device a plurality of columns from the first panoramic frame, wherein the first scene point location is in a first column of the plurality of columns; randomly selecting, by the computing device, a second scene point location in the first column, a third scene point location in the first column, and a fourth scene point location in the first column; generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point depicted at the second scene point location, a third trajectory of a third scene point depicted at the third scene point location, and a fourth trajectory of a fourth scene point depicted at the fourth scene point are basis vectors of the first vector subspace; determining, by the computing device, a first projection error for the first scene point to the first vector subspace; determining, by the computing device, that the first projection error is greater than a projection error threshold; and writing, by the computing device, an indication of the first pixel value to a listing of foreground pixel values. - View Dependent Claims (2, 3)
-
-
4. A computer-implemented system for detecting foreground in a video frame, comprising:
- at least one processor; and
a non-transitory computer readable data storage device comprising instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to;
receive video data comprising a plurality of frames and a video frame sequence for the plurality of frames, wherein a first frame of the plurality of frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values in a two-dimensional coordinate system;
select, from the first frame, a plurality of scene point locations in the two-dimensional coordinate system, wherein a first scene point is depicted at a first scene point location of the plurality of scene point locations;
identify a prior scene point location depicting the first scene point at a prior frame that is before the first frame in the video frame sequence;
determine a first trajectory for the first scene point based at least in part on the first scene point location and the prior scene point location;
determine a plurality of sections from the first frame, wherein a set of scene point locations is in a first section of the plurality of sections, and wherein the set of scene point locations includes the first scene point location;
generate a first vector subspace, wherein a second trajectory of a second scene point, a third trajectory of a third scene point, and a fourth trajectory of a fourth scene point are basis vectors of the first vector subspace, and wherein the second scene point, the third scene point, and the fourth scene point are depicted in the first section;
determine a projection error for the first scene point to the first vector subspace;
determine that the projection error is greater than a projection error threshold; and
write an indication of a first pixel value corresponding to the first scene point location to a listing of foreground pixel values. - View Dependent Claims (5, 6, 7, 8, 9, 10, 11, 12, 13, 14)
- at least one processor; and
-
15. A computer-implemented method for detecting foreground in a video frame, comprising:
- receiving, by a computing device, video data comprising a plurality of frames and a video frame sequence for the plurality of frames, wherein a first frame of the plurality of frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values in a two-dimensional coordinate system;
selecting, by the computing device and from the first frame, a plurality of scene point locations in the two-dimensional coordinate system, wherein a first scene point is depicted at a first scene point location of the plurality of scene point locations;
identifying, by the computing device, a prior scene point location depicting the first scene point at a prior frame that is before the first frame in the video frame sequence;
determining, by the computing device, a first trajectory for the first scene point based at least in part on the first scene point location and the prior scene point location;
determining, by the computing device, a plurality of sections from the first frame, wherein a set of scene point locations are in a first section of the plurality of sections, and wherein the set of scene point locations includes the first scene point location;
generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point, a third trajectory of a third scene point, and a fourth trajectory of a fourth scene point are basis vectors of the first vector subspace, and wherein the second scene point, the third scene point, and the fourth scene point are depicted in the first section;
determining, by the computing device, a projection error for the first scene point to the first vector subspace;
determining, by the computing device, that the projection error is greater than a projection error threshold; and
writing, by the computing device, an indication of a first pixel value corresponding to the first scene point location to a listing of foreground pixel values. - View Dependent Claims (16, 17, 18, 19, 20)
- receiving, by a computing device, video data comprising a plurality of frames and a video frame sequence for the plurality of frames, wherein a first frame of the plurality of frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values in a two-dimensional coordinate system;
Specification