Detecting foreground regions in video frames

• US 9,824,455 B1
• Filed: 12/21/2015
• Issued: 11/21/2017
• Est. Priority Date: 12/21/2015
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Singh, Abhishek, Mysore Vijaya Kumar, Rohith, Tyagi, Ambrish
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/976,555
• Time in Patent Office: 701 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/215 Motion-based segmentation

Abstract

Various examples are directed to systems and methods for determining foreground regions in video frames. A computing device may select from a first frame of a video, a plurality of scene point locations and divide the first frame into a plurality of sections. For a first section of the first frame, the computing device may generate a first vector subspace, basis vectors of the first vector subspace are trajectories of scene point locations in the first section. The computing device may determine that a projection error for a first scene point location in the first section is greater than a projection error threshold and write an indication of the first pixel value to a listing of foreground pixel values.

Claims

1. A method of detecting foreground regions in a panoramic video, the method comprising:

receiving, by a computing device, panoramic video data comprising a plurality of panoramic frames and video frame sequence data indicating a video frame sequence for the plurality of panoramic frames, wherein a first panoramic frame of the plurality of panoramic frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values on a two-dimensional grid described by an X-axis and a Y-axis;

selecting, by the computing device and from the first panoramic frame, a plurality of scene point locations positioned on the two-dimensional grid, wherein a first scene point location of the plurality of scene point locations depicts a first scene point from a scene shown by the panoramic video depicted by the panoramic video data, and wherein the first scene point location is at a first position on the two-dimensional grid corresponding to a location of a first pixel value of the first plurality of pixel values;

identifying, by the computing device, a first prior location depicting the first scene point at a second panoramic frame before the first panoramic frame in the video frame sequence;

identifying, by the computing device, a second prior location depicting the first scene point at a third panoramic frame before the second panoramic frame in the panoramic video;

determining, by the computing device, a first trajectory for the first scene point, wherein the first trajectory is a vector comprising a concatenation of a first coordinate set for the first scene point location, a second coordinate set for the first prior location and a third coordinate set for the second prior location;

determining, by the computing device a plurality of columns from the first panoramic frame, wherein the first scene point location is in a first column of the plurality of columns;

randomly selecting, by the computing device, a second scene point location in the first column, a third scene point location in the first column, and a fourth scene point location in the first column;

generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point depicted at the second scene point location, a third trajectory of a third scene point depicted at the third scene point location, and a fourth trajectory of a fourth scene point depicted at the fourth scene point are basis vectors of the first vector subspace;

determining, by the computing device, a first projection error for the first scene point to the first vector subspace;

determining, by the computing device, that the first projection error is greater than a projection error threshold; and

writing, by the computing device, an indication of the first pixel value to a listing of foreground pixel values.

2. The method of claim 1, further comprising generating, by the computing device, a predetermined number of vector subspaces from scene point trajectories of scene points depicted at scene point locations in the first column, wherein the predetermined number of vector subspaces comprises the first vector subspace, and wherein the first vector subspace has the lowest number of scene point locations depicting scene points with scene point trajectories having projection errors less than the projection error threshold.

3. The method of claim 1, further comprising:

determining, by the computing device, a displacement of the first scene point; and

determining, by the computing device, the projection error threshold based at least in part on the displacement of for the first scene point.

4. A computer-implemented system for detecting foreground in a video frame, comprising:

at least one processor; and

a non-transitory computer readable data storage device comprising instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to;

receive video data comprising a plurality of frames and a video frame sequence for the plurality of frames, wherein a first frame of the plurality of frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values in a two-dimensional coordinate system;

select, from the first frame, a plurality of scene point locations in the two-dimensional coordinate system, wherein a first scene point is depicted at a first scene point location of the plurality of scene point locations;

identify a prior scene point location depicting the first scene point at a prior frame that is before the first frame in the video frame sequence;

determine a first trajectory for the first scene point based at least in part on the first scene point location and the prior scene point location;

determine a plurality of sections from the first frame, wherein a set of scene point locations is in a first section of the plurality of sections, and wherein the set of scene point locations includes the first scene point location;

generate a first vector subspace, wherein a second trajectory of a second scene point, a third trajectory of a third scene point, and a fourth trajectory of a fourth scene point are basis vectors of the first vector subspace, and wherein the second scene point, the third scene point, and the fourth scene point are depicted in the first section;

determine a projection error for the first scene point to the first vector subspace;

determine that the projection error is greater than a projection error threshold; and

write an indication of a first pixel value corresponding to the first scene point location to a listing of foreground pixel values.

5. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine a displacement for the first scene point; and

determine the projection error threshold based at least in part on the displacement for the first scene point.

6. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

generate a plurality of trial vector subspaces; and

select the first vector subspace from the plurality of trial vector subspaces based at least in part on a number of scene points depicted at scene point locations selected from the set of scene point locations in the first section that have scene point trajectories with projection errors to the first vector subspace below the projection error threshold.

7. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

randomly select from the set of scene point locations in the first section, a second scene point location depicting the second scene point, a third scene point location depicting the third scene point, and a fourth scene point location depicting the fourth scene point;

randomly select from the set of scene point locations in the first section, a fifth scene point location depicting a fifth scene point, a sixth scene point location depicting a sixth scene point, and a seventh scene point location depicting a seventh scene point;

generate a second vector subspace, wherein a fifth trajectory of the fifth scene point, a sixth trajectory of the sixth scene point, and a seventh trajectory of the seventh scene point are basis vectors of the second vector subspace;

determine a first number of projection errors for scene points depicted by scene point locations of the set of scene point locations in the first section to the first vector subspace that exceed the projection error threshold;

determine a second number of projection errors for scene points depicted by scene point locations of the set of scene point locations in the first section to the second vector subspace that exceed the projection error threshold; and

determine that the second number is larger than the first number.

8. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

generate a second vector subspace, wherein a fifth trajectory of a fifth scene point, a sixth trajectory of a sixth scene point, and a seventh trajectory of a seventh scene point are basis vectors of the second vector subspace, wherein the fifth scene point is depicted at a fifth scene point location in a second section of the plurality of sections, wherein the sixth scene point is depicted at a sixth scene point location in the second section, and wherein the seventh scene point is depicted at a seventh scene point location in the second section;

determine a projection error for an eighth scene point to the second vector subspace, wherein the eighth scene point is depicted at an eighth scene point location in the second section;

determine that the projection error for the eighth scene point is greater than the projection error threshold; and

write an indication of a second pixel value at a location corresponding to the eighth scene point location to the listing of foreground pixel values.

9. The system of claim 4, wherein the first scene point location is also in a second section of the plurality of sections that is adjacent to the first section, and wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

generate a second vector subspace for the second section;

determine a second projection error for the first scene point to the second vector subspace; and

determine that the second projection error is greater than the projection error threshold.

10. The system of claim 4, wherein the plurality of scene point locations are uniformly spaced on the first frame.

11. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to determine a first region selected from the first frame that depicts an object, wherein a density of the plurality of scene point locations is higher at the first region than at a second region of the first frame.

12. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine that a texture level at a first region of the first frame is less than a texture threshold; and

omit scene point locations of the plurality of scene point locations from the first region of the first frame.

13. The system of claim 4, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

identify a plurality of scene point locations, including the first scene point location, that depict scene points with a projection error to the first vector subspace less than the projection error threshold; and

generate a foreground region of the first frame, wherein the foreground region comprises the plurality of scene point locations.

14. The system of claim 13, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to generate a binary mask comprising a plurality of binary mask pixel values, wherein a first binary mask pixel value corresponding to the first scene point location is asserted and a second binary mask pixel value not corresponding to any of the plurality of scene point locations is un-asserted.

15. A computer-implemented method for detecting foreground in a video frame, comprising:

receiving, by a computing device, video data comprising a plurality of frames and a video frame sequence for the plurality of frames, wherein a first frame of the plurality of frames comprises a first plurality of pixel values and first spatial data describing a spatial arrangement of the first plurality of pixel values in a two-dimensional coordinate system;

selecting, by the computing device and from the first frame, a plurality of scene point locations in the two-dimensional coordinate system, wherein a first scene point is depicted at a first scene point location of the plurality of scene point locations;

identifying, by the computing device, a prior scene point location depicting the first scene point at a prior frame that is before the first frame in the video frame sequence;

determining, by the computing device, a first trajectory for the first scene point based at least in part on the first scene point location and the prior scene point location;

determining, by the computing device, a plurality of sections from the first frame, wherein a set of scene point locations are in a first section of the plurality of sections, and wherein the set of scene point locations includes the first scene point location;

generating, by the computing device, a first vector subspace, wherein a second trajectory of a second scene point, a third trajectory of a third scene point, and a fourth trajectory of a fourth scene point are basis vectors of the first vector subspace, and wherein the second scene point, the third scene point, and the fourth scene point are depicted in the first section;

determining, by the computing device, a projection error for the first scene point to the first vector subspace;

determining, by the computing device, that the projection error is greater than a projection error threshold; and

writing, by the computing device, an indication of a first pixel value corresponding to the first scene point location to a listing of foreground pixel values.

16. The method of claim 15, further comprising:

generating, by the computing device, a plurality of trial vector subspaces; and

selecting, by the computing device, the first vector subspace from the plurality of trial vector subspaces based at least in part on a number of scene points depicted at scene point locations selected from the set of scene point locations in the first section that have scene point trajectories with projection errors to the first vector subspace below the projection error threshold.

17. The method of claim 15, wherein the first scene point location is also in a second section of the plurality of sections that is adjacent to the first section, further comprising:

generating, by the computing device, a second vector subspace for the second section;

determining, by the computing device, a second projection error for the first scene point to the second vector subspace; and

determining, by the computing device, that the second projection error is greater than the projection error threshold.

18. The method of claim 15, further comprising:

determining, by the computing device, a displacement for the first scene point; and

determining, by the computing device, the projection error threshold based at least in part on the displacement for the first scene point.

19. The method of claim 15, further comprising

generating, by the computing device, a second vector subspace, wherein a fifth trajectory of a fifth scene point, a sixth trajectory of a sixth scene point, and a seventh trajectory of a seventh scene point are basis vectors of the second vector subspace, wherein the fifth scene point is depicted at a fifth scene point location in a second section of the plurality of sections, wherein the sixth scene point is depicted at a sixth scene point location in the second section, and wherein the seventh scene point is depicted at a seventh scene point location in the second section;

determining, by the computing device, a projection error for an eighth scene point to the second vector subspace, wherein the eighth scene point is depicted at an eighth scene point location in the second section;

determining, by the computing device, that the projection error for the eighth scene point is greater than the projection error threshold; and

writing, by the computing device, an indication of a second pixel value at a location on the two-dimensional coordinate system corresponding to the eighth scene point location to the listing of foreground pixel values.

20. The method of claim 15, wherein the plurality of scene point locations are uniformly spaced on the first frame.