Detecting foreground regions in panoramic video frames

• US 9,842,402 B1
• Filed: 12/21/2015
• Issued: 12/12/2017
• Est. Priority Date: 12/21/2015
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Mysore Vijaya Kumar, Rohith, Singh, Abhishek, Tyagi, Ambrish
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US14/976,460
• Time in Patent Office: 722 Days
• Field of Search: None
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 1/00 Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Various examples are directed to systems and methods for detecting regions in video frames. For example, a computing device may receive a video comprising a plurality of frames and a video frame sequence of the plurality of frames. The computing device may select a plurality of scene point location from a first frame. The computing device may determine a plurality of columns in the first frame and fit a first sinusoidal function to a distribution of average column Y-axis displacements for the plurality of columns by column position. The computing device may determine a first difference based at least in part on the first scene point Y-axis displacement and an output of the first sinusoidal function at the X-axis position of the first scene point and determine that the first difference is greater than a threshold distance.

Claims

1. A method of detecting foreground regions in a panoramic frame, the method comprising:

receiving, by a computing device, panoramic video data comprising a plurality of panoramic frames and video frame sequence data indicating a video frame sequence for the plurality of panoramic frames, wherein a subject panoramic frame of the plurality of panoramic frames comprises a plurality of pixel values and spatial data describing a spatial arrangement of the plurality of pixel values on a two-dimensional grid described by an X-axis and a Y-axis;

selecting, by the computing device and from the subject panoramic frame, a plurality of scene point locations positioned on the two-dimensional grid, wherein a first scene point location of the plurality of scene point locations depicts a first scene point from a scene shown by a panoramic video depicted by the panoramic video data, and wherein the first scene point location is positioned on the two-dimensional grid at a first location corresponding to a location of a first pixel value of the plurality of pixel values;

identifying, by the computing device, a second scene point location depicting the first scene point at a second panoramic frame that is before the subject panoramic frame in the video frame sequence;

determining, by the computing device, a Y-axis displacement for the first scene point by determining a distance on the Y-axis between the second scene point location and the first scene point location;

determining, by the computing device, a plurality of columns from the subject panoramic frame, where each of the plurality of columns is characterized by a column position on the X-axis, and wherein the first scene point location is in a first column of the plurality of columns;

determining, by the computing device, an average Y-axis displacement for scene point locations from the plurality of scene point locations in the first column;

fitting, by the computing device, a sinusoidal function to a distribution of average Y-axis displacements of scene points by column position on the X-axis, wherein the sinusoidal function maps position on the X-axis to Y-axis displacement;

determining, by the computing device, that the Y-axis displacement of the first scene point differs from an output of the sinusoidal function at an X-axis position of the first scene point by more than a threshold distance; and

writing, by the computing device, an indication of the first pixel value to a listing of foreground pixel values.

2. The method of claim 1, further comprising:

determining a Fourier transform of the distribution of average Y-axis displacements of scene points by column position on the X-axis; and

determining at least three terms of a Fourier series based at least in part on the Fourier transform, wherein the sinusoidal function comprises the at least three terms of the Fourier series.

3. The method of claim 1, further comprising,

determining, by the computing device, an X-axis displacement for the position of the first scene point by determining a distance on the X-axis between the second scene point location and the first scene point location;

determining, by the computing device, an average X-axis displacement for scene point locations from the plurality of scene point locations that are in the first column; and

fitting, by the computing device, a second sinusoidal function to a distribution of average X-axis displacements of scene point locations by column position on the X-axis, wherein the second sinusoidal function maps position on the X-axis to X-axis displacement.

4. The method of claim 3, further comprising determining, by the computing device, that the X-axis displacement of the first scene point differs from an output of the second sinusoidal function at the X-axis position of the first scene point by more than a second threshold distance.

5. A computer-implemented system for detecting foreground in a video frame, comprising:

at least one processor; and

a data storage device comprising instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to;

receive video data comprising a plurality of frames and a video frame sequence of the plurality of frames;

from a first frame of the plurality of frames, select a plurality of scene point locations including a first scene point location depicting a first scene point, wherein the first scene point location is at a first position on a first axis of the first frame and a second position on a second axis of the first frame;

from a second frame of the plurality of frames, identify a second scene point location depicting the first scene point;

determine a first scene point displacement on the second axis between the first scene point location and the second scene point location;

determine a plurality of columns in the first frame, wherein the plurality of columns comprises a first column having a first column position on the first axis of the first frame, and wherein the first scene point location is in the first column;

determine an average column second axis displacement for scene points depicted in the first column;

selecting a first sinusoidal function based at least in part on a distribution of average column second axis displacements for the plurality of columns by column position;

determine a first difference based at least in part on the first scene point displacement and an output of the first sinusoidal function at the first position on the first axis of the first scene point; and

determine that the first difference is greater than a threshold distance.

6. The system of claim 5, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine a first scene point first axis displacement based at least in part on the first scene point location and the second scene point location;

determine an average column first axis displacement for scene points depicted in the first column; and

select a second sinusoidal function based at least in part on a distribution of average column first axis displacements for the plurality of columns by column position.

7. The system of claim 6, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine a second difference based at least in part on the first scene point first axis displacement and an output of the second sinusoidal function at the first position on the first axis of the first scene point;

determine a square root of a sum of;

a square of the first difference; and

a square of the second difference; and

determine that the square root is greater than a threshold value.

8. The system of claim 6, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine a second difference based at least in part on the first scene point first axis displacement and an output of the second sinusoidal function at the first position on the first axis of the first scene point; and

determine that the second difference is greater than a determine that the second difference is greater than a second threshold distance.

9. The system of claim 5, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to determine a first region selected from the first frame that depicts an object, wherein a density of the plurality of scene point locations is higher at the first region than at a second region of the first frame.

10. The system of claim 5, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine that a texture level at a first region of the first frame is less than a texture threshold; and

omitting scene point locations of the plurality of scene point locations from the first region of the first frame.

11. The system of claim 5, wherein the first column overlaps a second column of the plurality of columns.

12. The system of claim 5, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

determine a Fourier transform of the distribution of average column second axis displacements for the plurality of columns by column position; and

determine at least three terms of a Fourier series based at least in part on the Fourier transform, wherein the first sinusoidal function comprises the at least three terms of the Fourier series.

13. The system of claim 5, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to:

identify a plurality of scene point locations including the first scene point location, wherein each of the plurality of scene point locations has a scene point second axis displacement that differs from the output of the first sinusoidal function at the first position on the first axis of the first scene point by more than the threshold distance; and

generate a foreground region of the first frame, wherein the foreground region comprises the plurality of scene point locations.

14. The system of claim 13, wherein the data storage device further comprises instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to generate a binary mask comprising a plurality of binary mask pixel values, wherein a first binary mask pixel value corresponding to the first scene point location is asserted and a second binary mask pixel value not corresponding to any of the plurality of scene point locations is un-asserted.

15. A method for detecting foreground in a video frame, comprising:

receiving, by a computing device, a video comprising a plurality of frames and a video frame sequence of the plurality of frames;

from a first frame of the plurality of frames, selecting, by the computing device, a plurality of scene point locations including a first scene point location depicting a first scene point, wherein the first scene point location is at a first position on a first axis of the first frame and a second position on a second axis of the first frame;

from a second frame of the plurality of frames, identifying, by the computing device, a second scene point location depicting the first scene point;

determining by the computing device, a first scene point first axis displacement based at least in part on the first scene point location and the second scene point location;

determining, by the computing device, a plurality of columns in the first frame, wherein the plurality of columns comprises a first column having a first column position on the first axis of the first frame, and wherein the first scene point location is in the first column;

determining, by the computing device, an average column first axis displacement for scene points depicted in the first column;

selecting, by the computing device, a first sinusoidal function based at least in part on a distribution of average column first axis displacements for the plurality of columns by column position;

determining, by the computing device, a first difference based at least in part on the first scene point first axis displacement and an output of the first sinusoidal function at a first axis position of the first scene point; and

determining, by the computing device, that the first difference is greater than a threshold distance.

16. The method of claim 15, further comprising determining a first region selected from the first frame that depicts an object, wherein a density of the plurality of scene point locations is higher at the first region than at a second region of the first frame.

17. The method of claim 15, further comprising:

determining that a texture level at a first region of the first frame is less than a texture threshold; and

omitting scene point locations of the plurality of scene point locations from the first region of the first frame.

18. The method of claim 15, wherein the first column overlaps a second column of the plurality of columns.

19. The method of claim 15, further comprising:

determining a Fourier transform of the distribution of average column first axis displacements for the plurality of columns by column position; and

determining at least three terms of a Fourier series based at least in part on the Fourier transform, wherein the first sinusoidal function comprises the at least three terms of the Fourier series.

20. The method of claim 15, further comprising:

identifying a plurality of scene point locations including the first scene point location, wherein each of the plurality of scene point locations has a scene point first axis displacement that differs from the output of the first sinusoidal function at the first axis position of the first scene point by more than the threshold distance; and

generating a foreground region of the first frame, wherein the foreground region comprises the plurality of scene point locations.