SYSTEM AND METHOD FOR AUTOMATIC DETECTION OF SPHERICAL VIDEO CONTENT

• US 20170344830A1
• Filed: 08/18/2017
• Published: 11/30/2017
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Birkbeck, Neil Aylon Charles, Lam, Ka-kit, Kelly, Damien, Crawford, Andrew Joseph, Kokaram, Anil Christopher
• Granted Patent: US 10,268,893 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Systems and methods are described for identifying the video content as spherical video or non-spherical video in response to determining that frame scores and video scores satisfy a threshold level. For example, a plurality of image frames can be extracted from video content, classified in a dual stage process, and scored according to particular classification and scoring mechanisms.

Claims

1. A computer-implemented method comprising:

receiving, at a video server, video content;

extracting a plurality of image frames from the video content;

classifying, in a first stage, the plurality of image frames according to a first set of features, the classifying including identifying a portion of the plurality of image frames as candidate image frames;

classifying, in a second stage, the candidate image frames according to a second set of features, the classifying including assigning a frame score to each of the candidate image frames, the frame score being based on at least one of the second set of features;

selecting a portion of the candidate image frames having a frame score satisfying a threshold frame score condition;

generating a video score for the video content by aggregating together the portion of the candidate image frames having a frame score satisfying the threshold frame score condition; and

identifying the video content as spherical video content in response to determining that the video score satisfies a threshold video score.

2. The method of claim 1, further comprising monitoring for additional video content, at the video server and in response to detecting the additional video content, automatically performing the first stage and the second stage to classify the additional video content and identifying the additional video content as spherical video content or non-spherical video content based at least in part on an output of the first stage and the second stage.

3. The method of claim 1, wherein the first stage and the second stage are configured using a plurality of training videos that model spherical and non-spherical video content according to the first set of features and the second set of features.

4. The method of claim 1, wherein the first stage includes analyzing at least two edges of each of the plurality of image frames.

5. The method of claim 1, wherein the second stage includes analyzing central regions of the candidate image frames.

6. The method of claim 1, wherein the second stage includes performing at least one of:

generating a grayscale version of the candidate image frames and calculating a standard deviation of the grayscale version of the candidate image frames;

calculating entropy of the grayscale version of the candidate image frames; and

calculating entropy of an edge detected version of the candidate image frames.

7. The method of claim 1, wherein the first set of features includes at least two calculations corresponding to the plurality of image frames, the calculations including an aspect ratio, video dimension, standard deviation at both image poles, and left and right border variances.

8. The method of claim 1, wherein the second set of features includes at least two calculations corresponding to the candidate image frames, the calculations including a standard deviation, an edge detection, an image entropy, an image entropy of the edge detection, a Hough transform, a measure of texture for at least one pole defined in at least one image frame, and a metric corresponding to an edge of at least one image frame and a column of pixels adjacent to the edge.

9. The method of claim 1, wherein the second set of features are defined and calculated according to at least one image projection type selected from the group consisting of an equirectangular projection, a warped equirectangular projection, a rectilinear projection, a circular projection, and a stereographic projection.

10. A video management system including instructions stored on a non-transitory computer-readable storage medium, the system comprising:

a video server configured to receive, from a plurality of users, uploaded video content, the uploaded video content including a plurality of images;

an extractor module configured to extract a plurality of image frames from the video content;

a first classifier module configured to classify the plurality of image frames according to a first set of features being associated with the plurality of image frames and identify a portion of the plurality of image frames as candidate image frames;

a second classifier module configured to classify the candidate image frames according to a second set of features being associated with the candidate image frames; and

a scoring module configured to generate a plurality of frame scores corresponding to the plurality of image frames and a plurality of video scores corresponding to the video content, the scoring module generating a categorization marker applicable to the video content, the categorization marker indicating spherical video content or non-spherical video content and being based on the plurality of frames scores and the plurality of video scores.

11. The video management system of claim 10, further comprising an image analyzer configured to analyze image characteristics in the plurality of image frames, the image analyzer adapted to perform calculations based on the first set of features and on the second set of features.

12. The video management system of claim 10, wherein the first set of features includes at least two calculations corresponding to the plurality of image frames, the calculations including an aspect ratio, video dimension, standard deviation at both image poles, and left and right border variances.

13. The video management system of claim 10, wherein the second set of features includes at least two calculations corresponding to the candidate image frames, the calculations including a standard deviation, an edge detection, an image entropy, an image entropy of the edge detection, a Hough transform, a measure of texture for at least one pole defined in at least one image frame, and a metric corresponding to an edge of at least one image frame and a column of pixels adjacent to the edge.

14. The video management system of claim 10, wherein the second classifier module performs at least one of:

generating a grayscale version of the candidate image frames and calculating a standard deviation of the grayscale version of the candidate image frames;

calculating entropy of the grayscale version of the candidate image frames; and

calculating entropy of an edge detected version of the candidate image frames.

15. The video management system of claim 10, wherein classification criteria is generated for the first classifier module and the second classifier module using a plurality of training videos that model spherical and non-spherical video content according to the first set of features and the second set of features.

16. A non-transitory recordable storage medium having recorded and stored thereon instructions that, when executed, perform actions of:

receiving, at a video server, video content;

extracting a plurality of image frames from the video content;

classifying, during a first stage, the plurality of image frames according to a first set of features, the classifying including identifying a candidate image frame from the plurality of image frames;

assigning, during a second stage, a frame score to the candidate image based on at least one of a second set of features;

selecting the candidate image frame when the frame score satisfies a threshold frame score condition;

generating a video score for the video content based on the candidate image frame; and

defining the video content as spherical video content in response to determining that the video score satisfies a threshold video score.

17. The non-transitory recordable storage medium of claim 16, further comprising monitoring for additional video content, at the video server and in response to detecting the additional video content, automatically performing the stage and the second stage to classify the additional video content.

18. The non-transitory recordable storage medium of claim 16, wherein the first set of features includes at least two calculations corresponding to the plurality of image frames, the calculations including an aspect ratio, video dimension, standard deviation at both image poles, and left and right border variances.

19. The non-transitory recordable storage medium of claim 16, wherein the second set of features includes at least two calculations corresponding to the candidate image frames, the calculations including a standard deviation, an edge detection, an image entropy, an image entropy of the edge detection, a Hough transform, a measure of texture for at least one pole defined in at least one image frame, and a metric corresponding to an edge of at least one image frame and a column of pixels adjacent to the edge.

20. The non-transitory recordable storage medium of claim 16, wherein the second set of features are defined and calculated according to at least one image projection type selected from the group consisting of an equirectangular projection, a warped equirectangular projection, a rectilinear projection, a circular projection, and a stereographic projection.