System and method for automatic detection of spherical video content

• US 10,268,893 B2
• Filed: 08/18/2017
• Issued: 04/23/2019
• Est. Priority Date: 10/30/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Birkbeck, Neil Aylon Charles, Lam, Ka-kit, Kelly, Damien, Crawford, Andrew Joseph, Kokaram, Anil Christopher
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Bloom, Nathan J
• Application Number:
  US15/680,594
• Publication Number:
  US 20170344830A1
• Time in Patent Office: 613 Days
• Field of Search: None
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/7847 using low-level visual features of the video content
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/42 Global feature extraction by analysis of the whole pattern, e.g. using frequency domain transformations or autocorrelation
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 30/228 of three-dimensional handwriting, e.g. writing in the air
• CPC Class Codes: H04N 13/139 Format conversion, e.g. of frame-rate or size
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Systems and methods are described for identifying the video content as spherical video or non-spherical video in response to determining that frame scores and video scores satisfy a threshold level. For example, a plurality of image frames can be extracted from video content, classified in a dual stage process, and scored according to particular classification and scoring mechanisms.

Claims

1. A computer-implemented method to identify spherical video content, the method comprising:

receiving, at a video server, video content;

identifying a plurality of image frames in the video content as candidate image frames;

classifying the candidate image frames according to a plurality of image features, the classifying including assigning a frame score to each of the candidate image frames;

identifying a portion of the candidate image frames having a frame score satisfying a threshold condition pertaining to at least one of the plurality of image features; and

generating a video score for the video content by aggregating the frame scores for the portion of the candidate image frames having a frame score satisfying the threshold condition, wherein the video score is used to configure playback of the video content in response to determining that the video score satisfies a threshold video score.

2. The method of claim 1, further comprising:

labeling the video content as spherical video content in response to determining that the video score satisfies a threshold video score.

3. The method of claim 1, wherein the video score indicates a likelihood of spherical video image features being detected in at least one of the candidate image frames in the portion.

4. The method of claim 1, wherein the classifying of the candidate image frames includes:

unwarping the candidate image frames to generate unwarped candidate image frames;

detecting a plurality of edges of each unwarped candidate image frame and determining whether portions of the detected plurality of edges are curved.

5. The method of claim 4, further comprising assigning a frame score that satisfies the threshold condition, in response to determining an unwarped version of a candidate image frame in the candidate image frames includes more curved edges than straight edges.

6. The method of claim 1, wherein the plurality of image features include at least an aspect ratio determined for each of the candidate image frames, the aspect ratio being used as an indication that video content that includes a respective candidate image includes spherical content, if the aspect ratio associated with the respective candidate image indicates an equal number of pixels per degree of view.

7. The method of claim 1, wherein the plurality of image features include at least a determined variance level between columns of pixels in at least one candidate image frame.

8. A video management system including instructions stored on a non-transitory computer-readable storage medium, the system comprising:

at least one processor; and

a memory comprising instructions which, when executed by the at least one processor, cause the at least one processor to;

receive, at a video server, video content;

identify a plurality of image frames in the video content as candidate image frames;

classify the candidate image frames according to a plurality of image features, the classifying including assigning a frame score to each of the candidate image frames;

identify a portion of the candidate image frames having a frame score satisfying a threshold condition pertaining to at least one of the plurality of image features; and

generate a video score for the video content by aggregating the frame scores for the portion of the candidate image frames having a frame score satisfying the threshold condition, wherein the video score is used to configure playback of the video content in response to determining that the video score satisfies a threshold video score.

9. The video management system of claim 8, wherein the at least one processor is further operable to:

label the video content as spherical video content in response to determining that the video score satisfies a threshold video score.

10. The video management system of claim 8, wherein the video score indicates a likelihood of spherical video image features being detected in at least one of the candidate image frames in the portion.

11. The video management system of claim 8, wherein the classifying of the candidate image frames includes:

unwarping the candidate image frames to generate unwarped candidate image frames;

detecting a plurality of edges of each unwarped candidate image frame and determining whether portions of the detected plurality of edges are curved.

12. The video management system of claim 11, wherein the at least one processor is further operable to:

assign a frame score that satisfies the threshold condition, in response to determining an unwarped version of a candidate image frame in the candidate image frames includes more curved edges than straight edges.

13. The video management system of claim 8, wherein the plurality of image features include at least an aspect ratio determined for each of the candidate image frames, the aspect ratio being used as an indication that video content that includes a respective candidate image includes spherical content, if the aspect ratio associated with the respective candidate image indicates an equal number of pixels per degree of view.

14. The video management system of claim 8, wherein the plurality of image features include at least a determined variance level between columns of pixels in at least one candidate image frame.

15. A non-transitory recordable storage medium having recorded and stored thereon instructions that, when executed by a processor, perform actions of:

receiving image content;

identifying a plurality of image frames in the image content as candidate image frames;

classifying the candidate image frames according to a plurality of image features, the classifying including assigning a frame score to each of the candidate image frames;

identifying a portion of the candidate image frames having a frame score satisfying a threshold condition; and

generating an overall score for the image content by aggregating the frame scores for the portion of the candidate image frames having a frame score satisfying the threshold condition, wherein the overall score is used to configure playback of the image content in response to determining that the overall score satisfies a threshold score.

16. The non-transitory recordable storage medium of claim 15, wherein the actions further include:

labeling the image content as spherical image content in response to determining that the overall score satisfies a predefined threshold score.

17. The non-transitory recordable storage medium of claim 15, wherein the overall score indicates a likelihood of spherical image features being detected in at least one of the candidate image frames in the portion.

18. The non-transitory recordable storage medium of claim 15, wherein the classifying of the candidate image frames includes:

unwarping the candidate image frames to generate unwarped candidate image frames;

detecting a plurality of edges in each unwarped candidate image frame and determining whether portions of the detected plurality of edges are curved.

19. The non-transitory recordable storage medium of claim 18, wherein the actions further include:

assigning a frame score that satisfies the threshold condition, in response to determining an unwarped version of a candidate image frame in the candidate image frames includes more curved edges than straight edges.

20. The non-transitory recordable storage medium of claim 15, wherein the plurality of image features include at least an aspect ratio determined for each of the candidate image frames, the aspect ratio being used as an indication that image content that includes a respective candidate image includes spherical content, if the aspect ratio associated with the respective candidate image indicates an equal number of pixels per degree of view.