Detecting suitability for converting monoscopic visual content to stereoscopic 3D

• US 9,674,498 B1
• Filed: 03/15/2013
• Issued: 06/06/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Mukherjee, Debargha, Wu, Chen
• Primary Examiner(s): FINDLEY, CHRISTOPHER G
• Application Number:
  US13/840,623
• Time in Patent Office: 1,544 Days
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes

Abstract

A suitability detector identifies a plurality of frames of an input video. The suitability detector determines, based on characteristics of the plurality of frames, whether the input video is suitable for conversion from monoscopic visual content to stereoscopic 3D. The characteristics may include a visual cue present in the plurality of frames and a visual quality of the plurality of frames. If the input video is suitable for conversion, an image converter converts the input video to stereoscopic 3D.

Claims

1. A method comprising:

identifying, by a processing device, a plurality of frames of an input video;

determining, based on the plurality of frames, whether the input video is suitable for conversion from monoscopic visual content to stereoscopic 3D, wherein determining whether the input video is suitable for conversion comprises determining whether the input video is monoscopic, and wherein determining whether the input video is monoscopic comprises;

identifying a set of edge pixels in each of the plurality of frames of the input video,

calculating an edge cue for the input video using sets of edge pixels across the plurality of frames of the input video, and

determining whether the edge cue for the input video satisfies one or more stereo threshold conditions that define whether the input video is monoscopic; and

in response to determining the input video is suitable for conversion, causing the input video to be converted to stereoscopic 3D; and

in response to determining the input video is not suitable for conversion, causing the input video not to be converted to stereoscopic 3D.

2. The method of claim 1, wherein determining whether the input video is suitable for conversion further comprises:

generating a first color histogram for a first portion of each of the plurality of frames, a second color histogram for a second portion of each of the plurality of frames, a third color histogram for a third portion of each of the plurality of frames and a fourth color histogram for a fourth portion of each of the plurality of frames;

determining a first histogram distance between the first color histogram and the second color histogram and a second histogram distance between the third color histogram and the fourth color histogram for each of the plurality of frames;

calculating a histogram cue based on an average first histogram distance and an average second histogram distance for the plurality of frames; and

comparing the histogram cue to a plurality of histogram thresholds to determine whether each of the plurality of frames comprises the stereo image pair.

3. The method of claim 2, wherein the first portion comprises a left half and the second portion comprises a right half, wherein the third portion comprises a top half and the fourth portion comprises a bottom half, and wherein the histogram cue comprises the average second histogram distance minus the average first histogram distance divided by the sum of the average histogram distances.

4. The method of claim 1, wherein calculating the edge cue comprises:

identifying two center columns and two center rows in each of the plurality of frames;

determining a fraction of pixels in the center columns and center rows that comprise edge pixels in each of the plurality of frames; and

calculating the edge cue based on an average fraction of edge pixels in the center columns and an average fraction of edge pixels in the center rows for the plurality of frames.

5. The method of claim 4, wherein the edge cue comprises the average fraction of edge pixels in the center columns minus the average fraction of edge pixels in the center rows divided by the sum of the average fractions of edge pixels.

6. The method of claim 1, wherein determining whether the input video is suitable for conversion further comprises:

identifying two center columns and two center rows in each of the plurality of frames;

determining a fraction of pixels in the center columns and center rows that have a same color value as an adjacent pixel in the center columns or center rows in each of the plurality of frames;

determining a number of the plurality of frames where the fraction of pixels in the center columns or center rows exceeds a first threshold;

calculating a separator cue based on the number of frames where the fraction of pixels in the center columns or center rows exceeds the first threshold; and

comparing the separator cue to a plurality of separator thresholds to determine whether each of the plurality of frames comprises the stereo image pair.

7. The method of claim 6, wherein the separator cue comprises the number of frames where the fraction of pixels in the center columns exceeds the first threshold minus the number of frames where the fraction of pixels in the center rows exceeds the first threshold, divided by the sum of the number of frames where the fraction of pixels exceeds the first threshold in the center columns and the center rows.

8. The method of claim 1, wherein determining whether the input video is suitable for conversion further comprises determining whether at least one of a size of the input video satisfies a time threshold condition, or a visual quality of the input video satisfies one or more quality threshold conditions based on at least one of an average quantizer step size of the plurality of frames, a video resolution of the input video, or a noise level of the input video.

9. A computer system comprising:

a processing device;

a memory coupled to the processing device; and

a suitability detector, executable by the processing device from the memory, to;

identify a plurality of frames of an input video;

determine, based on the plurality of frames, whether the input video is suitable for conversion from monoscopic visual content to stereoscopic 3D, wherein to determine whether the input video is suitable for conversion, the suitability detector is to determine whether the input video is monoscopic, and wherein to determine whether the input video is monoscopic, the suitability detector is to;

identify a set of edge pixels in each of the plurality of frames of the input video,

calculate an edge cue for the input video using sets of edge pixels across the plurality of frames of the input video, and

determine whether the edge cue for the input video satisfies one or more stereo threshold conditions that define whether the input video is monoscopic; and

in response to determining the input video is suitable for conversion, cause the input video to be converted to stereoscopic 3D; and

in response to determining the input video is not suitable for conversion, cause the input video not to be converted to stereoscopic 3D.

10. The computer system of claim 9, wherein to determine whether the input video is suitable for conversion, the suitability detector is further to identify whether each of the plurality of frames comprises the stereo image pair in a known orientation.

11. The computer system of claim 9, wherein to determine whether the input video is suitable for conversion, the suitability detector is further to determine whether a histogram cue satisfies one or more histogram threshold conditions, wherein the histogram cue is based on a comparison of color values of pixels found in left and right sections of each of the plurality of frames and found in top and bottom sections of each of the plurality of frames.

12. The computer system of claim 9, wherein the edge cue is based on the presence of an image edge located along a horizontal center or a vertical center in at least a portion of the plurality of frames.

13. The computer system of claim 9, wherein to determine whether the input video is suitable for conversion, the suitability detector is further to determine whether a separator cue satisfies one or more separator threshold conditions, wherein the separator cue is based on the presence of a constant-color separator along a horizontal center or a vertical center in at least a portion of the plurality of frames.

14. The computer system of claim 9, wherein to determine whether the input video is suitable for conversion, the suitability detector is further to determine whether at least one of a size of the input video satisfies a time threshold condition, or a visual quality of the input video satisfies one or more quality threshold conditions based on at least one of an average quantizer step size of the plurality of frames, a video resolution of the input video, or a noise level of the input video.

15. A non-transitory machine-readable storage medium storing instructions which, when executed, cause a processing device to perform operations comprising:

identifying, by the processing device, a plurality of frames of an input video;

determining, based on the plurality of frames, whether the input video is suitable for conversion from monoscopic visual content to stereoscopic 3D, wherein determining whether the input video is suitable for conversion comprises determining whether the input video is monoscopic, and wherein determining whether the input video is monoscopic comprises;

identifying a set of edge pixels in each of the plurality of frames of the input video,

calculating an edge cue for the input video using sets of edge pixels across the plurality of frames of the input video, and

determining whether the edge cue for the input video satisfies one or more stereo threshold conditions that define whether the input video is monoscopic; and

in response to determining the input video is suitable for conversion, causing the input video to be converted to stereoscopic 3D; and

in response to determining the input video is not suitable for conversion, causing the input video not to be converted to stereoscopic 3D.

16. The non-transitory machine-readable storage medium of claim 15, wherein determining whether the input video is suitable for conversion further comprises identifying whether each of the plurality of frames comprises the stereo image pair in a known orientation.

17. The non-transitory machine-readable storage medium of claim 15, wherein determining whether the input video is suitable for conversion further comprises determining whether a histogram cue satisfies one or more histogram threshold conditions, wherein the histogram cue is based on a comparison of color values of pixels found in left and right sections of each of the plurality of frames and found in top and bottom sections of each of the plurality of frames.

18. The non-transitory machine-readable storage medium of claim 15, wherein the edge cue is based on the presence of an image edge located along a horizontal center or a vertical center in at least a portion of the plurality of frames.

19. The non-transitory machine-readable storage medium of claim 15, wherein determining whether the input video is suitable for conversion further comprises determining whether a separator cue satisfies one or more separator threshold conditions, wherein the separator cue is based on the presence of a constant-color separator along a horizontal center or a vertical center in at least a portion of the plurality of frames.

20. The non-transitory machine-readable storage medium of claim 15, wherein determining whether the input video is suitable for conversion further comprises determining whether at least one of a size of the input video satisfies a time threshold condition, or a visual quality of the input video satisfies one or more quality threshold conditions, wherein the visual quality of the plurality of frames is based on at least one of an average quantizer step size of the plurality of frames, a video resolution of the plurality of frames, or a noise level of the plurality of frames.

21. A method comprising:

identifying, by a processing device, a plurality of frames of an input video;

determining, based on the plurality of frames, whether the input video is suitable for conversion from monoscopic visual content to stereoscopic 3D, wherein determining whether the input video is suitable for conversion comprises determining whether the input video is monoscopic, and wherein determining whether the input video is monoscopic comprises;

identifying pixels having a same color value in a center of each of the plurality of frames of the input video,

calculating a separator cue for the input video in view of identified pixels with same pixel values across the plurality of frames of the input video, and

determining whether the separator cue for the input video satisfies one or more stereo threshold conditions that define whether the input video is monoscopic; and

in response to determining the input video is suitable for conversion, causing the input video to be converted to stereoscopic 3D; and

in response to determining the input video is not suitable for conversion, causing the input video not to be converted to stereoscopic 3D.

22. The method of claim 21, wherein identifying pixels having the same color value in the center of each of the plurality of frames comprises:

identifying two center columns and two center rows in each of the plurality of frames; and

comparing color values of adjacent pixels in the two center columns and the two center rows in each of the plurality of frames.

23. The method of claim 21, wherein calculating the separator cue for the input video comprises:

determining a fraction of pixels in the two center columns and the two center rows that have the same color value as adjacent pixels in the two center columns or the two center rows in each of the plurality of frames;

determining a number of the plurality of frames where the fraction of pixels in the two center columns or the two center rows exceeds a first separator threshold; and

calculating the separator cue based on the number of frames where the fraction of pixels in the two center columns or the two center rows exceeds the first separator threshold.