SCREEN DETECTOR

• US 20140293087A1
• Filed: 06/17/2014
• Published: 10/02/2014
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Brandt, Achiezer, Borenstein, Eran, Sharon, Eitan, Tek, Mehmet, Nataraju, Sunaad
• Granted Patent: US 9,413,477 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Technology described herein provides techniques for screen detection. This could be a dynamic screen, such as a display screen of an electronic device, or a static screen such as a street sign. In one aspect, a screen is detected in frames of image data. An active image and/or an average color image may be formed based on frames of image data. A set of lines that are candidate for boundaries of a screen are identified in the frames. The active image and/or an average color is integrated on both sides of each line in the set of lines to form integration results. A set of screen candidates are formed based on the integration results. Screens in the screen candidates are scored according to a criterion. One or top few of the screens in the set of screen candidates is selected based on the scoring.

Claims

1. A method, comprising:

forming a processing image based on corresponding pixels in a plurality of frames of image data, the processing image comprising values corresponding to the pixels in the plurality of frames;

identifying a set of lines in the plurality of frames that are candidate for boundaries of a screen;

for each line in the set, comparing the processing image on one side of the line with the other side of the line;

forming a set of screen candidates based on the comparisons;

scoring screens in the screen candidates according to a criterion; and

selecting a first of the screens in the set of screen candidates based on the scoring.

2. The method of claim 1, wherein the comparing the processing image on one side of the line with the other side of the line, for each line in the set, comprises forming a running total of values in the processing image on both sides of the line, and comparing at least one value in the running total on one side of the line with the corresponding value in the running total on the other side of the line.

3. The method of claim 1, wherein the forming a processing image based on corresponding pixels in a plurality of frames of image data comprises:

forming an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data.

4. The method of claim 1, wherein the forming a processing image based on corresponding pixels in a plurality of frames of image data comprises:

forming an average color image based on average color of corresponding pixels in the plurality of frames of image data.

5. The method of claim 1, further comprising:

detecting a rough region of the screen in the plurality of frames, including;

generating a reduced resolution version of an image in the plurality of frames of image data;

calculating a variance image from the reduced resolution version of the image; and

estimating vertical and horizontal boundaries of the screen from the variance image, including determining a region of the variance image that includes high action.

6. The method of claim 1, wherein the scoring screens in the screen candidates according to a criterion includes:

comparing a portion of the processing image that is inside of a candidate screen with a portion of the image that is just outside of the candidate screen.

7. The method of claim 1, wherein the scoring screens in the screen candidates according to a criterion includes:

comparing two portions of the processing image that are just outside of a candidate screen, but on opposites sides of the candidate screen.

8. A system comprising:

a camera that generates frames of image data;

a processor in communication with the camera and that is configured to;

form a processing image based on corresponding pixels in a plurality of the frames of image data, the processing image comprising values corresponding to the pixels in the plurality of frames;

identify a set of lines in the plurality of frames that are candidate for boundaries of a screen;

for each line in the set, form a running total of values in the processing image on both sides of the line, and compare at least one value in the running total on one side of the line with the corresponding value in the running total on the other side of the line;

form a set of screen candidates based on the comparisons;

score screens in the screen candidates according to a criterion; and

select a first of the screens in the set of screen candidates based on the scoring.

9. The system of claim 8, wherein the processor being configured to a processing image based on corresponding pixels in a plurality of frames of image data includes the processor being configured to form an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data.

10. The system of claim 8, wherein the processor being configured to a processing image based on corresponding pixels in a plurality of frames of image data includes the processor being configured to form an average color image based on average color of corresponding pixels in the plurality of frames of image data.

11. The system of claim 8, wherein the processor further detects a rough region of the screen in the plurality of frames, including the processor being configured to:

generate a reduced resolution version of an image in the plurality of frames of image data;

calculate a variance image from the reduced resolution version; and

estimate vertical and horizontal boundaries of the screen from the variance image, including determining a region of the variance image that includes high action.

12. The system of claim 8, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to compare a portion of the processing image that is inside of a candidate screen with a portion of the processing image that is just outside of the candidate screen.

13. The system of claim 8, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to compare two portions of the processing image that are just outside of a candidate screen, but on opposites sides of the candidate screen.

14. The system of claim 8, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to determine whether a candidate screen passes either a color separation test or an action separation test, the color separation test compares an average color of the processing image just inside of a candidate screen with an average color of the processing image just outside of the candidate screen, the action separation test compares action in the processing image just inside of the candidate screen with action of the processing image just outside of the candidate screen.

15. A computer-readable storage device having computer-readable instructions embodied thereon for use by a processor, the computer-readable instructions cause the processor to:

access a plurality of frames of image data captured by a camera;

form an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data, the action image comprising action values corresponding to pixels in the frames of image data;

select a set of lines in the plurality of frames that are candidates for boundaries of a screen;

integrate the action image on both sides of each selected line to form first integration results for a first side of each selected line and second integration results for a second side of each selected line, and to compare the first and second integration results for each selected line;

form a set of screen candidates based on the comparison of the first and second integration results;

score screens in the screen candidates according to a criterion; and

select a first of the screens in the set of screen candidates based on the scoring.

16. The computer-readable storage device of claim 15, wherein the instructions that cause the processor to integrate the action image cause the processor to move along the selected line and form a first running total of the action values in the action image on a first side of the selected line and a second running total for action values on a second side of the selected line, and to compare at least one action value in the first running total with the action value for a corresponding pixel location in the second running total.

17. The computer-readable storage device of claim 16, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to determine whether a candidate screen passes an action separation test, the action separation test compares action in the action image just inside of the candidate screen with action of the action image just outside of the candidate screen.

18. The computer-readable storage device of claim 15, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion cause the processor to compare a portion of the average color image that is just inside of a candidate screen at a corner junction with both a first region and a second region that are just outside of the candidate screen at the corner junction.

19. The computer-readable storage device of claim 15, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion that cause the processor to compare color uniformity at boundaries of a candidate screen based on the average color image.

20. The computer-readable storage device of claim 15, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion that cause the processor to compare a first portion of the average color image that is just outside of a candidate screen on one side with a second portion of the average color image that is just outside of the candidate screen on the opposite side, the processor compares the first portion and the second portion for color symmetry.