Screen detector

• US 9,413,477 B2
• Filed: 06/17/2014
• Issued: 08/09/2016
• Est. Priority Date: 05/10/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Brandt, Achiezer, Borenstein, Eran, Sharon, Eitan, Tek, Mehmet, Nataraju, Sunaad
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Kholdebarin, Iman K
• Application Number:
  US14/306,942
• Publication Number:
  US 20140293087A1
• Time in Patent Office: 784 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: H04H 2201/90 characterised by the use of signatures
• CPC Class Codes: H04H 60/37 for identifying segments of broadcast information, e.g. scenes or extracting programme ID
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

Technology described herein provides techniques for screen detection. This could be a dynamic screen, such as a display screen of an electronic device, or a static screen such as a street sign. In one aspect, a screen is detected in frames of image data. An active image and/or an average color image may be formed based on frames of image data. A set of lines that are candidate for boundaries of a screen are identified in the frames. The active image and/or an average color is integrated on both sides of each line in the set of lines to form integration results. A set of screen candidates are formed based on the integration results. Screens in the screen candidates are scored according to a criterion. One or top few of the screens in the set of screen candidates is selected based on the scoring.

Claims

1. A method, comprising:

forming a processing image based on corresponding pixels in a plurality of frames of image data, the processing image comprising values corresponding to the pixels in the plurality of frames;

identifying a set of lines in the plurality of frames that are candidate for boundaries of a screen;

for each line in the set, comparing the values in the processing image along one side of the line with the values in the processing image along the other side of the line, including forming a first running total of the values in the processing image along one side of at least a portion of the line and a second running total of the values in the processing image along the other side of the at least a portion of the line, and comparing a value in the first running total with the corresponding value in the second running total;

forming a set of screen candidates based on the comparison of the first running total with the second running total for each line in the set;

scoring screens in the set of screen candidates according to a criterion; and

selecting a first of the screens in the set of screen candidates based on the scoring as a potential screen.

2. The method of claim 1, wherein the forming a processing image based on corresponding pixels in a plurality of frames of image data comprises:

forming an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data.

3. The method of claim 1, wherein the forming a processing image based on corresponding pixels in a plurality of frames of image data comprises:

forming an average color image based on average color of corresponding pixels in the plurality of frames of image data.

4. The method of claim 1, further comprising:

detecting a rough region of the screen in the plurality of frames, including;

generating a reduced resolution version of an image in the plurality of frames of image data;

calculating a variance image from the reduced resolution version of the image; and

estimating vertical and horizontal boundaries of the screen from the variance image, including determining a region of the variance image that includes high action.

5. The method of claim 1, wherein the scoring screens in the screen candidates according to a criterion includes:

comparing a portion of the processing image that is inside of a candidate screen with a portion of the image that is just outside of the candidate screen.

6. The method of claim 1, wherein the scoring screens in the screen candidates according to a criterion includes:

comparing two portions of the processing image that are just outside of a candidate screen, but on opposites sides of the candidate screen.

7. A system comprising:

a camera that is configured to generate frames of image data;

a processor in communication with the camera and that is configured to;

form a processing image based on corresponding pixels in a plurality of the frames of image data, the processing image comprising values corresponding to the pixels in the plurality of frames;

identify a set of lines in the plurality of frames that are candidate for boundaries of a screen;

for each line in the set, form a first running total of values in the processing image in a first portion along one side of the line and a second running total of values in the processing image in a second portion along the other side of the line, wherein the first portion is adjacent to the second portion, and compare a value in the first running total with the corresponding value in the second running total;

form a set of screen candidates based on the comparison of the first running total with the second running total for each line in the set;

score screens in the screen candidates according to a criterion; and

select a first of the screens in the set of screen candidates based on the scoring.

8. The system of claim 7, wherein the processor being configured to form a processing image based on corresponding pixels in a plurality of frames of image data includes the processor being configured to form an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data.

9. The system of claim 7, wherein the processor being configured to form a processing image based on corresponding pixels in a plurality of frames of image data includes the processor being configured to form an average color image based on average color of corresponding pixels in the plurality of frames of image data.

10. The system of claim 7, wherein the processor further detects a rough region of the screen in the plurality of frames, including the processor being configured to:

generate a reduced resolution version of an image in the plurality of frames of image data;

calculate a variance image from the reduced resolution version; and

estimate vertical and horizontal boundaries of the screen from the variance image, including determining a region of the variance image that includes high action.

11. The system of claim 7, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to compare a portion of the processing image that is inside of a candidate screen with a portion of the processing image that is just outside of the candidate screen.

12. The system of claim 7, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to compare two portions of the processing image that are just outside of a candidate screen, but on opposites sides of the candidate screen.

13. The system of claim 7, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to determine whether a candidate screen passes either a color separation test or an action separation test, the color separation test compares an average color of the processing image just inside of a candidate screen with an average color of the processing image just outside of the candidate screen, the action separation test compares action in the processing image just inside of the candidate screen with action of the processing image just outside of the candidate screen.

14. The system of claim 7, wherein the set of lines in the plurality of frames comprises multiple lines in a single one of the frames.

15. The system of claim 7, wherein the processor is further configured to:

determine a new first portion and a new second portion for a first of the lines in the set; and

re-calculate the first running total and the second running total for the new first portion and the new second portion, including the processor being configured to modify the original first running total and the original second running total based on a running total of values along the line that were not used when determining the original first running total and the original second running total.

16. A computer-readable storage device having computer-readable instructions embodied thereon for use by a processor, the computer-readable instructions cause the processor to:

access a plurality of frames of image data;

form an action image based on differences between corresponding pixels in two or more of the plurality of frames of image data, the action image comprising action values corresponding to pixels in the frames of image data;

select a set of lines in the plurality of frames that are candidates for boundaries of a screen;

for each selected line;

a) move along the selected line to form a first running total of action values in the action image along a first side of the selected line and a second running total of action values along a second side of the selected line, and b) compare an action value in the first running total with an action value for a corresponding pixel location in the second running total;

form a set of screen candidates based on the comparison of the first and second running totals for each selected line;

score screens in the screen candidates according to a criterion; and

select a first of the screens in the set of screen candidates as a potential screen based on the scoring.

17. The computer-readable storage device of claim 16, wherein the processor being configured to score screens in the screen candidates according to a criterion includes the processor being configured to determine whether a candidate screen passes an action separation test, the action separation test compares action in the action image just inside of the candidate screen with action of the action image just outside of the candidate screen.

18. The computer-readable storage device of claim 16, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion cause the processor to compare a portion of the average color image that is just inside of a candidate screen at a corner junction with both a first region and a second region that are just outside of the candidate screen at the corner junction.

19. The computer-readable storage device of claim 16, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion that cause the processor to compare color uniformity at boundaries of a candidate screen based on the average color image.

20. The computer-readable storage device of claim 16, wherein the instructions further cause the processor to form an average color image based on average color of corresponding pixels in the plurality of frames of image data, wherein the instructions that cause the processor to score screens in the screen candidates according to a criterion that cause the processor to compare a first portion of the average color image that is just outside of a candidate screen on one side with a second portion of the average color image that is just outside of the candidate screen on the opposite side, the processor compares the first portion and the second portion for color symmetry.