Method and system for real time motion picture segmentation and superposition

• US 20040212738A1
• Filed: 05/24/2004
• Published: 10/28/2004
• Est. Priority Date: 06/28/2000
• Status: Active Grant

• Current Assignee: Rateze Remote Mgmt. L.L.C. (Intellectual Ventures LLC)
• Original Assignee: Somle Development LLC (Intellectual Ventures LLC)
• Inventors: Sheraizin, Vitaly S., Sheraizin, Semion M.
• Granted Patent: US 7,742,108 B2
• Time in Patent Office: Days
• US Class Current: 348/586
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Methods and systems for separating a picture part of interest from an arbitrary background in real time videos are presented. The background may be a moving or still frame. The picture part of interest and background frames may be in or out of focus. One separation approach employs the difference between luminance and chrominance values of the input and background frames where changes in luminance from frame to frame are compensated for. In another approach, picture part of interest separation is based solely on spatial resolution differences between the background and the picture part of interest frames. The present invention also teaches a method and system for parameter matching of the picture part of interest and the basic picture into which the picture part of interest is embedded. The invention further teaches a method and system for embedding a separated picture part of interest into a basic picture containing text.

Claims

1. A method for image processing of a frame, the method comprising the step of:

separating a picture part of interest from said frame, wherein said frame has an arbitrary background.

2. A method according to claim 1 wherein said step of separating comprises the steps of:

receiving a background frame having said arbitrary background;

receiving an input frame having said picture part of interest within said arbitrary background; and

separating said picture part of interest from said arbitrary background using said input and said background frames.

3. A method according to claim 1 wherein said background frame is a still image.

4. A method according to claim 1 wherein said background frame is a moving image.

5. A method according to claim 1 wherein said background frame is out of focus.

6. A method according to claim 1 wherein said picture part of interest is out of focus.

7. A method according to claim 2 wherein said step of separating comprises the step of spatially separating said picture part of interest and background in said input frame.

8. A method according to claim 2 wherein said step of separating comprises the step of generating the difference between luminance and chrominance signal values of said input frame and said background frame.

9. A method according to claim 2 wherein said step of separating further comprises the steps of:

filtering an input signal;

estimating the pulse signal maximum of said filtered input signal;

determining the time difference between adjacent pulse signal maxima of said input signal;

comparing said time difference to a threshold value; and

accepting a signal for use as part of a picture part of interest mask if said signal is below said threshold value, wherein said above steps generally are applied separately to the luminance, red chrominance and blue chrominance components of said input signal.

10. A system for separating a picture part of interest of an input frame, the input frame having an arbitrary background, said system comprising:

a mask generating unit for generating a picture part of interest mask using the difference between chrominance and luminance signal values in said input frame and an arbitrary background frame; and

a separator unit for separating a picture part of interest from said input frame using said picture part of interest mask.

11. A system according to claim 10 wherein said mask generating unit further comprises:

a luminance compensation unit for compensating for changes in background luminance signal when going from frame to frame; and

means for generating a picture part of interest by removing said compensated background luminance signal from a picture part of interest in an arbitrary background signal.

12. A system according to claim 11 wherein said mask generating unit further comprises:

means for generating a chrominance signal for a picture part of interest by subtracting out a background chrominance signal from said picture part of interest in an arbitrary background signal.

13. A system according to claim 10 wherein said mask generating unit further comprises:

a background luminance frame memory for providing a background luminance signal;

a first divider for dividing luminance signals from an input frame and a memorized background luminance signal (quotient A), said memorized background luminance being provided by said background luminance frame memory;

a second switch receiving a luminance window from a window pulse generator unit and quotient A from said first divider;

an averaging circuit for calculating an estimated light change coefficient K by averaging quotient A over the pixels of said luminance window;

a second divider for detecting changes in the input frame luminance signal by dividing an input frame luminance signal by said light change coefficient K (quotient B); and

a first summer for subtracting said quotient B from said memorized background luminance signal.

14. A system according to claim 13 wherein said mask generating unit further comprises:

a background chrominance frame memory for providing a memorized BG chrominance ratio signal, said memory set up by a switch;

a divider for dividing red and blue chrominance signals (quotient C), said chrominance signals received from a color unit; and

a summer for subtracting said quotient C from said memorized background chrominance ratio signal.

15. A system according to claim 10 wherein said separator unit comprises:

a time aligner for aligning color signals of said input frame; and

a switch receiving said time aligned input frame color signals, and a picture part of interest mask from said mask generating unit, said switch thereby generating a picture part of interest from said mask and said color signals.

16. A system for separating a picture part of interest from an input frame having an arbitrary background, said system comprising:

at least one high pass filter for determining the edges of an input signal of said picture part of interest within said arbitrary background;

at least one pulse signal maximum estimator for determining the time maxima of said filtered input signal;

at least one maximum-to-maximum time determiner for determining the time difference between consecutive signal maxima;

at least one comparator for comparing said time difference with a predetermined threshold value; and

an OR gate for generating a picture part of interest mask from signals received from said at least one comparartor.

17. A system according to claim 16 wherein said system processes the luminance, red chrominance, and blue chrominance signals of said input signal separately, each of said signals being processed by its own high pass filter, pulse signal maximum estimator, time determiner, and comparator.

18. A method for text detection for use in superpositioning a video image in a basic picture containing text, the method comprising the steps of:

storing signal data from a plurality of input frames, said data being selected based on predetermined criteria, said storage being effected on a pixel by pixel basis;

decoding a stored input frame by determining the number of consecutive stored frames that have text at a predetermined pixel position; and

determining the number of pixels with text in a given frame and comparing the number to a predetermined criteria, whereby a text is determined to exist if a sufficient number of pixels in enough consecutive frames have met the predetermined criteria.

19. A system for positioning a picture part of interest in a basic picture, said basic picture containing text, said system comprising;

a text detector unit for generating a text mask from a luminance signal of a window mask of said basic picture;

a scaler for scaling a separated picture part of interest, said scaling being controlled by a background/ foreground controller; and

a means for embedding said scaled picture part of interest in said text mask.

20. A system according to claim 19 wherein said text detector unit further comprises:

a frame storage unit for storing filtered signals on a pixel-by-pixel basis for each frame;

a decoder for determining the number of consecutive stored frames which have text in corresponding pixel positions; and

a counter for counting the number of pixels in each of said stored frames.

21. A method for superpositioning a picture part of interest in a basic picture, said method comprising the step of:

matching parameters between said basic picture and said picture part of interest wherein said parameters consist of at least one of the following;

luminance, chrominance and spatial resolution.

22. A system for matching parameters of a separated picture part of interest and a basic picture when superpositioning the picture part of interest in the basic picture, said system comprising:

a luminance matching unit for matching luminance signals of said basic picture and said picture part of interest;

a chrominance matching unit for matching chrominance signals of said basic picture and said picture part of interest; and

a spatial resolution matching unit for spatially resolving said basic picture and said picture part of interest using said luminance and matched luminance signals of said basic picture and said picture part of interest.

23. A system according to claim 22 wherein said spatial resolution matching unit comprises:

means for delaying a basic picture luminance signal for one and more than one arbitrary time units and comparing these delayed signals after subtracting out the original basic picture luminance;

means for time aligning a matched luminance signal where said matching is effected for the basic picture and picture part of interest luminances;

means for filtering a matched basic picture and picture part of interest luminance;

at least one switch for receiving either said once or said more than once delayed basic picture luminance and the time aligned and filtered matched luminance for producing a matched spatially resolved basic picture and picture part of interest.

24. A system according to claim 23 wherein said spatial resolution matching unit comprises:

a first memory pixel for delaying a basic picture luminance signal by one time unit;

a first summer for subtracting said once delayed luminance signal from said basic picture luminance signal;

a second memory pixel connected to said first memory pixel for delaying a basic picture luminance signal by a second time unit;

a second summer for subtracting said twice delayed luminance signal from said basic picture luminance signal;

a comparator for comparing the signal intensities of the differences between said once and twice delayed signals (S2) and the basic picture luminance signal (S1);

a time aligner for aligning a luminance matched picture part of interest signal (S3);

two low pass filters, one for filtering said luminance matched picture part of interest signal (S4) when said basic picture is out of focus and the second for filtering a luminance matched picture part of interest signal (S5) when said picture part of interest is positioned in the background;

a first switch for determining an acceptable luminance signal from among the filtered, aligned and compared signals (S1, S2, S3, S4);

a controller for controlling the positioning of the picture part of interest depending on whether the picture part of interest is to be placed in the foreground or background; and

a second switch for producing a mask from the luminance signal data and foreground-background information received from said first switch, said controller and luminance matched picture part of interest (S5) from said luminance matching unit.