Detecting apparatus for detecting a contour of a moving region in a dynamic image

• US 5,587,927 A
• Filed: 01/06/1994
• Issued: 12/24/1996
• Est. Priority Date: 03/26/1993
• Status: Expired due to Term

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Original Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation)
• Inventors: Nagao, Kenji, Sohma, Masaki, Kobayakawa, Michihiro, Ando, Shigeru
• Primary Examiner(s): Voeltz, Emanuel T.
• Assistant Examiner(s): Pipala, Edward
• Application Number:
  US08/178,163
• Time in Patent Office: 1,083 Days
• Field of Search: 364/514, 364/516, 364/554, 364/514 A, 382/16, 382/17, 382/22, 382/27, 382/236, 382/239, 382/254, 382/256, 382/276, 382/282
• US Class Current: 702/167
• CPC Class Codes: G06T 7/12 Edge-based segmentation
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A detecting apparatus consists of a covariance calculating section for calculating covariances Sij=∫∫∫(fi*fj)dxdydt (i,jε{x,y,t}) with gradients fx, fy and ft of variable densities of pixels, a characteristic value calculating section for calculating characteristic values by utilizing the covariances to express characteristic changes in the variable densities of the pixels, a first-stage pixel classifying section for classifying the pixels into a FIX type denoting that an observed pixel is positioned in a static region of the dynamic image, a MOT type denoting that the observed pixel is positioned inside the moving region and an ACR type denoting that the observed pixel is positioned in a boundary region between the moving and the static regions, a second-stage pixel classifying section for re-classifying the pixels to match a classification type of the observed pixel with those of pixels surrounding the observed pixel, a contour candidate limiting section for limiting a region occupied by pixels re-classified into the ACR type as a candidate region, and a contour drawing section for drawing a contour of the moving section passing through a middle portion of the candidate region.

Claims

1. A detecting method of a contour of a moving region of a dynamic image for distinguishing a moving object positioned in the moving region, comprising the steps of:

inputting a series of frames of dynamic image which consists of variable densities F(x,y,t) of pixels;

calculating a variable density gradient fx=∂

F(x,y,t)/∂

x defined as an x directional gradient of the variable densities F(x,y,t) for each of the pixels;

calculating a variable density gradient fy=∂

F(x,y,t)/∂

y defined as a y directional gradient of the variable densities F(x,y,t) for each of the pixels;

calculating a variable density gradient ft=∂

F(x,y,t)/∂

t defined as a time-based variation of the variable densities F(x,y,t) for each of the pixels;

calculating covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt according to an equation

space="preserve" listing-type="equation">Sij=∫

∫

∫

(fi*fj)dxdydt (i,jε

{x,y,t})

for each of the pixels by utilizing the variable density gradients fx, fy and ft, integration ranges being limited in a space-time region of an observed pixel selected from the pixels and another space-time region of neighboring pixels positioned near to the observed pixel;

extracting first, second, third and fourth inherent characteristics of the dynamic image by utilizing the covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt, the first inherent characteristic being utilized to judge whether or not an observed pixel selected from the pixels is positioned in a texture drawn in the dynamic image, the second inherent characteristic being utilized to judge whether the texture in which the observed pixel is positioned is a one-dimensional texture changing in a direction only or a two-dimensional texture changing in an x-y plane, the third inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the one-dimensional texture changes in a motional change or a non-motional change, and the fourth inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the two-dimensional texture changes in the motional change or the non-motional change;

classifying one or more pixels into first or second classification type by utilizing the first to fourth inherent characteristics, the first classification type (S1-ACR) denoting that an observed pixel selected from the pixels is positioned in the one-dimensional texture and the luminous intensity of the observed pixel changes with time in a non-motional change, and the second classification type (S2-ACR) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel changes with time in the non-motional change;

recognizing first pixels classified into the first classification type and second pixels classified into the second classification type as boundary pixels positioned in a boundary region between the moving region and a static region of the dynamic image and setting a region occupied by the boundary pixels as a candidate region for a contour of the moving region;

detecting a contour of the moving region by adopting a line passing through a middle portion of the candidate region as the contour of the moving region;

distinguishing a moving object positioned in the moving region of which the contour is detected; and

processing the dynamic image in which the moving object distinguished exists to display the dynamic image in a displaying unit or transfer the dynamic image to an external unit.

2. A detecting method according to claim 1 in which the step of detecting a contour of the moving region includes:

dividing the candidate region into a plurality of small candidate regions;

calculating a gravity center of each of the small candidate regions divided;

calculating an inclination of each of the small candidate regions;

drawing a line segment through the gravity center at the inclination for each of the small candidate regions; and

connecting the line segment of the small candidate regions to each other, a line composed of the connected line segments being adopted as the contour of the moving region.

3. A detecting method according to claim 1 in which the step of extracting first, second, third and fourth inherent characteristics includes:

formulating the first inherent characteristic P1 by an equation

space="preserve" listing-type="equation">P1=(Sxx+Syy).sup.2 /{(Sxx+Syy).sup.2 +σ

.sub.μ

.sup.2 }

where the symbol σ

.sub.μ

denotes a constant for suppressing noise, formulating the second inherent characteristic P2 by an equation

space="preserve" listing-type="equation">P2={(Sxx-Syy).sup.2 +4*(Sxy).sup.2 }/{(Sxx+Syy).sup.2 +σ

.sub.μ

.sup.2 },

formulating the third inherent characteristic P3 by an equation ##EQU8## where the symbols σ

_s and σ

_t respectively denote a constant for suppressing noise, and

formulating the fourth inherent characteristic P4 by an equation ##EQU9## where a covariance matrix is defined by an equation ##EQU10##

4. A detecting method according to claim 3 in which the step of recognizing first pixels includes:

classifying the first pixels into the first classification type on condition that the first inherent characteristic P1 is equal to or higher than a first constant, the second inherent-characteristic P2 is lower than a third constant, the covariance Stt is equal to or higher than a second constant and the third inherent characteristic P3 is equal to or higher than a fourth constant, and

classifying the second pixels into the second classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is equal to or higher than the third constant, the covariance Stt is equal to or higher than the second constant and the fourth inherent characteristic P4 is equal to or higher than a fifth constant.

5. A detecting method according to claim 2 in which the inclination of a small candidate region accords with an inclination of an eigen vector Ev corresponding to a maximum eigen value λ

max² of a 2×

2 matrix as formulated by equations ##EQU11## where the symbol Xki (i=1,2, . . . N) denotes x coordinates of pixels ##EQU12## included in the small candidate region and the symbol Yki denotes y coordinates of the pixels.

6. A detecting method of a contour of a moving region of a dynamic image for distinguishing a moving object positioned in the moving region, comprising the steps of:

inputting a series of frames of dynamic image which consists of variable densities F(x,y,t) of pixels;

calculating a variable density gradient fx=∂

F(x,y,t)/∂

x defined as an x directional gradient of the variable densities F(x,y,t), a variable density gradient fy=∂

F(x,y,t)/∂

y defined as a y directional gradient of the variable densities F(x,y,t) and a variable density gradient ft=∂

F(x,y,t)/∂

t defined as a time-based variation of the variable densities F(x,y,t) for each of the pixels;

calculating covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt according to an equation

space="preserve" listing-type="equation">Sij=∫

∫

∫

(fi*fj)dxdydt (i,jε

{x,y,t})

for each of the pixels by utilizing the variable density gradients fx, fy and ft, integration ranges being limited in a space-time region of an observed pixel selected from the pixels and another space-time region of neighboring pixels positioned near to the observed pixel;

extracting first, second, third and fourth inherent characteristics of the dynamic image by utilizing the covariances Sxx, Sxy, Sxt, Syx, Sxy, Syt, Stx, Sty and Stt, the first inherent characteristic being utilized to judge whether or not an observed pixel selected from the pixels is positioned in a texture drawn in the dynamic image, the second inherent characteristic being utilized to judge whether the texture in which the observed pixel is positioned is a one-dimensional texture changing in a direction only or a two-dimensional texture changing in an x-y plane, the third inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the one-dimensional texture changes in a motional change or a non-motional change, and the fourth inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the two-dimensional texture changes in the motional change or the non-motional change;

classifying each of the pixels into first, second, third, fourth, fifth, sixth or seventh classification type by utilizing the first to fourth inherent characteristics, the first classification type (S0-FIX) denoting that an observed pixel selected from the pixels is not positioned in any texture or the luminous intensity of the observed pixel does not change with time, the second classification type (S1-FIX) denoting that the observed pixel is positioned in the one-dimensional texture and the luminous intensity of the observed pixel does not change with time, the third classification type (S1-MOT) denoting that the observed pixel is positioned in the one-dimensional texture and the luminous intensity of the observed pixel changes with time in a motional change, the fourth classification type (S1-ACR) denoting that the observed pixel is positioned in the one-dimensional texture and the luminous intensity of the observed pixel changes with time in a non-motional change, the fifth classification type (S2-FIX) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel does not change with time, the sixth classification type (S2-MOT) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel changes with time in the motional change, and the seventh classification type (S2-ACR) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel changes with time in the non-motional change;

re-classifying each of the pixels classified into the first, second, third or fourth classification type into the first, second third or fourth classification type to match a classification type of the observed pixel with classification types of neighboring pixels positioned near to the observed pixel;

recognizing first pixels classified into the seventh classification type and second pixels re-classified into the fourth classification type as boundary pixels positioned in a boundary region between the moving region and a static region of the dynamic image and setting a region occupied by the boundary pixels to a candidate region for a contour of the moving region; and

detecting a contour of the moving region by adopting a line passing through a middle portion of the candidate region set to the contour of the moving region.

7. A detecting method according to claim 6 in which the step of re-classifying each of he pixels includes:

re-classifying the first classification type of the observed pixel into the fourth classification type on condition that the number N(acr) of the neighboring pixels classified into the fourth or seventh classification type satisfies N(acr)>

N/3, N(acr)≧

N(fix) and N(acr)≧

N(mot), where the symbol N denotes the number of the neighboring pixels, the symbol N(fix) denotes the number of the neighboring pixels classified into the first, second or fifth classification type, and the symbol N(mot) denotes the number of the neighboring pixels classified into the third of sixth classification type,

re-classifying the second classification type of the observed pixel into the fourth classification type on condition that relations N(acr)>

N/3, N(acr)≧

N(fix) and N(acr)≧

N(mot) are satisfied,

re-classifying the third classification type of the observed pixel into the fourth classification type on condition that relations N(acr)>

N/3 and N(acr)>

N(mot) are satisfied, and

maintaining the fourth classification type of the observed pixel to the fourth classification type on condition that a first relation N(acr)>

0 is satisfied or second relations N(fix)>

0 and N(mot)>

0 are satisfied.

8. A detecting method according to claim 6 in which the step of detecting a contour of the moving region includes:

dividing the candidate region into a plurality of small candidate regions;

calculating a gravity center of each of the small candidate regions divided;

calculating an inclination of each of the small candidate regions;

drawing a line segment through the gravity center at the inclination for each of the small candidate regions; and

connecting the line segments of the small candidate regions to each other, a line composed of the connected line segments being adopted as the contour of the moving region.

9. A detecting method according to claim 6 in which the step of extracting first, second, third and fourth inherent characteristics includes:

formulating the first inherent characteristic P1 by an equation

space="preserve" listing-type="equation">P1=(Sxx+Syy).sup.2 /{(Sxx+Syy).sup.2 +σ

.sub.μ

.sup.2 }

where the symbol σ

.sub.μ

denotes a constant for suppressing noise, formulating the second inherent characteristic P2 by an equation

space="preserve" listing-type="equation">P2={(Sxx-Syy).sup.2 +4*(Sxy).sup.2 }/{(Sxx+Syy).sup.2 +σ

.sub.μ

.sup.2 },

formulating the third inherent characteristic P3 by an equation ##EQU13## where the symbols σ

_s and σ

_t respectively denote a constant for suppressing noise, and

formulating the fourth inherent characteristic P4 by an equation ##EQU14## where a covariance matrix is defined by an equation ##EQU15##

10. A detecting method according to claim 9 in which the step of classifying each of the pixels includes:

classifying the observed pixel into the first classification type on condition that the first inherent characteristic P1 is lower than a first constant and the covariance Stt is lower than a second constant,

classifying the observed pixel into the second classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is lower than a third constant and the covariance Stt is lower than the second constant,

classifying the observed pixel into the third classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is lower than the third constant, the covariance Stt is equal to or higher than the second constant and the third inherent characteristic P3 is lower than a fourth constant,

classifying the observed pixel into the fourth classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is lower than the third constant, the covariance Stt is equal to or higher than the second constant and the third inherent characteristic P3 is equal to or higher than the fourth constant,

classifying the observed pixel into the fifth classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is equal to or higher than the third constant and the covariance Stt is lower than the second constant,

classifying the observed pixel into the sixth classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is equal to or higher than the third constant, the covariance Stt is equal to or higher than the second constant and the fourth inherent characteristic P4 is lower than a fifth constant, and

classifying the observed pixel into the seventh classification type on condition that the first inherent characteristic P1 is equal to or higher than the first constant, the second inherent characteristic P2 is equal to or higher than the third constant, the covariance Stt is equal to or higher than the second constant and the fourth inherent characteristic P4 is equal to or higher than the fifth constant.

11. A detecting method according to claim 8 in which the inclination of a small candidate region accords with an inclination of an eigen vector Ev corresponding to a maximum eigen value λ

max² of a 2×

2 matrix as formulated by equations ##EQU16## where the symbol Xki (i=1,2, . . . N) denotes x coordinates of pixels included in the small candidate region and the symbol Yki denotes y coordinates of the pixels.

12. A software directed apparatus, detecting a contour of a moving region of a dynamic image for distinguishing a moving object positioned in the moving region, comprising:

inputting means for inputting a series of frames of dynamic image which consists of variable densities F(x,y,t) of pixels;

variable density gradient calculating means directed by the software for calculating;

a variable density gradient fx=∂

F(x,y,t)/∂

x defined as an x directional gradient of the variable densities F(x,y,t) input to the inputting means, for each of the pixels;

a variable density gradient fy=∂

F(x,y,t)/∂

y defined as a y directional gradient of the variable densities F(x,y,t) input to the inputting means for each of the pixels; and

a variable density gradient ft=∂

F(x,y,t)/∂

t defined as a time-based variation of the variable densities F(x,y,t) for each of the pixels;

covariance calculating means directed by the software for calculating covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt according to an equation

space="preserve" listing-type="equation">Sij=∫

∫

∫

(fi*fj)dxdydt (i,jε

{x,y,t})

for each of the pixels by utilizing the variable density gradients fx, fy and ft calculated by the variable density gradient calculating means under direction of the software, integration ranges being limited in a space-time region of an observed pixel selected from the pixels and another space-time region of neighboring pixels positioned near to the observed pixel;

inherent characteristic extracting means directed by the software for extracting first, second, third and fourth inherent characteristics of the dynamic image by utilizing the covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt calculated by the covariance calculating means under direction of the software, the first inherent characteristic being utilized to judge whether or not an observed pixel selected from the pixels is positioned in a texture drawn in the dynamic image, the second inherent characteristic being utilized to judge whether the texture in which the observed pixel is positioned is a one-dimensional texture changing in a direction only or a two-dimensional texture changing in an x-y plane, the third inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the one-dimensional texture changes in a motional change or a non-motional change, and the fourth inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the two-dimensional texture changes in the motional change or the non-motional change;

pixel classifying means directed by the software for classifying one or more pixels into first or second classification type by utilizing the first to fourth inherent characteristics extracted by the inherent characteristic extracting means under direction of the software, the first classification type (S1-ACR) denoting that an observed pixel selected from the pixels is positioned in the one-dimensional texture and the luminous intensity of the observed pixel changes with time in a non-motional change, and the second classification type (S2-ACR) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel changes with time in the non-motional change;

contour candidate limiting means directed by the software for recognizing first pixels classified into the first classification type and second pixels classified into the second classification type by the pixel classifying means as boundary pixels positioned in a boundary region between the moving region and a static region of the dynamic image and setting a region occupied by the boundary pixels as a candidate region for a contour of the moving region;

contour detecting means directed by the software for detecting a contour of the moving region by adopting a line passing through a middle portion of the candidate region as the contour of the moving region;

distinguishing means directed by the software for distinguishing a moving object positioned in the moving region of which the contour is detected; and

processing means for processing the dynamic image in which the distinguished moving object exists to provide an output indication thereof.

13. Apparatus according to claim 12, wherein said processing means for processing the dynamic image comprises means for displaying the dynamic image in a displaying unit.

14. Apparatus according to claim 12, wherein said processing means for processing the dynamic image comprises means for transferring the dynamic image to a unit external to an external apparatus.

15. A method of detecting a contour of a moving region of a dynamic image to be performed with the aid of a computer for distinguishing a moving object positioned in the moving region, comprising the steps of:

inputting a series of frames of dynamic image which consists of variable densities F(x,y,t) of pixels;

calculating with the aid of the computer a variable density gradient fx=∂

F(x,y,t)/∂

x defined as an x directional gradient of the variable densities F(x,y,t) for each of the pixels;

calculating with the aid of the computer a variable density gradient fy=∂

F(x,y,t)/∂

y defined as a y directional gradient of the variable densities F(x,y,t) for each of the pixels;

calculating with the aid of the computer a variable density gradient ft=∂

F(x,y,t)/∂

t defined as a time-based variation of the variable densities F(x,y,t) for each of the pixels;

calculating with the aid of the computer covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt according to an equation

space="preserve" listing-type="equation">Sij=∫

∫

∫

(fi*fj)dxdydt (i,jε

{x,y,t})

for each of the pixels by utilizing the variable density gradients fx, fy and ft, integration ranges being limited in a space-time region of an observed pixel selected from the pixels and another space-time region of neighboring pixels positioned near to the observed pixel;

extracting with the aid of the computer first, second, third and fourth inherent characteristics of the dynamic image by utilizing the covariances Sxx, Sxy, Sxt, Syx, Syy, Syt, Stx, Sty and Stt, the first inherent characteristic being utilized to judge whether or not an observed pixel selected from the pixels is positioned in a texture drawn in the dynamic image, the second inherent characteristic being utilized to judge whether the texture in which the observed pixel is positioned is a one-dimensional texture changing in a direction only or a two-dimensional texture changing in an x-y plane, the third inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the one-dimensional texture changes in a motional change or a non-motional change, and the fourth inherent characteristic being utilized to judge whether luminous intensity of the observed pixel positioned in the two-dimensional texture changes in the motional change or the non-motional change;

classifying with the aid of the computer one or more pixels into first or second classification type by utilizing the first to fourth inherent characteristics, the first classification type (S1-ACR) denoting that an observed pixel selected from the pixels is positioned in the one-dimensional texture and the luminous intensity of the observed pixel changes with time in a non-motional change, and the second classification type (S2-ACR) denoting that the observed pixel is positioned in the two-dimensional texture and the luminous intensity of the observed pixel changes with time in the non-motional change;

recognizing with the aid of the computer first pixels classified into the first classification type and second pixels classified into the second classification type as boundary pixels positioned in a boundary region between the moving region and a static region of the dynamic image and setting a region occupied by the boundary pixels as a candidate region for a contour of the moving region;

detecting with the aid of the computer a contour of the moving region by adopting a line passing through a middle portion of the candidate region as the contour of the moving region;

distinguishing with the aid of the computer a moving object positioned in the moving region of which the contour is detected; and

processing with the aid of the computer the dynamic image in which the moving object distinguished exists to provide an output indication thereof.

16. A method according to claim 15, wherein said step of processing the dynamic image to provide an output indication thereof comprises displaying the dynamic image in a displaying unit.

17. A method according to claim 15, wherein said step of processing the dynamic image to provide an output indication thereof comprises transferring the dynamic image to a unit external to an apparatus implementing the method with the aid of the computer.