Moving body recognition apparatus

• US 5,625,702 A
• Filed: 04/25/1995
• Issued: 04/29/1997
• Est. Priority Date: 09/17/1991
• Status: Expired due to Fees

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Shiohara, Morito, Kamada, Hiroshi, Kubota, Takashi
• Primary Examiner(s): Razavi, Michael T.
• Assistant Examiner(s): Patel, Jayanti K.
• Application Number:
  US08/428,651
• Time in Patent Office: 735 Days
• Field of Search: 382/107, 382/103, 348/700
• US Class Current: 382/107
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A moving body recognition apparatus which recognizes a shape and movement of an object moving in relation to an image input unit by extracting feature points, e.g. a peak of the object and a boundary of color, each in said images captured at a plurality of instants in time for observation by the image input unit. The moving body recognition apparatus comprises an image input unit for capturing images of an object as a moving body having a rotation, a feature point extraction unit for extracting feature points from the images inputted by the image input unit, a feature point storage unit for storing known position data of the extracted feature points, and a shape/movement recognition unit for calculating the actual positions and movements of the feature points of the object in the images by using the known position data of the extracted feature points outputted from the feature point storage unit.

Claims

1. A moving body recognition apparatus for recognizing a movement of a moving object by positions of feature points in an image corresponding to said moving object, comprising:

image input means for capturing images of said moving object;

feature point extraction means for extracting feature points in said images captured by said image input means;

feature point position storage means for storing known position data corresponding to said extracted feature points; and

shape/movement recognition means for calculating actual positions of said extracted feature points and a movement of said object from said known position data of four of said extracted feature points, said four extracted feature points each being in three of said images captured at any three instances in time, under a condition indicating that said four feature points do not exist on a single plane, that an axis of rotation of said object is not parallel to a direction of an orthogonal projection of said object, said rotation representing said movement between any two of said three instances in time for observation, and that said rotation is not a rotation by an angle of one hundred and eight degrees around an axis parallel to a plane on which said orthogonal projection is made.

2. The moving body recognition apparatus according to claim 1, wherein said feature point extraction means comprises:

space filter means for extracting an outline image from said images captured by said image input means; and

edge point extraction means for extracting an edge point from said outline image and for storing it as feature point data in said feature point position storage means.

3. The moving body recognition apparatus according to claim 2, further comprising:

feature point position storage means for storing the actual positions of feature points outputted from said shape/movement recognition means; and

object movement storage means for storing the angle of rotation of an object outputted from said shape/movement recognition means.

4. The moving body recognition apparatus according to claim 3, configured as a computer system comprising:

a television camera in correspondence with said image input means;

image processing hardware including a space filter in correspondence with a feature point extraction means;

a first memory in correspondence with said feature point position storage means;

a central processing unit (CPU) in correspondence with said shape/movement recognition means;

a second memory in correspondence with said feature point position storage means and said object movement storage means; and

a sensor for outputting to said CPU known position data regarding the shape and movement of said object.

5. The moving body recognition apparatus according to claim 3, wherein:

said image input means captures image of said object monitored by a visual device of an automatically running vehicle or an image of said object inspected in an inspection of factory automation (FA); and

said feature point position storage means and said object movement storage means store the results of recognizing the movement of said object being monitored or inspected.

6. The moving body recognition apparatus according to claim 3, wherein said shape/movement recognition means comprises:

known data input means for receiving from an external sensor known data regarding movements and positions of said feature points of said object;

feature point position normalization means for obtaining as normalized known position data the relative positions of the other three feature points when one of said four feature points of said object each in said images captured at said three instances in time for observation falls on the origin of a three-dimensional coordinate space;

shape/movement determination means for determining an enablement or a disablement of recognizing said movements and positions of feature points of said object by using an output from said feature point position normalization means;

movement/shape calculation means, activated by a determination by said shape/movement determination means of an enablement of recognizing said movements and positions of said feature points of said object, for calculating an angle of rotation of said object around said origin and unknown position data other than said known position data of said four feature points of said object by using an output from said feature point position normalization means; and

a feature point position reconstruction means for outputting a movement of said object by combining an angle of rotation around said origin outputted from said shape/movement calculation means with said position data of said feature point falling on said origin in said images outputted from said feature point position normalization means, and for outputting positions of said four feature points by combining said unknown position data outputted from said movement/shape calculation means with said position data of four feature points in said images outputted from said feature point position normalization means.

7. The moving body recognition apparatus according to claim 6, further comprising external memory means for temporarily storing known data outputted from said external sensor on an offline basis, provided before an input to said known data input means.

8. The moving body recognition apparatus according to claim 6, wherein the whole processes of said shape/movement recognition means comprise:

a first step of determining whether or not said known data input means has received at least one of ##EQU161## where R and S are three-dimensional matrices, U is a two-dimensional orthogonal matrix,

u₁, u₂ and u₃ are not linearly independent,

v₁, v₂ and v₃ are not linearly independent, and

w₁, w₂ and w₃ are not linearly independent,

where the three-dimensional vectors u_i, v_i and w_i (i=1, 2,

     3) representing the relative positions of feature point i (three of four feature points not falling on the origin) from the origin on which feature point 0 falls respectively at said first, second and third instances in time for observation, are expressed as ##EQU162## ##EQU163## and where their relations are expressed as;

space="preserve" listing-type="equation">vi=R u.sub.i and w.sub.i =S u.sub.i (i=1, 2,

     3)

in which the Y axis is a direction perpendicular to an image plane, which is the XZ plane, and a pair or rotation matrices R and S represent three-dimensional rotations of said object around said origin from said first instance in time for observation respectively to said second and third instances in time for observation, which are expressed as ##EQU164## a second step, invoked when said shape/movement recognition means determines positively in said first step, of executing recognition disablement process {1} after said known data input means outputs a deactivation signal to said feature point position normalization means and recognition disablement data to said movement/shape calculation means;

a third step, invoked when said shape/movement recognition means determines negatively in said first step, of determining whether said feature point position storage means has stored positions of said four feature points 0, 1, 2 and 3 each in said images at three instances in time for observation, which is repeated until a positive determination is obtained;

a fourth step, invoked when said shape/movement recognition means determines positively in said third step, of having said feature point position storage means send an activation signal to said feature point position normalization means, thereby activating said feature point position normalization means;

a fifth step of having feature point position normalization means store positions of said four feature points 0, 1, 2 and 3 in said images stored in said feature point position storage means and obtain first and second components of vectors u_i, v_i and w_i as the X and Z coordinate values obtained by subtracting from the X and Z coordinate values of feature point i in said images the X and Z coordinate values of feature point 0 in said images, which falls on the origin of said XZ plane;

a sixth step of having said shape/movement determination means determine whether said first and second components u_i, v_i and w_i obtained by said feature point position normalization means satisfy all of next three conditions ##EQU165## a seventh step, invoked when said shape/movement determination means determines negatively in said sixth step, of having said shape/movement determination means output recognition disablement data to said movement/shape calculation means, thereby executing a recognition disablement process {1};

an eighth step, invoked when said shape/movement determination means determines positively in said sixth step, of having said shape/movement determination means activate said movement/shape calculation means for executing a process of said movement/shape calculation means by fixing one value of p in an expression ##EQU166## a ninth step of having movement/shape calculation means execute its process;

a tenth step of having said feature point position reconstruction means determine whether or not there is a set of results, calculated by said movement/shape calculation means, comprising a pair of rotation matrices R and S and three-dimensional vectors u₁, u₂, u₃, v₁, v₂, v₃, w₁, w₂ and w₃ satisfying said known data regarding said movement and said shape of said object received by said known data input means;

an eleventh step, invoked when said feature point position reconstruction means determines negatively in said tenth step, of having said feature point position reconstruction means execute a recognition disablement process {2};

a twelfth step, invoked when said feature point position reconstruction means determines positively in said tenth step, of having said feature point position reconstruction means store in said object movement storage means a pair of rotation matrices R and S satisfying said known data and the coordinate values of feature point 0 in said images outputted from said feature point position normalization means; and

a thirteenth step of having said feature point position reconstruction means store in said feature point position storage means third components of said three-dimensional vectors u₁, u₂, u₃, v₁, v₂, v₃, w₁, w₂ and w₃ satisfying said known data and coordinate values of feature points 0, 1, 2 and 3 in said images outputted from said feature point position normalization means, thereby ending the whole processes.

9. The moving body recognition apparatus according to claim 8, wherein said movement/shape calculation means:

selects a value of p (1≦

p≦

) such that ##EQU167## becomes regular, if necessary by rotating the XY coordinate system by other than an integer times of ninety degrees;

obtains a value of h (1≦

h≦

2), such that a_h3 ≠

0;

defines the values of α

_i (1≦

i≦

2), β

(=[β

₁, β

₂ ]) and Γ

(=[Γ

₁, Γ

₂, Γ

₃ ]) by using expressions ##EQU168##

space="preserve" listing-type="equation">β

=[a.sub.11 a.sub.23 -a.sub.21 a.sub.13, a.sub.12 a.sub.23 -a.sub.22 a.sub.13 ], and Γ

=[a.sub.23, -a.sub.13, 0]

calculates the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ of rotation matrix R;

calculates the values of element r₁₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.13 =±

(1-r.sub.11.sup.2 -r.sub.12.sup.2).sup.1/2 ;

determines whether or not the value of element r₁₃ is zero;

calculates the value of element r₂₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.23 =-β

.sub.1 r.sub.13 /β

.sub.2, if r.sub.13 =/0;

sends a set of w_i to feature point position reconstruction means;

calculates the values of element r₂₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.23 =±

(1-r.sub.21.sup.2 -r.sub.22.sup.2).sup.1/2, if r.sub.13 =0;

calculates the values of elements in p-th row of rotation matrix S by using an expression ##EQU169## calculates the third row a₃ of matrix a as ##EQU170## calculates third components of three-dimensional vectors u₁, u₂ and u₃ by using an expression ##EQU171## calculates the q-th row s_q of rotation matrix S(1≦

a≦

2, q≠

p) ##EQU172## calculates the values of elements in the third rows R₃ and S₃ of a pair of rotation matrices R and S, r_3j =(-1)^3+j det R_3j and s_3j =(-1)^3+j det S_3j (1<

j<

3);

calculates the values of third elements v_i3 and w_i3 of three-dimensional vectors v_i and w_i by using expressions v_i3 =r₃ u_i and w_i3 =s₃ u_i (1≦

i≦

3); and

sends to feature point position reconstruction means the calculated set of rotation matrices R and S and three-dimensional vectors u_i, v_i and w_i.

10. The moving body recognition apparatus according to claim 9, wherein said movement/shape calculation means calculates the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ of said rotation matrix R, by a method comprising:

a step 1 of determining whether or not the values of elements a₁₃ and a₂₃ in said matrix a are zero;

a step 2, invoked when said movement/shape calculation means determines in step 1 that neither the value of element a₁₃ nor the value of element a₂₃ in said matrix a is zero, of calculating the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ by using expressions ##EQU173## a step 3, invoked when said movement/shape calculation means determines in step 1 that either the value of element a₁₃ or the value of element a₂₃ in said matrix a is zero, of determining whether or not a₁₃ is zero;

a step 4, invoked when said movement/shape calculation means determines in step 3 that the value of element a₁₃ in said matrix a is zero, of calculating the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ by using expressions ##EQU174## a step 5, invoked when said movement/shape calculation means determines in step 3 that the value of element a₁₃ in said matrix a is not zero, of calculating the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ by using expressions ##EQU175##

11. The moving body recognition apparatus according to claim 9, wherein said recognition disablement process {1} comprises:

a step of having said movement/shape calculation means send to said feature point position reconstruction means recognition disablement data; and

a step of having said feature point position reconstruction means store said recognition disablement data in said object movement storage means and said feature point position storage means, for a process completion.

12. The moving body recognition apparatus according to claim 9, wherein said recognition disablement process {2} comprises:

a step of having said feature point position reconstruction means store recognition disablement data in said object movement storage means and said feature point position storage means, for a process completion.

13. The moving body recognition apparatus according to claim 8, wherein said movement/shape calculation means:

selects a value of p (1≦

p≦

2) such that ##EQU176## becomes regular, if necessary by rotating the XY coordinate system by other than an integer times of ninety degrees;

obtains a value of h (1≦

h≦

2), such that a_h3 ≠

0;

defines the values of α

₁ (1≦

i≦

2), β

(=[β

₁, β

₂ ]) and Γ

(=[Γ

₁, Γ

₂, Γ

₃ ]) by using expressions ##EQU177##

space="preserve" listing-type="equation">β

=[a.sub.11 a.sub.23 -a.sub.21 a.sub.13, a.sub.12 a.sub.23 -a.sub.22 a.sub.13 ], and Γ

=[a.sub.23, -a.sub.13, 0]

calculates the values of elements r₁₁, r₁₂, r₂₁ and r₂₂ of rotation matrix R;

calculates one of the value of element r₁₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.13 =±

(1-r.sub.11.sup.2 -r.sub.12.sup.2).sup.1/2,

determines whether or not the value of element r₁₃ is zero;

calculates the value of element r₂₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.23 =-β

.sub.1 r.sub.13 /β

.sub.2 if r.sub.13 ≠

0;

sends a set of w_i to feature point position reconstruction means;

calculates one of the value of element r₂₃ of rotation matrix R by using an expression

space="preserve" listing-type="equation">r.sub.23 =±

(1-r.sub.21.sup.2 -r.sub.22.sup.2).sup.1/2, if r.sub.13 =0;

calculates the values of elements in p-th row of rotation matrix S by using an expression ##EQU178## calculates the third row a₃ as of matrix a as ##EQU179## calculates third components of three-dimensional vectors u₁, u₂ and u₃ by using an expression ##EQU180## calculates the q-th row S_q of rotation matrix S (1≦

q≦

2, q≠

p) ##EQU181## calculates the value of elements in the third row R₃ and S₃ of a pair of rotation matrices R and S, r_3j =(-1)^3+j det R_3j and s_3j =(-1)^3+j det S_3j (1≦

j≦

3);

calculates the values of third elements v_i3 and w_i3 of three-dimensional vectors v_i and w_i by using expressions v_i3 =r₃ u_i and w_i3 =s₃ u_i (1≦

i≦

3);

sends to feature point position reconstruction means the calculated set of rotation matrices R and S and three-dimensional vectors u_i, v_i and w_i ;

has said feature point position reconstruction means calculate a pair of rotation matrices R'"'"' and S'"'"' inverse to said pair of rotation matrices R and S by using expressions ##EQU182## and has said feature point position reconstruction means calculate the mirror image transformations v_i '"'"', u_i '"'"' and w_i '"'"' (i=1, 2,

     3) of three-dimensional vectors v_i, u_i and w_i (i=1, 2,

     3) with respect to the XZ plane by using expressions ##EQU183## (where i=1, 2,

     3).

14. A moving body recognition apparatus for recognizing a movement of a moving object by positions of feature points in images of said moving object, comprising:

image input means for capturing images of said moving object;

feature point extraction means for extracting feature points in said images captured at any three instances in time by said image input means;

feature point position storage means for storing known position data of said extracted feature points; and

movement recognition means for calculating a movement of said object from said known position data of four of said extracted feature points of said object each in said images captured by said image input means, under a condition indicating that said four extracted feature points do not exist on a single plane, that an axis of rotation of said object is not parallel to a direction of an orthogonal projection of said object, said rotation representing said movement between any two of said three instances in time for observation, and that said rotation is not a rotation by an angle of one hundred and eighty degrees around an axis parallel to a plane on which said orthogonal projection is made.