Method and apparatus of recognizing a moving object

• US 5,511,133 A
• Filed: 12/28/1993
• Issued: 04/23/1996
• Est. Priority Date: 01/24/1991
• Status: Expired due to Fees

• Current Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation), Mitsubishi Electric Corporation
• Original Assignee: Mitsubishi Denki Kabushiki Kaisha (Mitsubishi Electric Corporation)
• Inventors: Shimizu, Masako, Tanaka, Ken-ichi
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): DEL ROSSO, GERARD DOMNICK
• Application Number:
  US08/174,622
• Time in Patent Office: 847 Days
• Field of Search: 382/14, 382/15, 382/22, 382/41, 382/155, 382/156, 382/157, 382/158, 382/199, 382/276, 395/21, 395/22, 395/24, 395/25
• US Class Current: 382/156
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

Disclosed is a moving object recognizing system, resistant to noise, which precisely obtains velocities, positions and configurations of an object moving in three-dimensional space. The system employs a bidirectional neural network including velocity neurons, coupled by twos and disposed, for respectively obtaining x-and-y components of an optical flow at respective points of the moving object and also line processes, interposed between the velocity neurons, for taking analog values for detecting edges of the moving object.

Claims

1. A method for recognizing a moving three-dimensional object and for computing a description of motion of the object, said method comprising the steps of:

detecting a physical property of the object to produce two-dimensional input image data;

applying the input image data to a bi-directional neural network comprising (I) pairs of velocity neurons for respectively obtaining x-and-y components of said optical flow and (ii) edge detecting line processes, each taking a value from a range between a high value and a low value in accordance with a function of the x and y components of optical flow and which is continuous within said range, interposed between said pairs of velocity neurons; and

obtaining an optical flow at respective points of said object based on the values taken by the line processes which optical flow provides the description of the motion of the object.

2. The method as set forth in claim 1, wherein said line process takes a value in accordance with a sigmoid function.

14. The method of claim 2, wherein the sigmoid function is downwardly convex and symmetrical with respect to the Y axis and is continuous, and nonmonotonic.

15. The method of claim 14, wherein the function of the first line process means is ##EQU5## wherein V (x) is the value taken by the line process, a is a constant and x is an input to the line process.

16. The method as set forth in claim 1, wherein said velocity neurons and said line process means are defined in a parallel computer system.

32. The method of claim 1, wherein the step of detecting a physical property of the object includes detecting visual characteristics of the object.

33. The method of claim 32, wherein the step of detecting visual characteristics of the object includes detecting the visual characteristics with a camera.

3. An apparatus for recognizing a moving object, comprising:

means for detecting a physical property of the object to produce a two-dimensional image;

means, connected to said means for projecting, for sampling the two-dimensional image at a plurality of points;

a bidirectional neural network, having an input connected to said sampling means, and including velocity neurons for obtaining X-components of an optical flow at the respective points from the sampling data at the respective points, and velocity neurons for obtaining Y-components of said optical flow at the respective points from the sampling data at the respective points;

first line process means, connected between said velocity neurons for obtaining said X-components, for detecting edges of said two-dimensional image and taking a value from a range between a low value and a high value according to a function of the X-components and which is continuous in the range; and

second line process means, connected between said velocity neurons for obtaining said Y-components, for detecting the edges of said two-dimensional image and taking a value from a range between a low value and a high value according to a function of the Y-components and which is continuous in the range.

4. The apparatus as set forth in claim 3, wherein both said line process means include means for taking values according to a signmid function.

5. The apparatus as set forth in claim 3 or 4, wherein said velocity neurons and said line process means are contained in a parallel computer system.

17. The apparatus of claim 4, wherein the sigmoid function is a continuous nonmonotonic function that is downwardly convex and symmetrical with respect to the Y axis.

18. The apparatus of claim 17, wherein the function of the first line process means is represented by the relationship ##EQU6## wherein V (x) is the value taken by the line process, a is a constant and x is an input to the line process.

34. The apparatus of claim 3, wherein the means for detecting a physical property of the object includes means for detecting visual characteristics of the object.

35. The apparatus of claim 34, wherein the means for detecting visual characteristics includes a camera.

6. An apparatus for recognizing a moving three-dimensional object, comprising:

means for detecting a physical property of the object to produce a two-dimensional image of the object wherein the image comprises sample data taken at a plurality of points in a two-dimensional plane;

a bi-directional neural network connected to receive said sample data and including, for each point in the sample data, an X-component velocity neuron for obtaining an X-component of an optical flow at the point, and a Y-component velocity neuron, coupled to the X-component velocity neuron, for obtaining a Y-component of an optical flow at the point, and

first line process means, connected between said X-component velocity neurons for detecting edges of said two-dimensional image and taking a value from a range between a low value and a high value according to a function of the X-component and which is continuous within the range; and

second line process means, connected between said Y-component velocity neurons for detecting the edges of said two-dimensional image and taking a value from a range between a low value and a high value according to a function of the Y-component and which is continuous within the range.

7. The apparatus as set forth in claim 6, wherein both said line process means include means for taking values according to a sigmoid function.

8. The apparatus as set forth in claim 6 or 7, wherein said velocity neurons and said line process means are contained in a parallel computer system.

19. The apparatus of claim 7, wherein the sigmoid function is a continuous nonmonotonic function that in downwardly convex and symmetrical with respect to the Y axis.

20. The apparatus of claim 19, wherein the function of the first line process means is represented by the relationship ##EQU7## wherein V (x) is the value taken by the line process, a is a constant and x is an input to the line process.

36. The apparatus of claim 6, wherein the means for detecting a physical property of the object includes means for detecting visual characteristics of the object.

37. The apparatus of claim 36, wherein the means for detecting visual characteristics includes a camera.

9. A method for recognizing a three-dimensional moving object comprising the steps of:

detecting a physical property of the object to produce two-dimensional image data of the object, the data being comprised of a plurality of points;

determining an optical flow from the image data by using a bi-directional neural network including, for each point, an X-component velocity neuron for obtaining an X-component of an optical flow at the point, and a Y-component velocity neuron, for obtaining a Y-component of an optical flow at the point, and

detecting edges of the image data using line processes, each taking a value from a range between a low value and a high value according to a function of the x and y components and which is continuous within the range, said line processes being interposed between a plurality of velocity neurons.

21. The method as set forth in claim 9, wherein said line process takes a value in accordance with a sigmoid function.

22. The method of claim 21, wherein the sigmoid function is downwardly convex and symmetrical with respect to the Y axis and is continuous, and nomnonotonic.

23. The method of claim 22, wherein the function of the first line process means is ##EQU8## wherein V (x) is the value taken by the line process, a is a constant and x is an input to the line process.

24. The method as set forth in claim 9, wherein said velocity neurons and said line process means are defined in a parallel computer system.

38. The method of claim 9, wherein the step of detecting a physical property of the object includes detecting visual characteristics of the object.

39. The method of claim 38, wherein the step of detecting visual characteristics of the object includes detecting the visual characteristics with a camera.

10. An apparatus for recognizing a moving three-dimensional object comprising:

a camera interface for receiving a signal from a CCD camera providing a two-dimensional representation of the three-dimensional object;

an image memory for storing the two-dimensional representation; and

a central processing unit including means defining a bidirectional neural network including;

velocity neurons for obtaining X components of an optical flow from corresponding points in the two-dimensional representation,

velocity neurons for obtaining Y components of said optical flow from corresponding points in the two-dimensional representation, and

line processes connected between the velocity neurons for obtaining the X components, for detecting edges of the two-dimensional image and each having a value taken from a range between a low value and a high value according to a function of the x components of the optical flow and which is continuous within the range.

11. The apparatus as set forth in claim 10, wherein both said line process means include means for taking values according to a sigmoid function.

12. The apparatus as set forth in claim 10, wherein said velocity neurons and said line process means are contained in a parallel computer system which also contains the camera interface, the image memory and the central processing unit.

13. The apparatus as set forth in claim 11, wherein said velocity neurons and said line process means are contained in a parallel computer system which also contains the camera interface, the image memory and the central processing unit.

25. The apparatus of claim 11, wherein the sigmoid function is a continuous and nonmonotonic function that is downwardly convex and symmetrical with respect to the Y axis.

26. The apparatus of claim 25, wherein the function of the first line process means is represented by the relationship ##EQU9## wherein V (x) is the value taken by the line process, a is a constant and x is an input to the line process.

27. A method for determining optical flow of an object using a bidirectional neural network that is described by a general function, comprising the steps of:

detecting a physical property of the object to produce image data;

applying the image data to the bidirectional neural network;

computing velocity vectors from the image data;

computing statuses of line processes, as a function of the computed velocity vectors, of the bidirectional neural network;

determining whether the general function converges; and

when the general function does not converge, repeating both of the steps of computing and the step of determining, otherwise providing the velocity vectors and status of line processes as a description of the motion of the object.

28. The method of claim 27, wherein the velocity vectors are determined according to the following equation:

##EQU10## wherein u_i,j and v_i,j are the x component and y component respectively and l_i,j^h and l_i,j-1^v are the statuses of the line process in the horizontal and vertical directions respectively of the point i,j.

29. The method of claim 27, wherein a line process l_i,j has a status according to a function of its internal status m_i,j defined by ##EQU11## wherein γ

, C_m, C_nl and nl are constants, m_i,j and n_i,j are internal statuses of a line process in the horizontal and vertical directions at a point i, j, and V is the function of the line processes.

40. The method of claim 27, wherein the step of detecting a physical property of the object includes detecting visual characteristics of the object.

41. The method of claim 40, wherein the step of detecting visual characteristics of the object includes detecting the visual characteristics with a camera.

30. A bidirectional neural network, comprising:

a first set of velocity neurons, each for determining a component of flow in a first dimension of two-dimensional data;

a second set of velocity neurons, each for determining a component of said flow in a second dimension of the two-dimensional data;

line processes, each connected between two velocity neurons of the first set, for detecting edges of the two-dimensional image in the first dimension and each having a value taken from a first range between a low value and a high value according to a function of the components of the flow obtained by the velocity neurons, which function is continuous within the first range; and

line processes connected between the velocity neurons of the second set, for detecting edges of the two-dimensional image in the second dimension and each having a value taken from a second range between a low value and a high value according to a function of the components of the flow obtained by the velocity neurons, which function is continuous within the second range.

31. A method for determining a measure of flow in two-dimensional data of an object by using a bidirectional neural network, the bidirectional neural network having a general function and comprising a first set of velocity neurons adapted to determine a component of the flow in a first dimension, a second set of velocity neurons adapted to determine a component of the flow in a second dimension and line process elements, each connected between a pair of elements in one of the first and second dimensions and having a status determined according to a function of the components of the flow determined by the velocity neurons and which is continuous within a range between a high value and a low value, the method comprising the steps of:

detecting a physical property of the object to produce image data;

applying said image data to the velocity neurons to obtain velocity vectors;

establishing the statuses of the line processes;

evaluating the general function of the bidirectional neural network; and

when the general function converges, obtaining the measure of the flow from the established statuses of the line processes.

42. The method of claim 31, wherein the step of detecting a physical property of the object includes detecting visual characteristics of the object.

43. The method of claim 42, wherein the step of detecting visual characteristics of the object includes detecting the visual characteristics with a camera.