Method and apparatus for automatically tracking an object

• US 5,323,470 A
• Filed: 05/08/1992
• Issued: 06/21/1994
• Est. Priority Date: 05/08/1992
• Status: Expired due to Fees

• Current Assignee: Vanderbilt University
• Original Assignee: Vanderbilt University
• Inventors: Kawamura, Kazuhiko, Kara, Atsushi
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): Fox, David
• Application Number:
  US07/880,537
• Time in Patent Office: 774 Days
• Field of Search: 382/1, 382/2, 382/42, 382/48, 358/125, 358/126, 364/413.02, 364/413.01, 364/413.13, 395/94, 395/924, 901/47
• US Class Current: 382/103
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/39387 Reflex control, follow movement, track face, work, hand, visual servoing
• CPC Class Codes: G05B 2219/45108 Aid, robot for aid to, assist human disabled
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

Methods and apparatus for automatically tracking the position of a moving object in real time, particularly the face of a person who is being fed by a robotic system, is disclosed. The object can be tracked by comparing a prestored object model image with the current image of the object using the square-distance criteria. The search area can be limited to a region in which the face is most likely to be found and the prestored object model image can be limited to robust points. The method can include motion prediction, including both continuous motion and sudden motion, such as the motion cause by a person sneezing. Alternatively, a computationally efficient approach employing a one-dimensional algorithm can be used.

Claims

1. A method of automatically tracking the position of a face of a patient in real time to allow controlled feeding of the patient by a robot, comprising the steps of:

A) positioning a camera such that the camera viewing space defines a two-dimensional image coordinate system;

B) automatically acquiring a model image which includes an image of the face, wherein the model image comprises a first two-dimensional array of intensity points referenced to the coordinate system;

C) selecting an initial tracking point in the model image and an initial prediction point in the model image, wherein the tracking point and the prediction point are referenced to the coordinate system;

D) at preselected times, repeating steps E) through H) until feeding is completed;

E) for each preselected time, automatically acquiring a scene image which includes an image of the face, wherein the scene image comprises a second two-dimensional array of intensity points referenced to the coordinate system;

F) automatically selecting a rectangular search region within the scene image, wherein the search region comprises a subset of the second two-dimensional array of intensity points and includes the intensity point corresponding to the prediction point;

G) automatically locating the best match point between the second intensity points in the search region and the first intensity points in the model image, wherein the best match point corresponds to the minimum of the square-distance criterion and automatically equating the best match point with the tracking point for the particular preselected time; and

H) automatically comparing the tracking point for the particular preselected time to the tracking point of a previous preselected time to select the prediction point for the next preselected time.

2. The method of claim 1, wherein the size of the search region is optimized to enhance the speed at which the position of the face can be tracked.

3. The method of claim 1, further comprising for each cycle of repeated steps E) through G) the following steps of:

i) storing the value of the square distance criterion associated with each first intensity point in the model image for the particular preselected time;

ii) selecting the set of first intensity points in the model image in which the value of the square distance criterion associated with each first intensity point remains less than a predetermined threshold value for a predetermined set of preselected times to create an array of robust intensity points; and

iii) limiting the model image first two-dimensional array to the robust intensity points to enhance the speed at which the position of the face can be tracked.

4. The method of claim 1, wherein the method of comparing the tracking point for the particular preselected time to the tracking point of a previous preselected period of time is selected from the group consisting of an autoregressive method, a constant velocity method, a constant acceleration method and a least-squares method.

5. The method of claim 1, further comprising the step of predicting sudden motion of the face by correlating a time series of at least one variable associated with sudden motion in a time-delay neural network trained with a back-propagation algorithm.

6. The method of claim 5, wherein the variable is the minimum of the square-distance criterion.

8. The method of claim 5 wherein a time series is stored in the time-delay neural network in a shift register having a fixed length T and having T input nodes and one output node.

7. A method controlling in real time the position of a robotic feeding means delivering food to a patient, comprising the steps of:

A) automatically acquiring a model image which includes an image of the face of the patient surrounded by a background, wherein the model image comprises a two-dimensional array of intensity points such that the intensity values of the intensity points corresponding to the background distinguishably contrast with the intensity values of the intensity points corresponding to the edge of the face so that the intensity points corresponding to the edge of the face can be determined by intensity thresholding;

B) selecting an initial tracking point (X_init, Y_init) in the model image such that the initial tracking point corresponds to a position within the image of the face;

C) automatically examining the horizontal line of intensity points containing the initial point to identify the intensity point X_min, which corresponds to the intensity point that identifies the right edge of the face on the horizontal line, and to identify the intensity point X_max, which corresponds to the intensity point that identifies the left edge of the face on the horizontal line, and to identify X_track, which equals (X_min +X_max)/2;

D) automatically examining the vertical line of intensity points containing the intensity point (X_track, Y_init) to identify the intensity point Y_min, which corresponds to the intensity point that identifies the top edge of the face on the vertical line;

E) setting Y_track =Y_min +D, wherein D is a predefined constant related to the distance between the top edge of the face and the mouth of the face, to define the tracking point (X_track, Y_track);

F) controlling the position of a robotic feeding means using the tracking point; and

G) at preselected times, repeating steps A) through F) to track the position of the face of the patient until the patient indicates that feeding is completed.

9. The method of claim 7, wherein the patient indicates that feeding is completed by issuing a voice command to a speech recognition system.

10. The method of claim 7 further comprising the step of estimating the distance of the face by reading a horizontal line at Y_track, measuring a length L between the first white-to-black edge and the last black-to-white edge corresponding to the width of the head, and calculating the distance of the face from the distance L.

11. A method of automatically controlling in real time the position of a robotic feeding means delivering food to a patient, comprising the steps of:

A) positioning a camera such that the camera viewing space defines a two-dimensional image coordinate system;

B) automatically acquiring a model image which includes an image of the patient'"'"'s face, wherein the model image comprises a first two-dimensional array of intensity points referenced to the coordinate system;

C) selecting an initial tracking point in the model image and an initial prediction point in the model image, wherein the tracking point and the prediction point in the model image, wherein the tracking point and the prediction point are referenced to the coordinate system;

D) at preselected times, repeating steps E) through I) until the patient indicates that feeding is completed;

E) for each preselected time, automatically acquiring a scene image which includes an image of the face, wherein the scene image comprises a second two-dimensional array of intensity points referenced to the coordinate system;

F) automatically selecting a rectangular search region within the scene image, wherein the search region comprises a subset of the second two-dimensional array of intensity points and includes the intensity point corresponding to the prediction point;

G) automatically locating the best match point between the second intensity points in the search region and the first intensity points in the model image, wherein the best match point corresponds to the minimum of the square distance criterion and automatically equating the best match point with the tracking point for the particular preselected time;

H) using the best match point to control the position of a robotic feeding means; and

I) automatically comparing the tracking point of the particular preselected time to the tracking point of a previous preselected time to select the prediction point for the next preselected time.

12. The method of claim 11, wherein the size of the search region is optimized to enhance the speed at which the position of the face can be tracked.

13. The method of claim 11, further comprising for each cycle of repeated steps E) through G) the following steps of:

i) storing the value of the square distance criterion associated with each intensity point in the model image for the particular preselected time;

ii) selecting the set of intensity points in the model image in which the value of the square distance criterion associated with each intensity point remains less than a predetermined threshold value for a predetermined set of preselected times to create an array of robust intensity points; and

iii) limiting the model image two-dimensional array to the robust intensity points to enhance the speed at which the position of the face can be tracked.

14. The method of claim 11, wherein the method of comparing the tracking point for the particular preselected time to the tracking point of a previous preselected period of time is selected from the group consisting of an auto-regressive method, a constant velocity method, a constant acceleration method, and a least squares method.

15. The method of claim 11, further comprising the step of predicting sudden motion of the face by correlating a time series of at least one variable associated with sudden motion in a time-delay neural network trained with a back-propagation algorithm.

16. The method of claim 15, wherein the variable is the minimum of the square-distance criterion.

17. The method of claim 11, wherein the patient speaks into a voice recognition means to indicate that feeding is complete.

18. The method of claim 15 wherein a time series is stored in the time-delay neural network in a shift register having a fixed length T and having T input nodes and one output node.

19. A robotic feeding apparatus for feeding a patient comprising:

camera means for acquiring a model image which includes an image of the patient'"'"'s face, wherein the model image comprises a first two-dimensional array of intensity points referenced to a two-dimensional coordinate system;

camera positioning means for positioning the camera means to define said two-dimensional coordinate system;

point selection means for selecting an initial tracking point in the model image and an initial prediction point in the model image, wherein the tracking point and the prediction point are referenced to the coordinate system;

camera control means for controlling the camera means such that, at preselected times, the camera acquires a scene image which includes an image of the patient'"'"'s face, wherein the scene image comprises a second two-dimensional array of intensity points referenced to the coordinate system;

rectangle selection means for automatically selecting a rectangular search region within the scene image, wherein the search region comprises a subset of the second two-dimensional array of intensity points and includes the intensity point corresponding to the prediction point;

match point locating and equating means for automatically locating the best match point between the second intensity points in the search region and the first intensity points in the model image, wherein the best match point corresponds to the minimum of the square distance criterion, the match point locating and equating means also for automatically equating the best match point with the tracking point for the particular preselected time;

robotic food delivery means for delivering food to the patient'"'"'s mouth in accordance with the location of the tracking point;

prediction point selection means for automatically comparing the tracking point for the particular preselected time to the tracking point of a previous preselected time to select the prediction point for the particular preselected time; and

repetition and sequencing means for receiving commands from the patient, and, in response thereto, controlling the acquisition of the scene image by the camera means, the selection of the rectangular search region by the rectangle selection means, the locating of the best match point and the equating of the best match point with the tracking point by the match point locating and equating means, the delivery of food to the patient'"'"'s mount by the robotic food delivery means, and the selection of the prediction point by the prediction point selection means.

20. The apparatus of claim 19, wherein the repetition and sequencing means comprises voice recognition means for receiving spoken commands from the patient.

21. The apparatus of claim 19, wherein the robotic food delivery means comprises a five degree-of-freedom manipulator, and microprocessor-based pneumatic control means for controlling the manipulator in accordance with the location of the tracking point.

22. The apparatus of claim 21, wherein the five degree-of-freedom manipulator comprises joints that are controlled by a plurality of transputers.

23. The apparatus of claim 19, wherein the camera means comprises a CCD camera.