Automated visual tracking for computer access

• US 20020039111A1
• Filed: 06/27/2001
• Published: 04/04/2002
• Est. Priority Date: 06/27/2000
• Status: Abandoned Application

• Current Assignee: Boston College
• Original Assignee: Boston College
• Inventors: Gips, James, Betke, Margrit
• Application Number:
  US09/892,254
• Publication Number:
  US 20020039111A1
• Time in Patent Office: Days
• US Class Current: 345/700
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements

Abstract

The invention comprises a system and method for permitting a computer user or the user of a system comprising a video display to control an indicator, such as a mouse pointer or cursor, on a computer monitor or video display screen. The system and method uses a video camera pointed at the user to capture the user'"'"'s image. The location in the video camera field of view of a chosen feature of the user'"'"'s image is used to control the location of the indicator on the monitor or display screen. Thus, by control of the motion of the chosen feature, which for example may be the user'"'"'s nose, the user may control or provide input into a computer program, video game or other device or system.

Claims

1. A method for providing input to a system which uses a visual display for providing user information, comprising:

(a) choosing a feature associated with a system user;

(b) determining a location of the feature in a video image from a video camera at an initial time;

(c) determining a subsequent location of the feature in a video image from the video camera at a subsequent given time; and

(d) providing input to the system at the subsequent given time based upon the location of the feature in the video image at the subsequent given time.

2. The method of claim 1, wherein in the step of choosing, the feature associated with a systems user includes one of a body, face, or article of clothing.

3. The method of claim 1 wherein in the step of choosing the feature includes a portion of a substance or device affixed to the system user.

4. The method of claim 1, wherein the step of providing input includes providing vertical and horizontal coordinates.

5. The method of claim 4, wherein the vertical and horizontal coordinates are used as a basis for locating an indicator on the video display being used by the system to display material for the user.

6. The method of claim 5, wherein locating an indicator includes determining the indicator location at the given time based upon a location of the indicator at a previous time, and a change between a location of the feature in the video image at the previous time and the location of the feature in the video image at the given time.

7. The method of claim 5, wherein the indicator location is determined at the given time based upon the location of the feature in the video image at the given time independent of previous indicator locations.

8. The method of claim 4, wherein the vertical and horizontal coordinates are used as a basis for determining a direction of movement of an indicator on a video display being used by the system to display material for the user.

9. The method of claim 4, wherein the vertical and horizontal coordinates are used as a basis for determining a direction of movement of a background image on a video display screen being used by the system to display material for the user, as an indicator on the video display screen remains in a fixed position.

10. The method of claim 1, wherein the system is a computer program.

11. The method of claim 1, wherein the input is provided in response to the location of the feature in the video image changing by less than a defined amount during a defined period of time.

12. The method of claim 11, wherein:

(a) the input provided is selected from a group consisting of letters, numbers, spaces, punctuation marks, other defined characters and signals associated with defined actions to be taken by the system; and

(b) the selection of the input is determined by the location of the feature in the video image.

13. The method of claim 1, wherein the input provided is based upon a change in the location of the feature in the video image between a previous time and the given time.

14. The method of claim 1, wherein the input provided at the given time is an affirmative signal or a negative signal based on whether the motion of the feature in the video image is in a vertical direction or a horizontal direction prior to the given time.

15. The method of claim 10, wherein:

(a) the computer program is running on a first computer; and

(b) the locations of the feature in the video images are determined by a second computer.

16. The method of claim 10, wherein:

(a) the computer program is running on a computer; and

(b) the locations of the feature in the video images are determined by the computer.

17. The method of claim 10, wherein:

(a) the computer program is running on a computer; and

(b) the locations of the feature in the video images are determined by a video acquisition board on the computer.

18. The method of claim 10, wherein the computer program is a Web browser.

19. The method of claim 1, wherein determining the location of the feature in the video image at the given time further comprises:

(a) choosing a fixed area of a video image from a prior time, the fixed area containing the chosen feature at a known point therein;

(b) comparing video input signals for specified trial areas of the video image at the given time with video input signals for the fixed area of the video image from the prior time;

(c) choosing the trial area most similar to the fixed area based on the compared video input signals; and

(d) selecting as the location of the feature in the video image at the given time, a point within the chosen trial area bearing the same relationship to the chosen trial area as the known point does to the fixed area.

20. The method of claim 19, wherein the known point and the point within the chosen trial area are located at the center of the fixed area and the chosen trial area, respectively.

21. The method of claim 19 wherein choosing the trial area comprises calculating normalized correlation coefficients between the video input signals for the fixed area and for each specified trial area.

22. The method of claim 21 wherein the video input signals are greyscale intensity signals.

24. The method of claim 23 wherein the feature is a portion of the system user'"'"'s body, face, or article of clothing.

25. The method of claim 23 wherein the feature is a portion of a substance or device affixed to the system user'"'"'s body, face, or article of clothing.

26. The method of claim 23, further comprising iteratively repeating steps (d), (e) and (f) with the successive pixel of one iteration used as the base pixel for the next iteration.

27. The method of claim 23, wherein choosing the successive pixel further comprises:

(a) creating a base template of pixels associated with the base pixel;

(b) selecting a window of trial pixels surrounding the base pixel;

(c) iteratively creating a trial template associated with each trial pixel, the trial template bearing the same relationship to the trial pixel as the base template does to the base pixel; and

(d) choosing as the successive pixel the trial pixel whose trial template most closely corresponds to the base template.

28. The method of claim 27, wherein choosing the successive pixel further comprises:

(a) determining a base greyscale intensity of the base template;

(b) determining a trial greyscale intensity of each trial template; and

(c) comparing each trial greyscale intensity with the base greyscale intensity.

29. The method of claim 28, wherein comparing the greyscale intensities further comprises calculating correlation coefficients for the base template with each trial template.

30. The method of claim 23, wherein:

(a) the feature comprises a plurality of sub-features;

(b) the base pixel is determined from a plurality of sub-base pixels, each sub-base pixel corresponding to a location of one of the sub-features;

(c) the successive pixel is determined from a plurality of sub-successive pixels, each sub-successive pixel corresponding to a location of one of the sub-features in the successive video image; and

(d) the successive pixel is determined from the sub-successive pixels by a same calculation as the base pixel is determined from the sub-base pixels.

31. The method of claim 30, wherein the base and successive pixels are a weighted average of the locations of the sub-base and sub-successive pixels, respectively.

32. The method of claim 23, wherein controlling the system input further comprises providing data signals to an input device of the system.

33. The method of claim 23, wherein the system is a computer program.

34. The method of claim 23, wherein controlling the input to the system comprises providing vertical and horizontal coordinates.

35. The method of claim 34, wherein the vertical and horizontal coordinates are used as a basis for locating an indicator on a video display being used by the system to display material for the user.

36. The method of claim 35, wherein the indicator location is determined at a given time based upon a location of the indicator at a previous time, and a difference between the locations of the base pixel and the successive pixel at the given time.

37. The method of claim 35, wherein the indicator location is determined at a given time based upon the location of the successive pixel at the given time independent of a previous indicator location.

38. The method of claim 34, wherein the vertical and horizontal coordinates are used as a basis for determining a direction of movement of an indicator on a video display being used by the system to display material for the user.

39. The method of claim 34, wherein the vertical and horizontal coordinates are used as a basis for determining a direction of movement of a background image on a video display screen being used by the system to display material for the user, as an indicator on the video display screen remains in a fixed position.

40. The method of claim 23, wherein the input is controlled in response to the locations of the base and successive pixels differing by less than a defined amount over a defined period of time.

41. The method of claim 40, wherein controlling the input further comprises selecting the input to the system from a group consisting of letters, numbers, spaces, punctuation marks, other defined characters and signals associated with defined actions to be taken by the system, the selection of the input being determined by the location of the successive pixel.

42. The method of claim 23, wherein the input to the system is controlled based upon the differences between the locations of the base and successive pixels.

43. The method of claim 23, wherein the input to the system is an affirmative signal or a negative signal based on whether the difference between the locations of the base and successive pixels defines a vertical or a horizontal motion.

23. A method of providing input to a system which uses a visual display for providing user information, comprising:

(a) capturing a first video image of at least a part of a system user;

(b) choosing a feature in the first video image associated with the user;

(c) choosing a base pixel corresponding to a location of the chosen feature in the first video image;

(d) capturing a successive video image of at least the part of the user;

(e) choosing a successive pixel corresponding to the location of the chosen feature in the successive video image; and

(f) controlling the input to the system based on the location of the base pixel and the successive pixel.

44. A system for providing input to a computer by a user, comprising:

(a) a video camera for capturing video images of at least a part of the user and outputting video signals corresponding to the video images;

(b) a tracker for receiving the video output signals from the camera and outputting data signals corresponding to a feature associated with the user; and

(c) a driver for receiving the data signals and controlling an input device of the computer in response thereto.

45. The system of claim 44, wherein the tracker further comprises:

(a) a video acquisition board for digitizing the output signals from the video camera;

(b) a memory for storing the digitized output signals as image data; and

(c) at least one processor for comparing stored image data, determining a location of the feature in the video images and generating data signals based on the determined locations.

46. The system of claim 45, wherein the at least one processor further comprises computer-readable medium containing instructions for controlling a computer system to compare the stored image data and determine the location of the feature, by:

(a) choosing stored image data of a fixed area of a prior video image, the fixed area containing the feature as a known position therein;

(b) comparing stored image data of specified trial areas of a subsequent video image with the stored image data of the fixed area;

(c) choosing the trial area most similar to the fixed area based on the compared image data; and

(d) selecting as the location of the feature in the subsequent video image, a point within the chosen trial area bearing the same relationship to the chosen trial area as the known point does to the fixed area.