Apparatus and method for using a target based computer vision system for user interaction

• US 6,901,561 B1
• Filed: 10/19/1999
• Issued: 05/31/2005
• Est. Priority Date: 10/19/1999
• Status: Expired due to Term

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Kirkpatrick, Edward Scott, Liu, Lurng-Kuo, Kjeldsen, Frederik C. M.
• Primary Examiner(s): Luu, Sy D.
• Application Number:
  US09/420,796
• Time in Patent Office: 2,051 Days
• Field of Search: 715/700, 715/764, 715/781, 715/856, 715/863, 715/866, 715/589, 715/157, 715/158, 382/118, 382/155, 382/159, 382/162, 382/165, 700/47
• US Class Current: 715/863
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method and apparatus are provided for training a computer system to recognize specific actions of a user. This may include displaying an image of a user within a window on a screen. The window includes a target area. This may also include associating a first computer event with a first user action displayed in the target area and storing information in a memory device such that the first user action is associated with the first computer event.

Claims

1. A method of training a computer system to recognize a specific action of a user and associate the specific action with a computer event, said method comprising:

capturing a first image of a user within a first predetermined window in a video stream;

displaying said user image including said first predetermined window;

determining that a state of said first predetermined window is to be associated with a first computer event;

associating said first computer event with said state of said first predetermined window in response to said determination; and

storing information in a memory device regarding said association.

2. The method of claim 1, further comprising:

capturing a second image of said user within said first predetermined window; and

displaying said second image, wherein a first indication indicates that a change of state between said first and second images in said first predetermined window is to be associated with said computer event, and wherein said associating said first computer event with said state of said first predetermined window comprises associating said change of state with said first computer event.

3. The method of claim 2, wherein said change of state comprises a change of color in said first predetermined window.

4. The method of claim 3, wherein said associating said first computer event with said state of said first predetermined window comprises storing a summary of said colors in said first predetermined window.

5. The method of claim 2, wherein said change of state comprises a pattern of activity within said video stream which indicates a change of position of said image of said user within said first predetermined window.

6. The method of claim 1, wherein said first image includes an image of an object in said first predetermined window.

7. The method of claim 1, wherein said first computer event comprises a specific computer function to execute.

8. The method of claim 1, wherein said first predetermined window is one of a rectangular area, a circular area and a square area.

9. The method of claim 1, wherein said first computer event comprises a plurality of computer events.

10. The method of claim 1, further comprising positioning said predetermined window within said video stream.

11. The method of claim 1, wherein positioning said predetermined window comprises locating said predetermined window within said video stream using one of a user input command and a program command.

12. The method of claim 1, wherein said first image further comprises a second predetermined window, the method further comprising:

determining that a state of said second predetermined window is to be associated with a second computer event; and

associating said second computer event with a state of said second predetermined window, wherein said storing of information further comprises storing information in said memory device regarding said association of said second computer event with a state of said second predetermined window.

13. The method of claim 1, wherein said first computer event comprises a mouse click action.

14. A method of using a computer system having an image capture system that displays an image of a user on a display screen, said method comprising:

enabling said computer system to associate a state of a first predetermined window within a video stream with a first computer event;

capturing said state of said first predetermined window with said image capture system; and

performing said first computer event in response to said state of said first predetermined window being captured by said image capture system.

15. The method of claim 14, wherein said enabling comprises:

capturing said first image;

displaying said first image;

determining that a state of said first predetermined window is to be associated with a first computer event;

associating said first computer event with said state of said first predetermined window in response to a first indication; and

storing information in a memory device regarding said association of said first computer event with said state of said first predetermined window.

16. The method of claim 14, further comprising positioning said first predetermined window within said video stream.

17. The method of claim 15, further comprising:

producing a second predetermined window within said video stream;

capturing a second image including said second predetermined window which includes a user image;

determining that a state of said second predetermined window is to be associated with a second computer event;

associating said second computer event with said state of said second predetermined window in response to a second indication; and

storing information in said memory device such that said state of said second predetermined window is associated with said second computer event.

18. A system that associates a specific user action with a first computer command, said system comprising:

an image capture system that captures a first image of a user within a video stream including a first predetermined window, wherein the state of the first predetermined window indicates said specific user action;

an image display system that displays said first image captured by said image capture system on a display screen; and

a computer system that recognizes a state of said first predetermined window as corresponding to an association of said state of said first predetermined window to said first computer command.

19. The system of claim 18, wherein said computer system is adapted to perform a training phase to train the computer system to recognize said specific user action, said training phase comprising:

displaying said first image that is captured by said image capture system including said user image which includes said first predetermined window;

associating said first computer command with said state of said first predetermined window; and

storing information in a memory device regarding said association of said first computer command with said state of said first predetermined window.

20. The system of claim 19, wherein said training phase further comprises:

capturing a second image including a second predetermined window;

associating a second computer command with a state of said second predetermined window; and

storing information in said memory device regarding said association of said state of said second predetermined window with said second computer command.

21. A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for training a system to recognize specific user actions, said method comprising:

capturing a first image of a user within a first predetermined window;

displaying said first image including said user image in said first predetermined window;

determining that a state of said first predetermined window is to be associated with a first computer event;

associating said first computer event with said state of said first predetermined window in response to said determination; and

storing information in a memory device regarding said association.

22. A method of enabling a computer system to recognize specific actions of a user, said method comprising:

capturing an image of the user within a predetermined window in a video stream;

associating a computer event with a state of the predetermined window; and

storing information in a memory device regarding said association.

23. The method of claim 22, further comprising displaying said image including said predetermined window.

24. The method of claim 22, further comprising determining that said state of the predetermined window is to be associated with said computer event before associating said computer event with said state.

25. The method of claim 22, further comprising capturing another image of said user within said predetermined window.

26. The method of claim 25, further comprising displaying said another image in said predetermined window.

27. The method of claim 25, further comprising receiving an indication that a change of state between said image and said another image in said predetermined window is to be associated with said computer event.

28. The method of claim 27, wherein associating of said computer event with said state of said predetermined window comprises associating said change of state with said computer event.

29. The method of claim 28, wherein said change of state comprises a change of color in said predetermined window.

30. The method of claim 29, wherein said associating said computer event with said change of state comprises storing a summary of said colors in said predetermined window.

31. The method of claim 28, wherein said change of state comprises a pattern of activity within said video stream which indicates a change of position of said image of said user within said predetermined window.

32. The method of claim 22, wherein said image comprises an image of an object in said predetermined window.

33. The method of claim 22, wherein said computer event comprises a specific computer function to execute.

34. The method of claim 22, wherein said predetermined window comprises one of a rectangular area, a circular area, and a square area.

35. The method of claim 22, wherein said computer event comprises a plurality of computer events.

36. The method of claim 22, further comprising positioning said predetermined window within said video stream.

37. The method of claim 36, wherein said positioning said predetermined window comprises locating said predetermined window within said video stream using one of a user input command and a program command.

38. The method of claim 22, wherein said computer event comprises a mouse click action.