Architecture for controlling a computer using hand gestures

• US 7,665,041 B2
• Filed: 03/25/2003
• Issued: 02/16/2010
• Est. Priority Date: 03/25/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Oliver, Nuria M., Wilson, Andrew D.
• Primary Examiner(s): Bashore; William L
• Assistant Examiner(s): NUNEZ, JORDANY
• Application Number:
  US10/396,653
• Publication Number:
  US 20040189720A1
• Time in Patent Office: 2,520 Days
• Field of Search: 715/863, 715/860
• US Class Current: 715/860
• CPC Class Codes: G06F 13/105 where the programme performs an input/output emulation function
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Architecture for implementing a perceptual user interface. The architecture comprises alternative modalities for controlling computer application programs and manipulating on-screen objects through hand gestures or a combination of hand gestures and verbal commands. The perceptual user interface system includes a tracking component that detects object characteristics of at least one of a plurality of objects within a scene, and tracks the respective object. Detection of object characteristics is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images. A seeding component iteratively seeds the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison. A filtering component selectively removes the tracked object from the object hypotheses and/or at least one object hypothesis from the set of object hypotheses based upon predetermined removal criteria.

Claims

1. A system that facilitates a user interface, comprising at least a processor executing the following components stored in a computer memory:

a tracking component that detects at least one of a plurality of objects within a scene and tracks at least one respective moving object selected from the plurality of objects if the respective object lies within the scene while the object is ignored when it lies outside the scene, detection of the object is based at least in part upon image comparison of a plurality of images relative to a course mapping of the images, the tracking component employs multiple object hypotheses utilizing multiple trackers comprising at least a first tracker and a second tracker to select and track the at least one moving object from the plurality of objects based on one or more criteria such that the second tracker continues to track the at least one moving object upon failure of the first tracker to track the at least one moving object, wherein the multiple object hypotheses are supported by frame-to-frame tracking through time in at least one view and stereo matching across two views, and wherein redundancy among the trackers is recognized by distance analysis, such that if a distance between the at least one moving object and a second moving object falls below a threshold distance the second object is removed from consideration;

a seeding component that iteratively seeds the tracking component with the object hypotheses based upon the presence of the object and the image comparison; and

a filtering component that selectively removes the tracked object from the object hypotheses or at least one object hypothesis from the object hypotheses, the tracked object removed based at least in part upon a region-based approach in determining depth to cursors and move windows.

2. The system of claim 1, the tracking component tracks the respective object according to predetermined criteria relating to object characteristics comprising at least one of object motion, object orientation, object features, and object rotation.

3. The system of claim 1, the tracking component tracking the object only if motion of the object meets a minimum threshold when the object is present within the scene.

4. The system of claim 1, the scene comprising a volume in space.

5. The system of claim 1, the tracking component detecting the object utilizing an imaging system comprising at least one imager.

6. The system of claim 1, the tracking component detecting the object utilizing an imaging system comprising at least two cameras whose rasters are in parallel.

7. The system of claim 1, the object being a hand of at least one user.

8. The system of claim 7, the hand controlled to provide at least one of a plurality of hand gestures that are detected and tracked by the tracking component.

9. The system of claim 8, the hand gestures comprising gesture characteristics detectable by the tracking component, which gesture characteristics include at least one of hand movement, finger count, finger orientation, hand orientation, and hand rotation.

10. The system of claim 1, the tracking component tracking the object in the scene by performing a frame comparison of frame images of the object over time.

11. The system of claim 1, the tracking component predicting the next location of the object.

12. The system of claim 1, a system processing device input data used in cooperation with the tracking component tracking the object.

13. The system of claim 12, the device input data including keyboard input data, mouse input data, and audio input data.

14. The system of claim 1, at least one of the tracking component and the filtering component including a support vector machine for classifying the object characteristics.

15. The system of claim 1, further comprising a user interface component for presenting a graphical response to a tracked object.

16. A system that facilitates a user interaction with a perceptual user interface, comprising at least a processor executing the following means:

means for tracking and detecting at least one of a plurality of objects within a scene based at least in part upon image comparison of a plurality of images relative to a course mapping of the images, the means for tracking and detecting runs multiple object hypotheses utilizing multiple trackers comprising at least a first tracker and a second tracker for tracking at least one moving object from the plurality of objects within the scene such that a second tracker tracks the at least one moving object upon failure of a first tracker to track the at least one moving object, wherein the multiple object hypotheses are supported by frame-to-frame tracking through time in at least one view and stereo matching across two views, wherein redundancy among the trackers is implemented by distance analysis, based on distance between the object and a second moving object, and wherein when the distance falls below a threshold distance the second object is removed from consideration;

means for storing the multiple object hypotheses;

means for iteratively seeding the tracking component with object hypotheses based upon the presence of the object characteristics and the image comparison; and

means for filtering that selectively removes the tracked object from the object hypotheses and at least one object hypothesis from the set of object hypotheses, the tracked object removed based at least in part upon a region-based approach in determining depth to cursors and move windows.

17. A system that facilitates a user interface, comprising at least a processor executing the following components stored in a computer memory:

a detecting component that detects at least one of a plurality of objects within a scene;

a tracking component that tracks a detected object wherein the detected object is selected for tracking by employing multiple hypotheses on the plurality of objects within the scene wherein at least two trackers are utilized to track the detected object such that a second tracker continues tracking the detected object upon failure of a first tracker, wherein the multiple hypotheses are supported by frame-to-frame tracking through time in at least one view and stereo matching across two views, and wherein redundancy between the at least two trackers is recognized by distance analysis such that when a distance between the detected object and a second moving object falls below a threshold distance the second object is removed from consideration;

a seeding component that iteratively seeds the tracking component with object hypotheses based upon the detected object that is tracked;

a filtering component that selectively removes the tracked object from the object hypotheses or at least one object hypothesis from the set of object hypotheses; and

an interpreting component that interprets an object characteristic of the tracked object and executes a command in response thereto.

18. The system of claim 17, the tracking component comprising a prediction algorithm to predict a next position of a tracked object of an object hypothesis.

19. The system of claim 18, the prediction algorithm comprising a Kalman filter.

20. The system of claim 17, the tracking component selecting a narrow range of points centered on the predicted next position to which a second algorithm is applied to calculate an actual next position.

21. The system of claim 20, the second algorithm determining an actual position by finding a location (u_t, v_t), within the narrow range that minimizes the sum of the absolute difference algorithm according to,

SAD(I_t-1,u_t-1,v_t-1,I_t,u_t,v_t)

where I_t-1 refers to an image at time t−

1, and I_t refers to an image at time t, and where (u_t, v_t) refers to a location at time t.

22. The system of claim 17, the object hypotheses maintained by frame-to-frame tracking through time in one of two views and stereo matching across both views.

23. The system of claim 17, the plurality of objects including a dominant object and a non-dominant object that are used to describe a spatial relationship.

24. The system of claim 17, further comprising the utilization of speech recognition to effect control of the system either separately or in combination with the detected object that is being tracked.

25. A computer system that facilitates interaction with a user, comprising a processor executing the following components stored in computer memory:

an object processing system for processing object information of one or more objects in a scene, the object processing system including;

a tracking component that at least tracks the one or more objects selected by running a set of multiple hypotheses on a plurality of moving objects detected within an engagement volume, where multiple trackers comprising at least a first and second tracker are utilized to simultaneously track at least one object from the one or more tracked objects such that the second tracker continues to track the at least one object upon failure of the first tracker to track the at least one object, wherein the set of multiple hypotheses are supported by frame-to-frame tracking through time in at least one view and stereo matching across two views, and wherein when a distance between the at least an object and a second moving object falls below a threshold distance the second object is removed from consideration on an assumption that trackers tracking the at least an object and the second moving object are tracking a single object;

a seeding component that iteratively seeds the tracking component with object hypotheses based upon the objects that are tracked;

a filtering component that selectively removes one or more of the tracked objects and corresponding object hypothesis from the set of object hypotheses in accordance with predetermined criteria; and

an interpreting component that interprets object characteristics of the at least an tracked object and causes a command to be executed in response thereto;

an input system for receiving user input separately or in combination with the object processing system; and

a presentation system for presenting information to the user in response to at least one of the command being executed and receiving user input via the input system.

26. The computer system of claim 25, the object processing system defining an engagement volume within the scene such that the engagement volume is automatically positioned proximate to the user.

27. The computer system of claim 25, the engagement volume automatically positioned proximate to the user as the user roams.

28. The computer system of claim 25, the object processing system defining the engagement volume within the scene such that the engagement volume automatically positioned according to predefined user preferences.

29. A network of computer systems according to claim 25.

30. The computer system of claim 25, the object processing system including a plurality of cameras for obtaining images of the one or more objects, the object hypotheses supported by frame-to-frame tracking through time and stereo matching of the images of the plurality of cameras.

31. A method of facilitating a human-computer interface, the method comprising:

employing a processor to execute computer executable instructions stored in memory to perform the following acts;

acquiring gesture characteristics of at least one of a plurality of gestures within a volume of space with an acquisition component, the gesture characteristics acquired based at least in part upon image comparison of a plurality of images relative to a course mapping of the images;

selecting the at least one gesture by employing a set of multiple object hypotheses on the plurality of gestures within the volume of space, where multiple trackers are utilized to track each of one or more objects performing the gestures such that upon failure of a tracker to track at least one of the one or more objects, an alternate tracker continues to track the at least one object, wherein the set of multiple object hypotheses are supported by frame-to-frame tracking through time in at least one view and stereo matching across two views, and wherein when a distance between a tracked object and a second moving object falls below a threshold distance the second object is removed from consideration;

iteratively seeding the acquisition component with at least one gesture hypothesis based upon the presence of the gesture characteristics in the volume of space and the image comparison; and

programmatically controlling a graphical representation of a graphical interface in response to acquiring the at least one gesture.

32. The method of claim 31, further comprising processing device input signals in cooperation with the acquiring the gesture, which device input includes at least one of keyboard input, mouse input, and audio input.

33. The method of claim 31, the graphical representation is at least an icon and text.