Gesture-based computer interface

• US 6,222,465 B1
• Filed: 12/09/1998
• Issued: 04/24/2001
• Est. Priority Date: 12/09/1998
• Status: Expired due to Fees

• Current Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Kumar, Senthil, Segen, Jakub
• Primary Examiner(s): Horabik, Michael
• Assistant Examiner(s): WONG, ALBERT KANG
• Application Number:
  US09/208,084
• Time in Patent Office: 867 Days
• Field of Search: 341/20, 345/156, 345/168, 345/173, 345/420, 345/358, 382/100, 382/107, 738/654
• US Class Current: 341/20
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A system and method for manipulating virtual objects in a virtual environment, for drawing curves and ribbons in the virtual environment, and for selecting and executing commands for creating, deleting, moving, changing, and resizing virtual objects in the virtual environment using intuitive hand gestures and motions. The system is provided with a display for displaying the virtual environment and with a video gesture recognition subsystem for identifying motions and gestures of a user'"'"'s hand. The system enables the user to manipulate virtual objects, to draw free-form curves and ribbons and to invoke various command sets and commands in the virtual environment by presenting particular predefined hand gestures and/or hand movements to the video gesture recognition subsystem.

Claims

1. A method of manipulating a virtual object in a three dimensional virtual environment implemented in a computer system having a display unit for displaying the virtual environment and a video gesture recognition subsystem that monitors a predefined identification zone, said method manipulating the virtual object in response to predetermined hand gestures and movements made by a user'"'"'s hand within the identification zone and identified by the video gesture recognition subsystem, comprising the steps of:

(a) generating, by the computer system, a first gripper element having a first end portion and a second gripper element having a second end portion for display on the display unit in the virtual environment, wherein said first and second gripper elements are positioned in substantially parallel relation to align said first and said second end portions with one another, wherein said first gripper element corresponds to and is operatively moved and oriented in accordance with a first finger of the user'"'"'s hand, wherein said second gripper element corresponds to and is operatively moved and oriented in accordance with a second finger of the user'"'"'s hand, wherein said first and second grippers are moved in accordance with movement of the user'"'"'s hand within the identification zone, and wherein said first gripper element is moved with respect to said second gripper element corresponding to movement of the user'"'"'s first finger with respect to the user'"'"'s second finger;

(b) grasping, by user movement of the user'"'"'s first and second fingers, the virtual object by aligning said first and second gripper elements with the virtual object and moving said first and said second gripper elements into proximity with one another until said first and said second end portions intersect with the virtual object; and

(c) manipulating, by user movement of the user'"'"'s hand and the user'"'"'s first and second fingers, the grasped virtual object.

2. The method of claim 1, wherein said step (c) comprises the step of moving by the user, the grasped virtual object within the virtual environment by moving said first and said second gripper elements while maintaining the intersection of said first and said second end portions with the virtual object.

3. The method of claim 1, further comprising the step of:

(d) releasing, by user movement of the user'"'"'s first and second fingers, the grasped virtual object by moving said first and said second gripper elements apart from one another until said first and said second end portions no longer intersect the virtual object.

4. The method of claim 1, wherein said virtual object is of a predetermined first color, further comprising the steps of,

(e) after said step (b), changing, by the computer system, the color of the grasped virtual object to a predetermined second color to indicate to the user that the virtual object has been successfully grasped;

and

(f) after said step (d), changing, by the computer system, the color of the grasped virtual object back to said first color to indicate to the user that the virtual object has been successfully released.

5. The method of claim 1, further comprising the steps of:

(g) displaying, by the computer system on the display unit when the video gesture recognition subsystem identifies a predetermined first hand gesture made by the user, a first menu having a first plurality of menu commands, each of said first plural menu commands defining a unique visual command area;

(h) selecting, by the user, a particular menu command of said displayed first plural menu commands by generating, by the computer system for display on the display unit in the virtual environment when the video gesture recognition subsystem identifies a second hand gesture made by the user, a visual selection identifier responsive to movement of a predefined finger of the user'"'"'s hand so that when the user points to said particular menu command of said first plurality of menu commands, said visual selection identifier is positioned within said visual command area of said particular menu command to thereby select said particular menu command; and

(h) determining, by the control system, the filter matrix for adjusting the frequency of the filter with respect to the input signal;

(i) executing by the computer system, when the video gesture recognition subsystem identifies said first hand gesture made by the user, said particular selected menu command.

6. The method of claim 5, further comprising the step of:

(j) generating, by the control system under direction of the user, one of a curve and a ribbon in the virtual environment.

7. The method of claim 6, wherein said particular menu command executed at said step (i) comprises a drawing command, and wherein said step (j) further comprises the step of:

(k) replacing, by the computer system, said first menu displayed on the display unit in the virtual environment with a drawing menu having a plurality of drawing menu commands.

11. The method of claim 5, further comprising the step of:

(y) creating by the computer system under direction of the user, a new virtual object for display by the display unit.

12. The method of claim 11, wherein said particular menu command executed at said step (i) comprises an object command, wherein said step (y) further comprises the step of:

(z) replacing, by the computer system, said first menu displayed on the display unit in the virtual environment with an object menu having a plurality of object menu commands.

14. The method of claim 5, further comprising the step of:

(gg) eliminating by the control system under direction of the user, the virtual object from the virtual environment.

15. The method of claim 14, wherein said particular menu command executed at said step (i) comprises an object command, wherein said step (gg) further comprises the step of:

(hh) replacing said first menu displayed on the display unit in the virtual environment with an object menu having a plurality of object menu commands.

17. The method of claim 5, wherein said step (c) comprises the step of:

(mm) changing by the control system under direction of the user, a particular predefined characteristic of the virtual object to a particular alternate characteristic.

18. The method of claim 17, wherein said particular menu command executed at said step (i) comprises an object command, wherein said step (mm) further comprises the step of:

(nn) replacing, by the computer system, said first menu displayed on the display unit in the virtual environment with an object menu having a plurality of object menu commands.

24. The method of claim 5, further comprising the step of:

(ww) changing, by the control system under direction of the user, a view of the virtual environment.

25. The method of claim 24, wherein said particular menu command executed at said step (i) comprises a view command.

27. The method of claim 1, wherein the virtual environment comprises a plurality of peripheral boundary surfaces enclosing at least a portion of the virtual environment, displayed by the display unit, further comprising the step of:

(aaa) displaying on at least a portion of said plural boundary surfaces, by the display unit, a shadow generated by the computer system for each of said first and said second gripping elements, so as to enable the user to visually assess a position of said first and said second gripping elements with respect to said plural boundary surfaces.

28. The method of claim 1, wherein said first finger of the user'"'"'s hand is an index finger and wherein said second finger of the user'"'"'s hand is a thumb.

29. The method of claim 5, wherein said first hand gesture comprises an open-handed reaching gesture.

30. The method of claim 5, wherein said second hand gesture comprises a pointing gesture, and wherein said predefined finger is an index finger.

8. The method of step 7, wherein said plurality of drawing commands comprises a curve draw command, and wherein said step (j) further comprises the steps of:

(l) selecting, by the user, the curve draw command from said drawing menu by moving said visual selection indicator to an area of the drawing menu corresponding to the curve draw command;

(m) executing, by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the curve draw command;

(n) terminating, by the computer system, display of said drawing menu;

(o) generating, by the computer system when the video gesture recognition subsystem identifies said second hand gesture made by the user, a curve in the virtual environment, wherein appearance and position of said curve are responsive to movement of said predefined finger of the user'"'"'s hand; and

(p) terminating generation of said curve and re-displaying said first menu on the display unit, by the computer system, when the video gesture recognition subsystem identifies said first hand gesture made by the user.

9. The method of step 7, wherein said plurality of drawing commands comprises a ribbon draw command, and wherein said step (j) further comprises the steps of:

(q) selecting, by the user, the ribbon draw command from said drawing menu by moving said visual selection indicator to an area of the drawing menu corresponding to the ribbon draw command;

(r) executing, by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the ribbon draw command;

(s) terminating, by the computer system, display of said drawing menu;

(t) generating, by the computer system when the video gesture recognition subsystem identifies said second hand gesture made by the user, a planar ribbon of a predetermined width in the virtual environment, wherein orientation, appearance and position of said ribbon are responsive to movement and orientation of said predefined finger of the user'"'"'s hand; and

(u) terminating generation of said ribbon and redisplaying said first menu on the display unit by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user.

10. The method of step 7, wherein said plurality of drawing commands comprises a return command, further comprising the steps of:

(v) selecting, by the user, the return command from said drawing menu by moving said visual selection indicator to an area of the drawing menu corresponding to the return command;

(w) executing by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the return draw command; and

(x) replacing, by the computer system, said drawing menu displayed on the display unit in the virtual environment with said first menu.

13. The method of step 11, wherein said plurality of object commands comprises a create object command, and wherein said step (y) further comprises the steps of:

(aa) selecting, by the user, the create object command from said object menu by moving said visual selection indicator to an area of the object menu corresponding to the create object command;

(bb) executing by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the create object command;

(cc) displaying by the computer system in the virtual environment on the display unit, an object palette menu having a plurality of displayed selectable object types;

(dd) selecting, by the user, a particular object type from said object palette menu;

(ee) terminating, by the computer system, display of said object menu and said object palette menu; and

(ff) generating by the computer system for display by the display unit when the video gesture recognition subsystem identifies said first hand gesture made by the user, a new virtual object corresponding to said particular object type and thereafter repeating said step (a).

16. The method of step 15, wherein said plurality of object commands comprises a delete object command, and wherein said step (gg) further comprises the steps of:

(ii) selecting, by the user, the delete object command from said object menu by moving said visual selection indicator to an area of the object menu corresponding to the delete object command;

(jj) executing by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the delete object command;

(kk) terminating, by the computer system, display of said object menu and thereafter repeating said step (a); and

(ll) eliminating, by the computer system, the virtual object from the virtual environment when the virtual object is grasped by said first and said second gripper elements.

19. The method of step 18, wherein said plurality of object commands comprises a change characteristic command, and wherein said step (mm) further comprises the steps of:

(oo) selecting, by the user, the change characteristic command from said object menu by moving said visual selection indicator to an area of the object menu corresponding to the change characteristic command;

(pp) executing by the computer system when the video gesture recognition subsystem identifies said first hand gesture made by the user, the change characteristic command;

(qq) displaying by the computer system in the virtual environment on the display unit, a change characteristic palette menu having a plurality of selectable alternate object characteristics;

(rr) selecting, by the user, said particular alternate characteristic from said change characteristic palette menu for application to the virtual object;

(ss) terminating, by the computer system, display of said object menu and said change characteristic palette menu, and repeating said step (a); and

(tt) changing, by the computer system, said predefined particular characteristic of the virtual object to said selected particular alternate characteristic, when the virtual object is grasped by said first and said second gripper elements.

23. The method of claim 19, wherein said particular characteristic is a size of the virtual object and wherein said plurality of alternate object characteristics comprises a plurality of scaling parameters, each plural scaling parameter corresponding to a different coordinate axis having a positive and a negative direction and being defined in the virtual environment, further comprising the steps of:

(uu) repeating, by the computer system, said step (a) and said step (b) and thereafter moving, by the user, said first and said second gripper elements along one of said coordinate axes to change the size of the grasped virtual object along said one axis, wherein when said first and said second gripper elements are moved in the positive direction of said one axis the computer system increases the size of the virtual object along said one axis, and when said first and said second gripper elements are moved in the negative direction of said one axis the computer system decreases the size of the virtual object along said one axis; and

(vv) replacing by the computer system when said virtual object is released from said first and said second gripping elements, said first and said second gripper elements with said change characteristic palette.

20. The method of step 19, wherein said predefined particular characteristic is a color of the virtual object and wherein said plurality of alternate object characteristics comprises a plurality of different colors.

21. The method of step 19, wherein said predefined particular characteristic is a shininess of the virtual object and wherein said plurality of alternate object characteristics comprises a plurality of different shininess types.

22. The method of step 19, wherein said predefined particular characteristic is a texture of the virtual object and wherein said plurality of alternate object characteristics comprises a plurality of different texture types.

26. The method of step 25, wherein said step (ww) further comprises the steps of:

(xx) terminating, by the computer system, display of said first menu;

(yy) changing by the computer system in response to movement of the user'"'"'s hand and fingers thereof, at least one of a viewing vantage point and orientation from which the virtual environment is displayed to the user on said display unit; and

(zz) terminating said changing of the virtual environment view and re-displaying said first menu on the display unit by the computer system, when the video gesture recognition subsystem identifies said first hand gesture made by the user.

31. A system for manipulating a virtual object in a three dimensional virtual environment, comprising:

display means for displaying the virtual environment to a user;

video gesture recognition means for identifying predetermined hand gestures and movements made by the user'"'"'s hand within a predefined identification zone; and

control means, connected to said display means and said video gesture recognition means, for;

(1) generating, for display by said display means in the virtual environment, a first gripper element having a first end portion and a second gripper element having a second end portion for display on the display unit in the virtual environment, wherein said first and second gripper elements are positioned in substantially parallel relation to align said first and said second end portions with one another, wherein said first gripper element corresponds to and is operatively moved and oriented in accordance with a first finger of the user'"'"'s hand, wherein said second gripper element corresponds to and is operatively moved and oriented in accordance with a second finger of the user'"'"'s hand, wherein said first and second grippers are moved in accordance with movement of the user'"'"'s hand within the identification zone, and wherein said first gripper element is moved with respect to said second gripper element corresponding to movement of the user'"'"'s first finger with respect to the user'"'"'s second finger;

(2) enabling the user to grasp, by user movement of the user'"'"'s first and second fingers, the virtual object by aligning said first and second gripper elements with the virtual object and moving said first and second gripper elements into proximity with one another until said first and said second end portions intersect with the virtual object; and

(3) enabling the user to manipulate, by user movement of the user'"'"'s hand and the user'"'"'s first and second fingers, the grasped virtual object by moving the user'"'"'s hand and said first and second fingers within said predefined identification zone while maintaining contact between said first and said second ends and the virtual object.