Apparatus, methods and computer program products providing finger-based and hand-based gesture commands for portable electronic device applications

• US 8,086,971 B2
• Filed: 06/28/2006
• Issued: 12/27/2011
• Est. Priority Date: 06/28/2006
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Corporation
• Inventors: Radivojevic, Zoran, Zou, Yanming, Wang, Kong Qiao, Takala, Roope Tapio, Lantz, Vuokko Tuulikki, Lehtiniemi, Reijo, Rantala, Jukka Ilmari, Vatanparast, Ramin
• Primary Examiner(s): Lo, Weilun
• Assistant Examiner(s): BECKER, SHASHI KAMALA
• Application Number:
  US11/477,508
• Publication Number:
  US 20080005703A1
• Time in Patent Office: 2,008 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A method includes executing a gesture with a user-manipulated physical object in the vicinity of a device; generating data that is descriptive of the presence of the user-manipulated object when executing the gesture; and interpreting the data as pertaining to at least one object, such as an object displayed by the device.

Claims

1. A method, comprising:

detecting an execution of a gesture by a user'"'"'s hand by tracking movements of said hand in the vicinity of a device within a limited sensing space defined by a plurality of ultrasonic transducers;

identifying at least one object displayed by the device from the tracked movements;

generating data that is descriptive of at least of a shape of the user'"'"'s hand in three dimensional space when executing the gesture;

capturing an image of fingertips of the user'"'"'s hand;

determining a number of fingertips from the captured image, a spacing between each fingertip and an angular orientation of each fingertip relative to one another;

determining whether to begin a gesture recognition process, wherein determining whether to begin the recognition process comprises detecting a predetermined pattern of fingertip movement, a velocity of the fingertip movement or a size of the fingertip, the number, spacing and angular orientation of the fingertips corresponding to the predetermined fingertip pattern, and if the recognition process is begun;

comparing a current image of the fingertips with at least one prior image of the fingertips to determine a fingertip trace corresponding to the executed gesture;

displaying a pointer in at least one position corresponding to the executed gesture; and

determining at least one command for said at least one object displayed by the device based at least in part on the data and the executed gesture.

2. The method of claim 1, where generating data generates data that is further descriptive of motion made by the user'"'"'s hand over a period of time.

3. The method of claim 1, where generating data comprises sequentially creating individual ones of a plurality of records, where individual ones of the plurality of records comprise data descriptive of a location of the user'"'"'s hand at a corresponding point in time while the gesture is executed.

4. The method of claim 1, where generating data uses a plurality of imaging devices.

5. The method of claim 4, where the plurality of imaging devices comprise a part of a display screen.

6. The method of claim 1, where the gesture comprises at least one of a substantially circular motion, a substantially linear motion, at least one substantially circular motion in combination with at least one substantially linear motion, a substantially circular motion and a substantially linear motion, in combination with a period of substantially no motion, a substantially curved motion, and a tapping motion.

7. The method of claim 1, where the gesture executed with the user'"'"'s hand is comprised of the use of at least two fingers, and where a motion made when executing the gesture comprises movement of one finger relative to at least one other finger.

8. The method of claim 1, where the data is further descriptive of at least one of a velocity and an acceleration of the user'"'"'s hand.

9. The method of claim 1, where the data is further descriptive of at least one of a spatial orientation of the user'"'"'s hand in two or three dimensional space, and the repose of the user'"'"'s hand in two or three dimensional space.

29. The method of claim 1, further comprising projecting a view of an image to be displayed.

32. The method of claim 1, wherein the pattern of the executed gesture is determined from one or more of the number of fingertips, a spacing between each of the fingertips or an angular orientation of each of the fingertips.

10. An apparatus, comprising:

a display configured to display information;

a sensor arrangement that is responsive to the user executing a gesture with a user'"'"'s hand in the vicinity of a surface of the apparatus and configured to track movements of said hand within a limited sensing space defined by a plurality of ultrasonic transducers, the sensor arrangement configured to provide data descriptive of at least the shape of the user'"'"'s hand in three dimensional space when executing the gesture, capture an image of fingertips of the user'"'"'s hand, and determine a number of fingertips from the captured image, a spacing between each fingertip and angular orientation of each fingertip relative to one another;

a processor coupled to the sensor arrangement and configured to determine whether to begin a gesture recognition process, wherein determining whether to begin the recognition process comprises detecting a predetermined pattern of fingertip movement, a velocity of the fingertip movement or a size of the fingertip, the number, the spacing and angular orientation of the fingertips corresponding to the predetermined fingertip pattern, and if the recognition process is begun, compare a current image of the fingertips with at least one prior image of the fingertips to determine a fingertip trace corresponding to the executed gesture; and

the processor further being configured to interpret the data to identify at least one object from the tracked movements, to identify the executed gesture, and to determine at least one command for said at least one object based at least in part on the identified gesture and the executed gesture.

11. The apparatus of claim 10, where said data generating means generates data that is further descriptive of motion being made by the hand or portion of the hand in two or three dimensional space, and where data generating means is comprised of at least one of a plurality of acoustic sensors and a plurality of light sensors.

12. The apparatus of claim 10, embodied in a device that comprises means for conducting wireless communications.

13. The apparatus of claim 10, where the data is further descriptive of at least one of a spatial orientation of at least a part of the user'"'"'s hand in two or three dimensional space, and the repose of at least a part of the user'"'"'s hand in two or three dimensional space.

33. The apparatus of claim 10, wherein the pattern of the executed gesture is determined from one or more of the number of fingertips, a spacing between each of the fingertips or an angular orientation of each of the fingertips.

14. A non-transitory computer readable medium tangibly embodying a computer program, execution of the computer program by at least one data processor resulting in operations that comprise:

detecting an execution of a gesture by a user'"'"'s hand by tracking movements of said hand in the vicinity of a device within a limited sensing space defined by a plurality of ultrasonic transducers;

in response to detecting a user executing a gesture with a user'"'"'s hand in the vicinity of a device, identifying at least one object from the tracked movements and generating data that is descriptive of at least the shape of the user'"'"'s hand in three dimensional space when executing the gesture, capturing an image of fingertips of the user'"'"'s hand, and determining a number of fingertips from the captured image, a spacing between each fingertip and angular orientation of each fingertip relative to one another;

determining whether to begin a gesture recognition process, wherein determining whether to begin the recognition process comprises detecting a predetermined pattern of fingertip movement, a velocity of the fingertip movement or a size of the fingertip, the number, spacing and angular orientation of the fingertips corresponding to the predetermined fingertip pattern, and if the recognition process is begun;

comparing a current image of the fingertips with at least one prior image of the fingertips to determine a fingertip trace corresponding to the executed gesture;

displaying a pointer in at least one position corresponding to the executed gesture; and

determining at least one command for said at least one object to the user based at least in part on the data and the executed gesture.

15. The computer readable medium of claim 14, where generating data generates data that is further descriptive of a motion made by the user'"'"'s hand over a period of time.

16. The computer readable medium of claim 14, where generating data comprises sequentially creating individual ones of a plurality of records, where individual ones of the plurality of records comprise data descriptive of a location of the user'"'"'s hand at a corresponding point in time while the gesture is executed.

17. The computer readable medium of claim 14, where generating data uses a plurality of imaging devices.

18. The computer readable medium of claim 17, where the plurality of imaging devices comprise a part of a display screen.

19. The computer readable medium of claim 14, where generating data generates data that is further descriptive of a motion made by the user'"'"'s hand in at least one of two dimensional space and three dimensional space, where the motion comprises at least one of:

a substantially circular motion;

a substantially linear motion;

at least one substantially circular motion in combination with at least one substantially linear motion;

a substantially circular motion and a substantially linear motion, in combination with a period of substantially no motion;

a substantially curved motion; and

a tapping motion.

20. The computer readable medium of claim 19, where the user'"'"'s hand is comprised of at least two fingers of a user, and where the motion comprises movement of one finger relative to at least one other finger.

21. The computer readable medium of claim 14, where the data is further descriptive of at least one of a velocity and an acceleration of the user'"'"'s hand.

22. The computer readable medium of claim 14, where the data is further descriptive of at least one of a spatial orientation of at least a part of the user'"'"'s hand in two or three dimensional space, and the repose of at least a part of the user'"'"'s hand in two or three dimensional space.

30. The computer readable medium of claim 14, where the operations further comprise projecting a view of an image to be displayed.

23. A device, comprising:

a unit configured to display information;

an imaging system configured to generate data that is descriptive of at least the shape of a user'"'"'s hand in three dimensional space when executing a gesture, where the imaging system comprises a plurality of ultrasonic transducers arranged so as to transmit acoustic energy into a volume of space substantially adjacent to a surface of the device;

a data processor; and

at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the device to perform at least the following;

detecting an execution of the gesture by said hand by tracking movements of said hand in the vicinity of a device within a limited sensing space, capturing an image of fingertips of the hand, and determining a number of fingertips from the captured image, including a spacing between each fingertip and angular orientation of each fingertip relative to one another;

determining whether to begin a gesture recognition process, wherein determining whether to begin the recognition process comprises detecting a predetermined pattern of fingertip movement, a velocity of the fingertip movement or a size of the fingertip, the number, spacing and angular orientation of the fingertips corresponding to the predetermined fingerprint;

identifying at least one object from the tracked movements;

displaying a pointer in at least one position corresponding to the tracked movements; and

determining at least one command for said at least one object based at least in part on the data and the executed gesture.

24. The device of claim 23, where said imaging system generates data by sequentially creating individual ones of a plurality of records, where individual ones of the plurality of records comprise data descriptive of a location of the user'"'"'s hand at a corresponding point in time while the gesture is executed.

25. The device of claim 23, where said imaging system employs at least one of acoustic energy and optical energy.

26. The device of claim 23, said imaging system being responsive to a motion made by the user'"'"'s hand in three dimensional space, where the motion comprises at least one of:

a substantially circular motion;

a substantially linear motion;

at least one substantially circular motion in combination with at least one substantially linear motion;

a substantially circular motion and a substantially linear motion, in combination with a period of substantially no motion;

a substantially curved motion;

a tapping motion; and

movement of one finger relative to at least one other finger.

27. The device of claim 23, where the data is further descriptive of at least one of a velocity and an acceleration of the user'"'"'s hand.

28. The device of claim 23, where the data is further descriptive of at least one of a spatial orientation of at least a part of the user'"'"'s hand in two or three dimensional space, and the repose of at least a part of the user'"'"'s hand in two or three dimensional space.

31. The device of claim 23, further comprise a projector configured to project a view of an image to be displayed.