Gesture based interface system and method

• US 9,377,867 B2
• Filed: 08/08/2012
• Issued: 06/28/2016
• Est. Priority Date: 08/11/2011
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay, Israel, Nadav, Anavi, Tamir, Grofit, Shahaf, Bar-Yosef, Itay
• Primary Examiner(s): Haley, Joseph
• Assistant Examiner(s): Frank, Emily
• Application Number:
  US14/237,834
• Publication Number:
  US 20140306877A1
• Time in Patent Office: 1,420 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

A user interface apparatus for controlling any kind of a device. Images obtained by an image sensor in a region adjacent to the device are input to a gesture recognition system which analyzes images obtained by the image sensor to identify one or more gestures. A message decision maker generates a message based upon an identified gesture and a recognition mode of the gesture recognition system. The recognition mode is changed under one or more various conditions.

Claims

1. A device comprising:

at least one processor configured to;

receive input from at least one image sensor;

analyze images obtained by the at least one image sensor to detect at least one predefined gesture associated with at least a portion of a hand;

change a recognition mode of the device from a first recognition mode to a second recognition mode if a predefined condition is met, wherein the second recognition mode is associated with the first recognition mode, a detected gesture, and situational awareness information of the device, wherein the situational awareness information includes at least one of information related to a state of the device, information received by a sensor associated with the device, information related to one or more processes running on the device, information related to applications running on the device, information related to a power condition of the device, information related to a notification of the device, information related to movement of the device, information related to a spatial orientation of the device, information relating to an interaction with one or more users, information relating to user behavior, and information relating to one or more triggers; and

generate a message based upon the first recognition mode and a detected gesture,

wherein the first recognition mode differs from the second recognition mode by one or more of a capture rate of images obtained by the at least one image sensor or a resolution of images obtained by the at least one image sensor.

2. The device of claim 1, wherein the at least one processor is further configured to generate the message based, at least in part, on information from at least one system sensor.

3. The device of claim 2, wherein the at least one processor is further configured to detect the gesture based, at least in part, on analysis of system sensor signals.

4. The device of claim 3, wherein the at least one processor is further configured to define at least one of the first and second recognition modes based, at least in part, on predefined gestures to be detected, and one of the resolution of images obtained by the at least one image sensor, or the capture rate of images obtained by the at least one image sensor.

5. The device of claim 1, wherein the at least one processor is further configured to provide an indication that a gesture was recognized, the indication including at least one of a visual indication, an audio indication, a tactile indication, an ultrasonic indication, and a haptic indication.

6. The device of claim 5, wherein the at least one processor is further configured to provide the indication based, at least in part, on the situational awareness information.

7. The device of claim 6, wherein the situational awareness information is information related to at least one trigger, wherein the trigger includes at least one of a change in user interface of an application, a change in a visual appearance of an application, a change in mode of an application, a change in state of an application, an event occurring in software running on the device, a change in behavior of an application, a notification received via a network, an online service notification, a notification generated by the device or an application or by a service, from a touch on a touch screen, a pressing of a virtual or real button, a sound received by a microphone connected to the device, detection of a user holding the device, a signal from a proximity sensor, an incoming voice or video call via a cellular network, a wireless network, TCPIP, or a wired network, an incoming 3D video call, a text message notification, a notification of a meeting, a community network based communication, a Skype notification, a facebook notification, a twitter notification, an on-line service notification, a missed call notification, an email notification, a voice mail notification, a device notification, a beginning or an end of a song on a player, and a beginning or an end of a video.

8. The device of claim 1, wherein the situation awareness information relates to at least one trigger selected from a gesture, passing an object over the device at a distance less than a predefined distance, detection of a user looking at the device, detection of a face in an image obtained by the at least one image sensor, detection of one or more eyes in an image obtained by the at least one image sensor, detection of one or more hands or portions of a hand in an image obtained by the at least one image sensor, and detection of a predefined individual looking at the device.

9. The device of claim 1, wherein the situational awareness information is related to a state of the device, further wherein the message is addressed to the device, and further wherein the at least one processor is further configured to:

change a state of the device based, at least in part, on receipt of the message.

10. The device of claim 1, wherein the message is a command.

11. The device of claim 10, wherein the command includes at least one of a command to run an application on the device, a command to stop an application running on the device, a command to activate a service running on the device, and a command to stop a service running on the first device.

12. The device of claim 10, wherein the command provides an indication of at least one of an interface system that is ready to recognize gestures, an interface system that is ready to recognize one or more predefined gestures, and an interface system that is not ready to recognize gestures.

13. The device of claim 1, wherein the at least one processor is further configured to:

change the recognition mode of the device from the second recognition mode back to the first recognition mode after a predefined amount of time has elapsed.

14. The device of claim 1, further including at least one of an ultrasonic sensor, a proximity sensor, and an accelerometer.

15. The device of claim 1, wherein the message is a first message adapted for interpretation by a first running application on the device in a first manner, and wherein a second message is adapted for interpretation by a second running application on the device in a second manner.

16. A non-transitory computer-readable medium including instructions that, when executed by at least one processor, cause the processor to perform operations comprising:

receiving input from at least one image sensor;

analyzing images obtained by the at least one image sensor to detect at least one predefined gesture associated with at least a portion of a hand;

changing a recognition mode of the device from a first recognition mode to a second recognition mode if a predefined condition is met, wherein the second recognition mode is associated with the first recognition mode, a detected gesture, and situational awareness information of the device, wherein the situational awareness information includes at least one of information related to a state of the device, information received by a sensor associated with the device, information related to one or more processes running on the device, information related to applications running on the device, information related to a power condition of the device, information related to a notification of the device, information related to movement of the device, information related to a spatial orientation of the device, information relating to an interaction with one or more users, information relating to user behavior, and information relating to one or more triggers; and

generating a message based upon the first recognition mode and a detected gesture,

wherein the first recognition mode differs from the second recognition mode by one or more of a capture rate of images obtained by the at least one image sensor or a resolution of images obtained by the at least one image sensor.

17. The non-transitory computer-readable medium of claim 16, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

generating the message based, at least in part, on information from at least one system sensor.

18. The non-transitory computer-readable medium of claim 17, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

detecting the gesture based, at least in part, on analysis of system sensor signals.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

defining at least one of the first and second recognition modes based, at least in part, on predefined gestures to be detected, and one of the resolution of images obtained by the at least one image sensor, or the capture rate of images obtained by the at least one image sensor.

20. The non-transitory computer-readable medium of claim 16, wherein the message is a command.

21. The non-transitory computer-readable medium of claim 20, wherein the command includes at least one of a command to run an application on the device, a command to stop an application running on the device, a command to activate a service running on the device, and a command to stop a service running on the first device.

22. The non-transitory computer-readable medium of claim 20, wherein the command provides an indication of at least one of an interface system that is ready to recognize gestures, an interface system that is ready to recognize one or more predefined gestures, and an interface system that is not ready to recognize gestures.

23. The non-transitory computer-readable medium of claim 16, wherein the instructions, when executed by at least one processor, further cause the processor to perform at least one operation comprising:

changing the recognition mode of the device from the second recognition mode back to the first recognition mode after a predefined amount of time has elapsed.

24. The non-transitory computer-readable medium of claim 16, wherein the message is a first message adapted for interpretation by a first running application on the device in a first manner, and wherein a second message is adapted for interpretation by a second running application on the device in a second manner.

25. A gesture recognition method, comprising:

receiving input from at least one image sensor;

analyzing images obtained by the at least one image sensor to detect at least one predefined gesture associated with at least a portion of a hand;

changing a recognition mode of the device from a first recognition mode to a second recognition mode if a predefined condition is met, wherein the second recognition mode is associated with the first recognition mode, a detected gesture, and situational awareness information of the device, wherein the situational awareness information includes at least one of information related to a state of the device, information received by a sensor associated with the device, information related to one or more processes running on the device, information related to applications running on the device, information related to a power condition of the device, information related to a notification of the device, information related to movement of the device, information related to a spatial orientation of the device, information relating to an interaction with one or more users, information relating to user behavior, and information relating to one or more triggers; and

generating a message based upon the first recognition mode and a detected gesture,

wherein the first recognition mode differs from the second recognition mode by one or more of a capture rate of images obtained by the at least one image sensor or a resolution of images obtained by the at least one image sensor.