Systems and methods for proximity sensor and image sensor based gesture detection

• US 9,977,507 B2
• Filed: 03/13/2014
• Issued: 05/22/2018
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: eyeSight Mobile Technologies Ltd.
• Original Assignee: eyeSight Mobile Technologies Ltd.
• Inventors: Katz, Itay
• Primary Examiner(s): Siddiqui, Md Saiful A
• Application Number:
  US14/775,657
• Publication Number:
  US 20160026255A1
• Time in Patent Office: 1,531 Days
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 23/64 Computer-aided capture of images, e.g. transfer from script file into camera, check of taken image quality, advice or proposal for image composition or decision on when to take image
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Systems, methods, and non-transitory computer-readable media are disclosed. For example, a dual sensor control device is disclosed that includes at least one processor for receiving information from a proximity sensor and an image sensor. The processor may be configured to receive first data from the proximity sensor while the image sensor is in a first state, determine, using the first data, a presence of an object in proximity to the proximity sensor. The processor may also be configured to output, based on the determined presence of the object in proximity to the proximity sensor, a signal to the image sensor to cause the image sensor to enter a second state, different from the first state. The processor may also be configured to receive second data from the image sensor in the second state, and output at least one of a message and a command associated with the second data.

Claims

1. A dual sensor control device, comprising:

at least one processor for receiving information from a proximity sensor and an image sensor, the at least one processor being configured to;

receive first data from the proximity sensor while the image sensor is in a first state;

detect, using the received first data, a presence of an object in proximity to the proximity sensor;

output, based on the detected presence of the object, a signal to cause the image sensor to enter a second state different from the first state before the object reaches a field of view of the image sensor;

receive, from the image sensor in the second state, second data reflective of images of the object;

determine, based on a combination of the first data and the second data, a gesture performed by the object; and

output at least one of a message and a command associated with the determined gesture.

2. The device of claim 1, wherein the object is at least one of a hand, a finger, and a handheld object used for pointing.

3. The device of claim 2, wherein the at least one processor is further configured to distinguish between a plurality of predefined gestures.

4. The device of claim 2, wherein the first data is reflective of at least a one-dimensional position of the object relative to the proximity sensor, and wherein the at least one of a message and a command is further associated with the one-dimensional position.

5. The device of claim 2, wherein the second data is reflective of at least a two-dimensional position of the object relative to the image sensor, and wherein the at least one of a message and a command is further associated with the two-dimensional position.

6. The device of claim 2, wherein the first data is reflective of at least a one-dimensional position of the object relative to the proximity sensor, wherein the second data is reflective of at least a two-dimensional position of the object relative to the image sensor, and wherein the at least one of a message and a command is further associated with the one-dimensional position and the two-dimensional position.

7. The device of claim 1, wherein the at least one of a message and a command is further associated with the first data obtained before the object reaches the field of view of the image sensor and the second data obtained after the object reaches the field of view of the image sensor.

8. The device of claim 1, wherein the at least one processor is further configured to enable gesture control based on the determined presence of the object in proximity to the proximity sensor.

9. The device of claim 1, wherein the at least one processor is further configured to turn on a display based on the determined presence of the object in proximity to the proximity sensor.

10. The device of claim 1, wherein the first state is at least one of an off state and a first power state, and wherein the second state is at least one of an on state and a second power state higher than the first power state.

11. The device of claim 1, wherein the first state is at least one of a first sampling rate, a first resolution, and a first exposure length, and the second state is at least one of a second sampling rate different than the first sampling rate, a second resolution different than the first resolution, and a second exposure length different than the first exposure length.

12. The device of claim 1, wherein the first data is associated with a first resolution and the second data is associated with a second resolution greater than the first resolution.

13. The device of claim 1, wherein the at least one processor is further configured to determine whether the proximity sensor and the image sensor are facing a user based on information received from at least one of an accelerometer and a gyroscope.

14. The device of claim 1, wherein the at least one processor is further configured to:

determine, using data from the proximity sensor, if the object exits a field of view of the proximity sensor; and

output, based on the determined exit, a second signal to the image sensor to cause the image sensor to enter a third state.

15. The device of claim 1, wherein the at least one processor is further configured to output a second signal to the image sensor to cause the image sensor to enter a third state after a predefined condition is met.

16. The device of claim 1, wherein the at least one processor is further configured to output a second signal to the proximity sensor to cause the proximity sensor to change to a different state.

17. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations including:

receiving first data from a proximity sensor while an image sensor is in a first state;

detecting, using the received first data, a presence of an object in proximity to the proximity sensor;

outputting, based on the detected presence of the object, a signal to cause the image sensor to enter a second state different from the first state before the object reaches a field of view of the image sensor;

receiving, from the image sensor in the second state, second data reflective of images of the object;

determining, based on a combination of the first data and the second data, a gesture performed by the object; and

outputting at least one of a message and a command associated with the determined gesture.

18. A three-dimensional control device, comprising:

at least one processor for receiving information from a proximity sensor and an image sensor, the at least one processor being configured to;

receive first data, associated with a detected object, from the proximity sensor while the image sensor operates at a first level of power consumption, wherein the first data is reflective of at least a one-dimensional position of the object relative to the proximity sensor;

output, based on the detected object, a signal to cause the image sensor to enter a second level of power consumption different from the first level of power consumption before the object reaches a field of view of the image sensor;

receive second data associated with the detected object, from the image sensor while the image sensor operates at the second level of power consumption greater than the first level, wherein the second data is reflective of at least a two-dimensional position of the object relative to the image sensor;

coordinate the first data and the second data to obtain three-dimensional information associated with the detected object; and

determine, based on the obtained three-dimensional information, a gesture performed by the object.

19. The device of claim 18, wherein the at least one processor is further configured to output at least one of a message and a command associated with the three-dimensional information.

20. The device of claim 18, wherein the gesture is performed by at least one of a hand and a finger.

21. The device of claim 20, wherein the at least one processor is further configured to:

determine, using the first data, a distance of the object from the proximity sensor;

determine, using the second data, a movement of the object; and

determine, using the distance and the movement, a speed of the object.

22. The device of claim 21, wherein the at least one processor is further configured to determine at least one of a left-to-right gesture and right-to-left gesture using the three-dimensional information, wherein the object is a hand and the speed is reflective of a hand speed.

23. The device of claim 21, wherein the at least one processor is further configured to determine a circular gesture using the three-dimensional information, wherein the object is a finger and the speed is reflective of a finger speed.

24. The device of claim 18, wherein the at least one processor is further configured to output at least one of a message associated with the determined gesture and a command associated with the determined gesture to thereby enable gesture control.

25. The device of claim 18, wherein the at least one processor is further configured to distinguish between a zoom-in gesture and a zoom-out gesture using the obtained three-dimensional information.

26. The device of claim 18, wherein the at least one processor is further configured to:

determine, using the first data, whether the detected object is moving toward or away from the device;

determine, using the second data, whether one or more fingers associated with the detected object are spread or closed; and

determine at least one of a zoom-in gesture and a zoom-out gesture based on the determination of whether the detected object is moving toward or away from the device and the determination of whether the one or more fingers associated with the detected object are spread or closed.

27. The device of claim 18, wherein the at least one processor is further configured to determine, using the obtained three-dimensional information, a circular gesture.

28. The device of claim 27, wherein the at least one processor is further configured to distinguish between a clockwise circular gesture and a counter-clockwise circular gesture.

29. The device of claim 18, wherein the first data is associated with a first resolution and the second data is associated with a second resolution that is greater than the first resolution.

30. The device of claim 18, wherein the at least one processor is further configured to determine whether the proximity sensor and the image sensor are facing a user based on information received from at least one of an accelerometer and a gyroscope.

31. The device of claim 18, wherein the at least one processor is further configured to determine, using the first data, whether the object performed a gesture parallel to a display.

32. A non-transitory computer-readable medium comprising instructions that, when executed by at least one processor, cause the at least one processor to perform operations including:

receiving first data, associated with a detected object, from a proximity sensor while an image sensor operates at a first level of power consumption, wherein the first data is reflective of at least a one-dimensional position of the object relative to the proximity sensor;

outputting, based on the detected object, a signal to cause the image sensor to enter a second level of power consumption different from the first level of power consumption before the object reaches a field of view of the image sensor;

receiving second data associated with the detected object, from the image sensor while the image sensor operates at the second level of power consumption greater than the first level, wherein the second data is reflective of at least a two-dimensional position of the object relative to the image sensor; and

coordinating the first data and the second data to obtain three-dimensional information associated with the detected object; and

determining, based on the obtained three-dimensional information, a gesture performed by the object.

33. The non-transitory computer-readable medium of claim 32, wherein the instructions, when executed by the at least one processor, further causes the at least one processor to perform additional operations including:

determining, using the first data, whether the detected object is moving toward or away from the device;

determining, using the second data, whether one or more fingers associated with the detected object are spread or closed; and

determining at least one of a zoom-in gesture and a zoom-out gesture based on the determination of whether the detected object is moving toward or away from the device and the determination of whether the one or more fingers associated with the detected object are spread or closed.

34. The non-transitory computer-readable medium of claim 32, wherein the instructions, when executed by the at least one processor, further causes the at least one processor to perform an additional operation including distinguishing between a clockwise circular gesture and a counter-clockwise circular gesture.

35. The non-transitory computer-readable medium of claim 32, wherein the instructions, when executed by the at least one processor, further causes the at least one processor to perform an additional operation including determining, using the first data, whether the object performed a gesture parallel to a display.