Hover detection

• US 8,897,496 B2
• Filed: 07/03/2013
• Issued: 11/25/2014
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan Robert
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US13/934,734
• Publication Number:
  US 20140028628A1
• Time in Patent Office: 510 Days
• Field of Search: 382/100, 382/103, 382/107, 382/236, 348/154, 348/155, 348/169, 348/170, 348/172, 356/156, 356/157, 356/158
• US Class Current: 382/103
• CPC Class Codes: E05Y 2800/106 Lighting
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/56 provided with illuminating means

Abstract

Hover detection technology, in which an image is captured from a camera while an illumination source is illuminating an area in front of a display surface and the image captured by the camera is analyzed to detect an object within an anticipated input region based on illumination of the illumination source. User input is determined based on the object detected within the anticipated input region and an application is controlled based on the determined user input.

Claims

1. An electronic system comprising:

an image sensor;

an illumination source configured to illuminate a first area in front of a surface; and

a processing unit configured to perform operations comprising;

receiving information captured by the image sensor while the illumination source is illuminating the first area in front of the surface; and

analyzing the information received from the image sensor to determine user input, wherein analyzing the information comprises;

detecting one or more illuminated objects illuminated by the illumination source;

based on detecting the illuminated objects illuminated by the illumination source, determining that the illuminated objects are within a predetermined distance from the surface; and

determining user input based on the objects determined to be within the predetermined distance from the surface.

2. The electronic system of claim 1 wherein the processing unit is configured to perform operations further comprising:

controlling an illumination source to illuminate a second area while the camera is capturing information to define the anticipated input region.

3. The electronic system of claim 1, wherein:

the image sensor has a field of view of the first area in front of the surface; and

the illumination source is further configured to illuminate a second area in front of the surface, the second area intersecting the first area to define an anticipated input region within the field of view of the image sensor.

4. The electronic system of claim 3 wherein the image sensor comprises a first camera, the electronic system further comprising:

a second camera having a second field of view of a third area in front of the surface; and

wherein the processing unit is configured to perform operations further comprising;

receiving images from the first camera and the second camera while the illumination source is illuminating the second area in front of the surface;

detecting a first object and a second object within the received images based on illumination from the illumination source, wherein the first object is the object;

determining positions of the detected first and second objects based on the received camera images; and

determining a multiple object user input command based on the determined positions.

5. The electronic system of claim 3 wherein the illumination source is a first illumination source positioned at a first position, the electronic system further comprising:

a second illumination source positioned at a second position, the second illumination source configured to;

illuminate a third area, the third area intersecting the first area and the second area; and

wherein the information comprises a first image and the processing unit is configured to perform operations further comprising;

identifying the first image captured when the first illumination source was illuminated and the second illumination source was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination source was illuminated; and

analyzing the first image and the second image to determine a position of the object.

6. The electronic system of claim 1, wherein the image sensor comprises a camera and wherein the processing unit is configured to perform operations further comprising:

controlling the illumination source to illuminate the first area in front of the surface in sequence with images captured by the camera in an alternating pattern such that a first image captured by the camera is captured when the illumination source is illuminated and a second image captured by the camera is captured when the illumination source is not illuminated, wherein the information comprises the first image.

7. The electronic system of claim 1 wherein analyzing the information by the processing unit further comprises:

ignoring one or more objects that are within a field of view of the image sensor but not illuminated by the illumination source.

8. The electronic system of claim 1 wherein the predetermined distance is between one and six inches.

9. The electronic system of claim 1 wherein the predetermined distance is less than or equal to one inch.

10. The electronic system of claim 1 wherein the illumination source is a first illumination source positioned at a first position, the electronic system further comprising:

a second illumination source positioned at a second position, the second illumination source configured to;

illuminate a third area, the third area intersecting the first area and a second area;

wherein the information comprises a first image and the processing unit is configured to perform operations further comprising;

identifying the first image captured when the first illumination source was illuminated and the second illumination source was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination source was illuminated; and

analyzing the first image and the second image to determine a position of the object.

11. The electronic system of claim 1 wherein determining user input based on the object detected within the anticipated input region comprises:

mapping a position of the detected object to an interface displayed by an application being controlled; and

determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled.

12. The electronic system of claim 1 wherein the surface comprises a surface of a display device and wherein determining user input based on the objects determined to be within the predetermined distance from the surface comprises:

mapping a position of the detected objects to a position on the surface of the display device, wherein the detected object is a finger; and

determining user input based on the mapped position of the detected object to the surface of the display device.

13. The electronic system of claim 1 wherein the information comprises first information and the image sensor comprises a first image sensor, the electronic system further comprising:

a second image sensor configured to capture second information; and

wherein the processing unit is configured to perform operations further comprising;

triangulating a position of the detected object based on the first information and the second information.

14. The electronic system of claim 1, wherein the one or more objects comprise a first object, the processing unit is configured to perform operations further comprising:

determining user input based on a combination of the first object determined to be within the predetermined distance from the surface and a second object determined to be within the predetermined distance from the surface.

15. The electronic system of claim 1 wherein the one or more objects comprise a finger and wherein determining the user input comprises determining that the finger is hovering within the predetermined distance from the surface.

16. The electronic system of claim 1 wherein the one or more objects comprise a finger and wherein determining the user input comprises determining that the finger is touching the surface.

17. The electronic system of claim 1 wherein the one or more objects comprise a plurality of fingers, wherein the surface comprises a display screen of a display device, and wherein determining the user input comprises determining a multi-touch user input based on the plurality of fingers touching the display screen.

18. The electronic system of claim 1 wherein the surface comprises a surface of a display, and wherein the image sensor and the illumination source are both positioned at a bottom edge of the display.

19. A method for determining user input, the method comprising:

receiving an image from a camera while an illumination source is illuminating a first area in front of a surface of a display device; and

analyzing the image received from the camera to determine whether an object within an anticipated input region is substantially in contact with the surface using illumination of the illumination source, wherein analyzing the image comprises;

distinguishing illuminated objects illuminated by the illumination source from unilluminated objects unilluminated by the illumination source;

based on distinguishing the illuminated objects illuminated by the illumination source, determining that the illuminated objects are within a predetermined distance from the surface of the display device;

detecting an object within the anticipated input region based on determining that the illuminated objects are within a predetermined distance from the surface of the display device; and

determining user input based on the object detected within the anticipated input region.

20. The method for determining user input of claim 19, the method further comprising:

controlling the illumination source to illuminate a second area while the camera is capturing the image to define the anticipated input region.

21. The method for determining user input of claim 20, wherein controlling the illumination source comprises:

controlling the illumination source to illuminate the second area in front of the surface in sequence with images captured by the camera in an alternating pattern such that a first image captured by the camera is captured when the illumination source is illuminated and a second image captured by the camera is captured when the illumination source is not illuminated, wherein the image is the first image.

22. The method for determining user input of claim 19 wherein analyzing the image further comprises:

ignoring one or more objects that are within the camera'"'"'s field of view but not illuminated by the illumination source.

23. The method for determining user input of claim 19 wherein the predetermined distance is between one and six inches.

24. The method for determining user input of claim 19 wherein the predetermined distance is less than or equal to one inch.

25. The method for determining user input of claim 19 wherein the illumination source is a first illumination source positioned at a first position;

the method further comprising;

illuminating a third area using a second illumination source, the third area intersecting the first area and a second area;

wherein the image is a first image and the processing unit is configured to perform operations further comprising;

identifying the first image captured when the first illumination source was illuminated and the second illumination source was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination source was illuminated; and

analyzing the first image and the second image to determine a position of the object.

26. The method for determining user input of claim 19, wherein the object is a first object, the method further comprising:

determining user input based on a combination of the first object detected within the anticipated input region and a second object detected within anticipated input region.

27. A non-transitory processor-readable medium for determining user input, comprising processor-readable instructions configured to cause one or more processors to:

receive an image from a camera while an illumination source is illuminating an area in front of a surface of a display device; and

analyzing the image received from the camera to determine whether an object within an anticipated input region is substantially in contact with the surface using illumination of the illumination source, wherein the processor-readable instructions configured to cause the one or more processors to analyze the image comprise processor-readable instructions configured to cause the one or more processors to;

distinguish illuminated objects illuminated by the illumination source from unilluminated objects unilluminated by the illumination source;

based on distinguishing the illuminated objects illuminated by the illumination source, determine that the illuminated objects are within a predetermined distance from the surface of the display device;

detect an object within the anticipated input region based on determining that the illuminated objects are within a predetermined distance from the surface of the display device; and

determine user input based on the object detected within the anticipated input region.

28. The non-transitory processor-readable medium for determining user input of claim 27, wherein the processor-readable instructions configured to cause the one or more processors to control the illumination source comprise processor-readable instructions configured to cause the one or more processors to:

control the illumination source to illuminate the second area in front of the surface in sequence with images captured by the camera in an alternating pattern such that a first image captured by the camera is captured when the illumination source is illuminated and a second image captured by the camera is captured when the illumination source is not illuminated, wherein the image is the first image.

29. The non-transitory processor-readable medium for determining user input of claim 27 wherein the object is a first object, the processor-readable instructions are further configured to cause the one or more processors to:

determine user input based on a combination of the first object detected within the anticipated input region and a second object detected within anticipated input region.

30. An apparatus for determining user input, the apparatus comprising:

means for receiving an image from a camera while an illumination source is illuminating a first area in front of a surface of a display device; and

means for analyzing the image received from the camera to determine whether an object within an anticipated input region is substantially in contact with the surface using illumination of the illumination source, wherein the means for analyzing the image comprises;

means for distinguishing illuminated objects illuminated by the illumination source from unilluminated objects unilluminated by the illumination source;

based on distinguishing the illuminated objects illuminated by the illumination source, means for determining that the illuminated objects are within a predetermined distance from the surface of the display device;

means for detecting an object within the anticipated input region based on determining that the illuminated objects are within a predetermined distance from the surface of the display device; and

means for determining user input based on the object detected within the anticipated input region.