Hover detection

• US 8,515,128 B1
• Filed: 10/07/2010
• Issued: 08/20/2013
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US12/900,183
• Time in Patent Office: 1,048 Days
• Field of Search: 382/100, 382/103, 382/107, 382/236, 348/154, 348/155, 348169-172, 348/143, 345/156, 345/157, 345/158
• US Class Current: 382/103
• CPC Class Codes: E05Y 2800/106 Lighting
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/56 provided with illuminating means

Abstract

Hover detection technology, in which an image is captured from a camera while an illumination source is illuminating an area in front of a display surface and the image captured by the camera is analyzed to detect an object within an anticipated input region based on illumination of the illumination source. User input is determined based on the object detected within the anticipated input region and an application is controlled based on the determined user input.

Claims

1. An electronic system comprising:

a display device configured to display images at a display surface where users are able to perceive images displayed by the display device;

a camera having a field of view of a first area in front of the display surface, the first area in front of the display surface being an area in which users are able to perceive the display surface;

an illumination source that is configured to illuminate a second area in front of the display surface, the second area intersecting the first area to define an anticipated input region within the field of view of the camera, the anticipated input region being less than all of the field of view of the camera and corresponding to a location in front of the display surface where physical objects are expected to be positioned when user input is being provided to the electronic system and where physical objects are expected to be absent when user input is not being provided to the electronic system; and

a processing unit configured to perform operations comprising;

capturing an image from the camera while the illumination source is illuminating the second area in front of the display surface;

analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source, wherein analyzing the image comprises;

comparing portions of the image captured by the camera to a brightness threshold that defines a brightness level at which an illuminated object should surpass, but an unilluminated object should not surpass;

identifying portions of the image captured by the camera that are above the brightness threshold based on the comparison; and

detecting the object within the anticipated input region based on the identified portions of the image captured by the camera that are above the brightness threshold;

determining user input based on the object detected within the anticipated input region; and

controlling an application based on the determined user input.

2. The electronic system of claim 1 wherein the illumination source is an infrared emitter.

3. The electronic system of claim 1 wherein the processing unit is configured to perform operations further comprising controlling the illumination source to illuminate the second area in front of the display surface while the camera is capturing the image to define the anticipated input region within the image captured by the camera.

4. The electronic system of claim 3:

wherein controlling the illumination source to illuminate the second area in front of the display surface while the camera is capturing the image to define the anticipated input region within the image captured by the camera comprises controlling the illumination source to illuminate the second area in front of the display surface in sequence with images captured by the camera in an alternating pattern such that a first image captured by the camera is captured when the illumination source is illuminated and a second image captured by the camera is captured when the illumination source is not illuminated; and

wherein analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source comprises;

subtracting the second image from the first image to produce a resulting image; and

analyzing the resulting image to detect an illuminated object.

5. The electronic system of claim 1 wherein analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source comprises ignoring objects that are within the camera'"'"'s field of view, not illuminated by the illumination source, and outside of the anticipated input region.

6. The electronic system of claim 1 wherein comparing portions of the image captured by the camera to a brightness threshold that defines a brightness level at which an illuminated object should surpass, but an object that is not illuminated should not surpass comprises:

accessing a background model of the first area in front of the display surface included in the field of view of the camera;

subtracting the background model from the image captured by the camera to produce a resulting image; and

comparing portions of the resulting image to the brightness threshold that defines the brightness level at which an illuminated object should surpass, but an object that is not illuminated should not surpass.

7. The electronic system of claim 1:

wherein the camera has a field of view of the first area in front of the display surface that includes an area that extends more than one inch from the display surface;

wherein the illumination source is oriented to illuminate the second area in front of the display surface that includes an area that extends one inch or less from the display surface over a majority of the display surface;

wherein the second area intersects the first area to define an anticipated input region that extends one inch or less from the display surface over a majority of the display surface;

wherein analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source comprises analyzing the image captured by the camera to detect an object within one inch from the display surface based on illumination of the illumination source; and

wherein determining user input based on the object detected within the anticipated input region comprises detecting a touch input based on the object detected within one inch from the display surface; and

wherein controlling an application based on the determined user input comprises controlling an application based on the detected touch input.

8. The electronic system of claim 1:

wherein the camera has a field of view of the first area in front of the display surface that includes an area that extends more than one inch from the display surface;

wherein the illumination source is oriented to illuminate the second area in front of the display surface that includes an area that extends, on average for a portion covering the display surface, one inch or less from the display surface;

wherein the second area intersects the first area to define an anticipated input region that extends, on average for the portion covering the display surface, one inch or less from the display surface;

wherein analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source comprises analyzing the image captured by the camera to detect an object within one inch from the display surface based on illumination of the illumination source; and

wherein determining user input based on the object detected within the anticipated input region comprises detecting a touch input based on the object detected within one inch from the display surface; and

wherein controlling an application based on the determined user input comprises controlling an application based on the detected touch input.

9. The electronic system of claim 1 wherein the illumination source is a first illumination source that is positioned at a first side of the display surface, further comprising:

a second illumination source that is positioned at a second side of the display surface that is different than the first side of the display surface and that is configured to illuminate a third area, the third area intersecting the first area, being different than the second area, and, in combination with the second area, defines a combined anticipated input region that covers the display surface by combining a first subset of an image captured by the camera when the first illumination source was illuminated and the second illumination source was not illuminated with a second subset of an image captured by the camera when the first illumination source was not illuminated and the second illumination source was illuminated; and

wherein the processing unit is configured to perform operations further comprising;

controlling the first and second illumination sources to illuminate in sequence with images captured by the camera in an alternating pattern;

identifying a first image captured when the first illumination source was illuminated and the second illumination source was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination source was illuminated;

extracting a first subset of the first image that covers a first subset of the display surface;

extracting a second subset of the second image that covers a second subset of the display surface, the first subset of the display surface and the second subset of the display surface covering the display surface when taken together;

analyzing the first subset of the first image and the second subset of the second image in combination to determine a position of an object within the combined anticipated input region defined by the first and second illumination sources.

10. The electronic system of claim 1:

wherein the camera is positioned at a first side of the display surface, is angled with respect to the display surface, and has a field of view of the first area in front of the display surface that includes an area that extends more than six inches from the display surface;

wherein the illumination source is positioned at a second side of the display surface that is opposite of the first side of the display surface, is angled with respect to the display surface, and illuminates the second area in front of the display surface that extends more than six inches from the display surface;

wherein the second area intersects the first area to define an anticipated input region that extends six inches or less from the display surface; and

wherein analyzing the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source comprises analyzing the image captured by the camera to detect an object within six inches from the display surface based on illumination of the illumination source.

11. The electronic system of claim 10 wherein the illumination source is a first illumination source, further comprising:

a second illumination source that is positioned at the second side of the display surface that is opposite of the first side of the display surface, that is angled with respect to the display surface differently than the first illumination source, and that is configured to illuminate a third area, the third area intersecting the first area, being different than the second area, and, in combination with the second area, defines a combined anticipated input region within the field of view of the camera; and

wherein the processing unit is configured to perform operations further comprising;

controlling the first and second illumination sources to illuminate in sequence with images captured by the camera in an alternating pattern;

identifying a first image captured when the first illumination source was illuminated and the second illumination source was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination source was illuminated; and

analyzing the first and second images in combination to determine a position of an object within the combined anticipated input region defined by the first and second illumination sources.

12. The electronic system of claim 11, further comprising:

a third illumination source that is positioned at the second side of the display surface that is opposite of the first side of the display surface, that is angled with respect to the display surface differently than the first illumination source and the second illumination source, and that is configured to illuminate a fourth area, the fourth area intersecting the first area, being different than the second area and the third area, and, in combination with the second area and the third area, defines the combined anticipated input region within the field of view of the camera; and

wherein the processing unit is configured to perform operations further comprising;

controlling the first, second, and third illumination sources to illuminate in sequence with images captured by the camera in an alternating pattern;

identifying a first image captured when the first illumination source was illuminated and the second and third illumination sources were not illuminated;

identifying a second image captured when the second illumination source was illuminated and the first and third illumination sources were not illuminated;

identifying a third image captured when the third illumination source was illuminated and the first and second illumination sources were not illuminated; and

analyzing the first, second, and third images in combination to determine a position of an object within the combined anticipated input region defined by the first, second, and third illumination sources.

13. The electronic system of claim 1 wherein the camera is a first camera, further comprising:

a second camera having a field of view of a third area in front of the display surface, the third area in front of the display surface being an area in which users are able to perceive the display surface and being different than the first area; and

wherein the processing unit is configured to perform operations further comprising;

capturing multiple images from the first and second cameras while the illumination source is illuminating the second area in front of the display surface;

detecting multiple objects within the multiple images based on illumination from the illumination source;

determining a position of the detected multiple objects relative to the display surface based on the multiple camera images; and

determining a multiple object user input command based on the determined positions.

14. The electronic system of claim 13 wherein determining a position of the detected multiple objects relative to the display surface based on the multiple camera images comprises:

determining a first position of a first object within a first image captured by the first camera;

determining a second position of the first object within a second image captured by the second camera;

determining a third position of a second object within the first image captured by the first camera;

determining a fourth position of the second object within the second image captured by the second camera;

determining a position of the first object relative to the display surface by triangulating the first position of the first object within the first image captured by the first camera and the second position of the first object within the second image captured by the second camera; and

determining a position of the second object relative to the display surface by triangulating the third position of the second object within the first image captured by the first camera and the fourth position of the second object within the second image captured by the second camera.

15. The electronic system of claim 14 wherein determining a position of the detected multiple objects relative to the display surface based on the multiple camera images further comprises:

determining that ambiguity exists in the position of the first object and the position of the second object; and

in response to the determination that ambiguity exists in the position of the first object and the position of the second object, estimating an expected position of the first object and the second object to resolve the ambiguity.

16. The electronic system of claim 13:

wherein detecting multiple objects within the multiple images based on illumination from the illumination source comprises;

detecting first and second objects within a first image captured by the first camera,

detecting only the first object within a second image captured by the second camera, and

determining that the second object is occluded in the second image captured by the second camera based on detecting first and second objects within the first image captured by the first camera and detecting only the first object within a second image captured by the second camera; and

wherein determining a position of the detected multiple objects relative to the display surface based on the multiple camera images comprises estimating a position of the second object relative to the display surface based on a position of the second object within the first image and the determination that the second object is occluded in the second image.

17. The electronic system of claim 1 wherein determining user input based on the object detected within the anticipated input region comprises:

mapping a position of the detected object to an interface displayed by the application being controlled; and

determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled.

18. The electronic system of claim 17 wherein determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled comprises determining whether the mapped position of the detected object corresponds to an element displayed in the interface displayed by the application being controlled.

19. The electronic system of claim 17:

wherein mapping the position of the detected object to the interface displayed by the application being controlled comprises mapping the position of the detected object to a cursor position in the interface displayed by the application being controlled; and

wherein determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled comprises determining user input based on the cursor position in the interface displayed by the application being controlled.

20. A method, comprising:

capturing an image of a field of view of a first area in front of a display surface at which images are displayed, the first area in front of the display surface being an area in which users are able to perceive the display surface;

illuminating a second area in front of the display surface while the image is being captured, the second area intersecting the first area to define an anticipated input region within the field of view, the anticipated input region being less than all of the field of view and corresponding to a location in front of the display surface where physical objects are expected to be positioned when user input is being provided to an electronic system and where physical objects are expected to be absent when user input is not being provided to the electronic system;

analyzing the captured image to detect an object within the anticipated input region based on the illuminating, wherein analyzing the image comprises;

comparing portions of the image to a brightness threshold that defines a brightness level at which an illuminated object should surpass, but an unilluminated object should not surpass;

identifying portions of the image that are above the brightness threshold based on the comparison; and

detecting the object within the anticipated input region based on the identified portions of the image that are above the brightness threshold;

determining user input based on the object detected within the anticipated input region; and

controlling an application based on the determined user input.

21. An apparatus, comprising:

means for displaying images at a display surface where users are able to perceive images displayed;

means for capturing an image having a field of view of a first area in front of the display surface, the first area in front of the display surface being an area in which users are able to perceive the display surface;

means for illuminating a second area in front of the display surface, the second area intersecting the first area to define an anticipated input region within the field of view of the means for capturing the image, the anticipated input region being less than all of the field of view of the means for capturing the image and corresponding to a location in front of the display surface where physical objects are expected to be positioned when user input is being provided and where physical objects are expected to be absent when user input is not being provided, wherein the means for capturing the image captures the image while the means for illuminating the second area in front of the display surface is illuminating the second area;

means for analyzing the image captured by the means for capturing the image to detect an object within the anticipated input region based on illumination by the means for illuminating, wherein the means for analyzing the image comprises;

means for comparing portions of the image captured by the means for capturing the image to a brightness threshold that defines a brightness level at which an illuminated object should surpass, but an unilluminated object should not surpass;

means for identifying portions of the image captured by the means for capturing the image that are above the brightness threshold based on the comparison; and

means for detecting the object within the anticipated input region based on the identified portions of the image captured by the means for capturing the image that are above the brightness threshold;

means for determining user input based on the object detected within the anticipated input region; and

means for controlling an application based on the determined user input.

22. A computer program product residing on a non-transitory processor-readable medium, the computer program product comprising processor-readable instructions configured to cause a processor to:

capture an image from a camera, wherein;

the camera has a field of view of a first area in front of a display surface, the first area in front of the display surface being an area in which users are able to perceive the display surface; and

the image is captured while an illumination source is illuminating a second area in front of the display surface, the second area intersecting the first area to define an anticipated input region within the field of view of the camera, the anticipated input region being less than all of the field of view of the camera and corresponding to a location in front of the display surface where physical objects are expected to be positioned when user input is being provided to the processor and where physical objects are expected to be absent when user input is not being provided to the processor;

analyze the image captured by the camera to detect an object within the anticipated input region based on illumination of the illumination source, wherein the processor-readable instructions configured to cause the processor to analyze the image comprises processor-readable instructions configured to cause the processor to;

compare portions of the image captured by the camera to a brightness threshold that defines a brightness level at which an illuminated object should surpass, but an unilluminated object should not surpass;

identify portions of the image captured by the camera that are above the brightness threshold based on the comparison; and

detect the object within the anticipated input region based on the identified portions of the image captured by the camera that are above the brightness threshold;

determine user input based on the object detected within the anticipated input region; and

control an application based on the determined user input.