User input using proximity sensing

• US 8,432,372 B2
• Filed: 11/30/2007
• Issued: 04/30/2013
• Est. Priority Date: 11/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Butler, David Alexander, Izadi, Shahram, Hodges, Stephen E., Hall, Malcolm
• Primary Examiner(s): Lao, Lun-Yi
• Assistant Examiner(s): Abebe, Sosina
• Application Number:
  US11/948,802
• Publication Number:
  US 20090139778A1
• Time in Patent Office: 1,978 Days
• Field of Search: 345156-158, 345173-178, 345/207, 178/18.09, 178/18.03, 178 1801- 2001
• US Class Current: 345/174
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 2203/0339 Touch strips, e.g. orthogonal touch strips to control cursor movement or scrolling; single touch strip to adjust parameter or to implement a row of soft keys
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A device is described which enables users to interact with software running on the device through gestures made in an area adjacent to the device. In an embodiment, a portable computing device has proximity sensors arranged on an area of its surface which is not a display, such as on the sides of the device. These proximity sensors define an area of interaction adjacent to the device. User gestures in this area of interaction are detected by creating sensing images from data received from each of the sensors and then analyzing sequences of these images to detect gestures. The detected gestures may be mapped to particular inputs to a software program running on the device and therefore a user can control the operation of the program through gestures.

Claims

1. Control operation of the plurality of proximity sensing points by adjusting one or more of resolution and sensitivity of gesture detection in the interaction area, wherein a number of operational proximity sensing points increases from a first number to a second number after motion is detected in the interaction area based on an application being executed on the device;

a processor; and

a memory arranged to store executable instructions, which when executed by the processor, cause the processor to;

control operation of the plurality of proximity sensing points by adjusting one or more of resolution and sensitivity of gesture detection in the interaction area based on an application being executed on the device, wherein a number of operational proximity sensing points increases from a first number to a second number after motion is detected in the interaction area;

create a sequence of proximity sensor images from data received from the second number of operational proximity sensing points; and

analyze the sequence of proximity sensor images to identify a user gesture in the interaction area.

2. A device according to claim 1, wherein the memory is further arranged to store executable instructions which when executed by the processor, cause the processor to:

control a software program running on the device based on the identified gesture.

3. A device according to claim 1, wherein the memory is further arranged to store executable instructions which when executed by the processor, cause the processor to:

map the identified gesture to a control input to a software program running on the device.

4. A device according to claim 1, wherein each proximity sensing point comprises a proximity sensor.

5. A device according to claim 4, wherein each proximity sensor comprises an emitter and a detector.

6. A device according to claim 5, wherein each emitter is modulated and data received at each detector is filtered.

7. A device according to claim 1, further comprising:

at least one emitter;

at least one detector; and

a plurality of waveguides,

and wherein each proximity sensing point is optically connected to a detector by one of the plurality of waveguides and each proximity sensing point is optically connected to an emitter by one of the plurality of waveguides.

8. A device according to claim 1, wherein each proximity sensor image comprises a map of intensities detected at each of the operational proximity sensing points in the second number of operational proximity sensing points.

9. A device according to claim 1, wherein analyzing the sequence of proximity sensing images comprises:

subtracting a background image from each image to create a processed version of each image; and

analyzing a sequence of processed versions of images to detect the user gesture.

10. A device according to claim 1, wherein analyzing the sequence of proximity sensing images comprises:

analyzing one of the sequence of proximity sensing images; and

determining a position of a sensed object from the one of the sequence of proximity sensing images.

11. A device according to claim 1, wherein the non-display portion of the device comprises one or more sides of the device.

12. A device according to claim 1, wherein the device comprises a portable computing device.

13. A device according to claim 1, wherein the device comprises a cover for a portable computing device.

15. A device according to claim 1, wherein the first number of operational proximity sensing points is associated with a reduced power consumption state and the second number of operational proximity sensing points is associated with an expected gesture detection state associated with the application being executed on the device.

14. One or more tangible, device-readable media with device-executable instructions for performing operations comprising:

determining an operational arrangement of a plurality of proximity sensing points on a surface of a device based on an application running on the device;

collecting data by the plurality of proximity sensing points, the data being collected from an interaction area that extends outward from a display portion housed in the device and in a plane of a display portion housed in the device;

generating a sequence of proximity sensing images from the data; and

analyzing the sequence of proximity sensing images to detect a gesture.

16. One or more tangible, device-readable media according to claim 14 further comprising device-executable instructions for mapping the gesture to an input to the application running on the device.

17. One or more tangible, device-readable media according to claim 16, further comprising device-executable instructions for controlling the application according to the input.

18. One or more tangible, device-readable media according to claim 14, wherein analyzing the sequence of proximity sensing images comprises:

subtracting a background image from each image to create a processed version of each image; and

analyzing a sequence of processed versions of images to detect the gesture.

19. One or more tangible, device-readable media according to claim 14, wherein each proximity sensing point comprises an emitter and a detector and wherein the emitter is modulated, and wherein collecting the data by the plurality of proximity sensing points comprises:

filtering a signal received at each detector to produce filtered data; and

collecting the filtered data from each detector.

20. A handheld computing device comprising:

a plurality of operational proximity sensors arranged in a pattern along a side of the handheld computing device, the pattern of the operationally plurality of proximity sensors being configured to define an interaction zone based on an active application on the handheld computing device; and

a processing apparatus arranged to;

collect data from each operational proximity sensor;

generate a sequence of proximity sensing images from the data; and

analyze the sequence of proximity sensing images to detect a gesture.