User input using proximity sensing

• US 9,063,577 B2
• Filed: 04/28/2013
• Issued: 06/23/2015
• Est. Priority Date: 11/30/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hodges, Stephen E., Butler, Alex, Izadi, Shahram, Hall, Malcolm
• Primary Examiner(s): Kumar, Srilakshmi K
• Assistant Examiner(s): Abebe, Sosina
• Application Number:
  US13/872,124
• Publication Number:
  US 20130234970A1
• Time in Patent Office: 786 Days
• Field of Search: 345156-183, 178/18.03
• US Class Current: 1/1
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 2203/0339 Touch strips, e.g. orthogonal touch strips to control cursor movement or scrolling; single touch strip to adjust parameter or to implement a row of soft keys
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0446 using a grid-like structure of electrodes in at least two directions, e.g. using row and column electrodes
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A device is described which enables users to interact with software running on the device through gestures made in an area adjacent to the device. In an embodiment, a portable computing device has proximity sensors arranged on an area of its surface which is not a display, such as on the sides of the device. These proximity sensors define an area of interaction adjacent to the device. User gestures in this area of interaction are detected by creating sensing images from data received from each of the sensors and then analyzing sequences of these images to detect gestures. The detected gestures may be mapped to particular inputs to a software program running on the device and therefore a user can control the operation of the program through gestures.

Claims

1. A device comprising:

a plurality of sensors, the plurality of sensors configured to detect user input in an interaction area that extends outwardly from a surface of a casing of the device and that is at least in a plane of a display portion associated with the device;

a processor; and

a memory arranged to store executable instructions, which when executed by the processor, cause the processor to;

control operation of the plurality of sensors by sensitivity of gesture detection in the interaction area, wherein a set of operational sensors changes from a first set comprising a plurality of high sensitivity sensors to a second set comprising at least a plurality of low sensitivity sensors after motion is detected in the interaction area based on an application being executed on the device;

create a sequence interaction data from data received from the second number of operational sensors; and

analyze the sequence interaction data to identify a user gesture in the interaction area.

2. A device according to claim 1, wherein the memory is further arranged to store executable instructions which when executed by the processor, cause the processor to:

control a software program running on the device based on the identified gesture.

3. A device according to claim 1, wherein the memory is further arranged to store executable instructions which when executed by the processor, cause the processor to:

map the identified gesture to a control input to a software program running on the device.

4. A device according to claim 1, wherein the plurality sensors are further configured to detect user input in the interaction area when the interaction area is devoid of any electronic devices.

5. A device according to claim 1, wherein the display portion is attached to the casing.

6. A device according to claim 1, wherein the plurality of sensors are arranged relative to each other in a sensor plane, the sensor plane being non-parallel to the plane of the display portion.

7. A device according to claim 1, further comprising:

at least one emitter;

at least one detector; and

a plurality of waveguides,

and wherein each proximity sensing point is optically connected to a detector by one of the plurality of waveguides and each proximity sensing point is optically connected to an emitter by one of the plurality of waveguides.

8. A device according to claim 1, wherein each proximity sensor image comprises a map of intensities detected at each of the plurality of proximity sensing points.

9. A device according to claim 1, wherein the plurality of sensors are arranged relative to each other in a sensor plane, the sensor plane being perpendicular to the plane of the display portion.

10. A device according to claim 1, wherein analyzing the sequence of proximity sensing images comprises:

subtracting a background image from each image to create a processed version of each image; and

analyzing a sequence of processed versions of images to detect a gesture.

11. A device according to claim 1, wherein analyzing the sequence of proximity sensing images comprises:

analyzing one of the sequence of proximity sensing images; and

determining a position of a sensed object from the one of the sequence of proximity sensing images.

12. A device according to claim 1, wherein the non-display portion of the device comprises one or more sides of the device.

13. A device according to claim 1, wherein the memory is further arranged to store executable instructions, which when executed by the processor, cause the processor to:

create a sequence of interaction data from data received from an active or passive tagged object.

14. A device according to claim 1, wherein the device comprises a cover for a portable computing device.

15. One or more tangible device-readable media, the tangible device-readable media being hardware, with device-executable instructions for performing steps comprising:

collecting data from a plurality of sensors, the plurality sensors configured to detect user input in an interaction area that extends outwardly from a surface of a casing of the device and that is at least in a plane of a display portion associated with the device;

controlling operation of the plurality of sensors by sensitivity of gesture detection in the interaction area, wherein a set of operational sensors changes from a first set comprising a plurality of high sensitivity sensors to a second set comprising at least a plurality of low sensitivity sensors after motion is detected in the interaction area based on an application being executed on the device;

creating a sequence interaction data from data received from the second number of operational sensors; and

analyzing the sequence interaction data to identify a user gesture in the interaction area.

16. One or more tangible device-readable media according to claim 15 further comprising device-executable instructions for performing steps comprising:

mapping the gesture to an input to software running on the device.

17. One or more tangible device-readable media according to claim 16, further comprising device-executable instructions for performing steps comprising:

controlling the software according to the input.

18. One or more tangible device-readable media according to claim 15, wherein analyzing the sequence of proximity sensing images comprises:

subtracting a background image from each image to create a processed version of each image; and

analyzing a sequence of processed versions of images to detect a gesture.

19. One or more tangible device-readable media according to claim 15, wherein each proximity sensing point comprises an emitter and a detector and wherein the emitter is modulated, and wherein collecting data from a plurality of proximity sensing points comprises:

filtering a signal received at each detector to produce filtered data; and

collecting filtered data from each detector.

20. A method comprising:

collecting data from a plurality of sensors, the plurality sensors configured to detect user input in an interaction area that extends outwardly from a surface of a casing of the device and that is at least in a plane of a display portion associated with the device;

controlling operation of the plurality of sensors by sensitivity of gesture detection in the interaction area, wherein a number of operational sensors changes from a first number to a second number after motion is detected in the interaction area based on an application being executed on the device;

creating a sequence interaction data from data received from the second number of operational sensors; and

allowing a user to store the sequence interaction data as a user-defined gesture.