Pointing device using proximity sensing

• US 9,703,398 B2
• Filed: 06/16/2009
• Issued: 07/11/2017
• Est. Priority Date: 06/16/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Butler, David Alexander, Villar, Nicolas, Helmes, John, Izadi, Shahram, Hodges, Stephen E., Benko, Hrvoje, Rosenfeld, Daniel
• Primary Examiner(s): Spar, Ilana
• Assistant Examiner(s): Rayan, Mihir K
• Application Number:
  US12/485,593
• Publication Number:
  US 20100315336A1
• Time in Patent Office: 2,947 Days
• Field of Search: 345173-178
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence

Abstract

A pointing device using proximity sensing is described. In an embodiment, a pointing device comprises a movement sensor and a proximity sensor. The movement sensor generates a first data sequence relating to sensed movement of the pointing device relative to a surface. The proximity sensor generates a second data sequence relating to sensed movement relative to the pointing device of one or more objects in proximity to the pointing device. In embodiments, data from the movement sensor of the pointing device is read and the movement of the pointing device relative to the surface is determined. Data from the proximity sensor is also read, and a sequence of sensor images of one or more objects in proximity to the pointing device are generated. The sensor images are analyzed to determine the movement of the one or more objects relative to the pointing device.

Claims

1. A pointing device, comprising:

a housing;

a movement sensor arranged in the housing to generate a first data sequence relating to sensed movement of the pointing device relative to a surface on which the pointing device is placed;

a proximity sensor arranged in the housing to generate a second data sequence relating to sensed movement of one or more objects in proximity to the surface on which the pointing device is placed relative to the pointing device, the proximity sensor further arranged to detect a multi-touch gesture, detecting the multi-touch gesture comprising detecting at least one user-manipulated object;

a tag sensor arranged to read a barcode or RFID tag associated with the one or more objects;

a processor located within the housing and configured to determine a control input from the first data sequence and the second data sequence; and

the sensors being arranged such that a user can provide the first and second data sequences simultaneously using one hand.

2. The pointing device according to claim 1, wherein the proximity sensor comprises at least one emitter and at least one detector.

3. The pointing device according to claim 1, wherein the one or more objects comprise at least one of:

a stylus;

a pen;

a brush;

a further pointing device;

or a mobile telephone.

4. The pointing device according to claim 1, wherein the proximity sensor comprises an illumination source arranged to illuminate the one or more objects, and a camera arranged to capture images of the one or more objects.

5. The pointing device according to claim 1, wherein the proximity sensor comprises at least one of:

an optical proximity sensor;

an ultrasonic proximity sensor;

an inductive proximity sensor;

an electrostatic proximity sensor;

a capacitive proximity sensor;

a thermal imaging sensor;

an acoustic sensor;

or a magnetic proximity sensor.

6. The pointing device according to claim 1, further comprising a projection device arranged to project a visual indication of an interaction zone in which the proximity sensor can sense the movement of the one or more objects.

7. The pointing device according to claim 1, further comprising a transmitter arranged to transmit an electrostatic signal into a first location of a hand of a user in contact with the pointing device, and wherein the proximity sensor is arranged to detect the electrostatic signal being emitted from a second location of the hand of the user, the first location being different from the second location.

8. The pointing device according to claim 1, further comprising a communication interface arranged to transmit the first and second data sequence to a user terminal.

9. The pointing device according to claim 1 wherein the second data sequence comprises a sequence of sensor images of the one or more objects, the sequence of sensor images being generated at least in part by ignoring background objects based at least in part on relative movement.

10. The pointing device according to claim 1 wherein the processor is further configured to control the launch of a software application or function in response to detection of a predefined object or code by at least one of the tag or proximity sensor.

11. One or more tangible device-readable media, the tangible device-readable media being hardware, with device-executable instructions comprising:

device-executable instructions to read data from a movement sensor of a pointing device and determine a movement of the pointing device relative to a surface on which the pointing device rests, the pointing device comprising at least one of;

a pointing device comprising a movement sensor, a proximity sensor, and a tag sensor arranged such that a user can provide a data sequence generated by the movement sensor and a data sequence generated by the proximity sensor simultaneously with one hand;

or

a computer mouse comprising an electrostatic proximity sensor arranged to detect a signal generated by the computer mouse traveling through at least a hand of the user and being emitted from one or more fingers of the user;

device-executable instructions to read data from a first proximity sensor of the pointing device and generate a sequence of first proximity sensor images of one or more objects in proximity to the surface on which the pointing device rests, wherein the one or more images are captured by the first proximity sensor;

device-executable instructions to analyze the first proximity sensor images to determine a movement of the one or more objects relative to the pointing device;

device-executable instructions to read data from at least one second proximity sensor, the at least one second proximity sensor located on the underside of the pointing device, and analyzing the data from the at least one second proximity sensor to determine at least a tilt of the pointing device; and

device-executable instructions to control a position of a cursor in a user interface such that movement of the pointing device relative to the surface on which the pointing device rests causes a different displacement of the cursor than a corresponding movement of the one or more objects relative to the pointing device, the movement of the pointing device relative to the surface on which the pointing device rests and the corresponding movement having a same distance and a same direction.

12. One or more tangible device-readable media according to claim 11, further comprising:

device-executable instructions to analyze the movement of the one or more objects in proximity to the surface on which the pointing device rests; and

device-executable instructions to identify a user gesture based on the analyzing the movement of the one or more objects.

13. One or more tangible device-readable media according to claim 12, further comprising:

device-executable instructions to control a software program in accordance with the identified gesture.

14. One or more tangible device-readable media according to claim 11, wherein the steps of reading data from the movement sensor and reading data from at least the first proximity sensor are performed substantially concurrently.

15. One or more tangible device-readable media according to claim 11, wherein the movement of the pointing device relative to the surface on which the pointing device rests causes a larger displacement of the cursor than a corresponding movement of the one or more objects relative to the pointing device.

16. One or more tangible device-readable media according to claim 11, wherein generating the sequence of first proximity sensor images of one or more objects in proximity to the pointing device comprises, for each first proximity sensor image in the sequence:

subtracting a background image from the proximity sensor image;

generating a processed version of the first proximity sensor image based on the subtracting;

locating the one or more objects in the processed version of the first proximity sensor image; and

estimating a position of the one or more objects based on the locating.

17. One or more tangible device-readable media according to claim 16, wherein subtracting the background image from the first proximity sensor image comprises:

assigning a pixel value for each pixel in the first proximity sensor image;

comparing the pixel value to a predefined threshold;

in response to the comparing, setting the pixel color to black when the pixel value is less than or equal to the predefined threshold;

in response to the comparing, setting the pixel color to white when the pixel value is greater than the predefined threshold; and

removing the black pixels from the first proximity sensor image.

18. One or more tangible device-readable media according to claim 17, wherein the generating the processed version of the first proximity sensor image based on the subtracting comprises:

analyzing the white pixels;

identifying one or more objects based on the analyzing; and

labeling each of the identified one or more objects based on the identifying.

19. One or more tangible device-readable media according to claim 11, wherein the device-executable instructions are performed on a user terminal in communication with the pointing device.

20. A computer mouse, comprising:

a housing;

a movement sensor arranged with the housing to generate a first data sequence relating to sensed movement of the computer mouse relative to a supporting surface on which the computer mouse operates;

an electrostatic proximity sensor arranged with the housing to generate a second data sequence relating to sensed movement relative to the computer mouse of one or more fingers of a user in proximity to the supporting surface on which the computer mouse operates, the sensed movement of the one or more fingers being determined by the proximity sensor detecting a signal generated by the mouse, traveling through at least a hand of the user, and being emitted from the one or more fingers; and

a processor located within the housing and configured to determine a control input from the first data sequence and the second data sequence.