Systems and methods using computer vision and capacitive sensing for cursor control

• US 7,692,627 B2
• Filed: 08/10/2004
• Issued: 04/06/2010
• Est. Priority Date: 08/10/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D
• Primary Examiner(s): Abdulselam; Abbas I
• Application Number:
  US10/914,923
• Publication Number:
  US 20060033701A1
• Time in Patent Office: 2,065 Days
• Field of Search: 345/173, 345/156, 178/18.01, 178/18.06, 715/856, 715/863, 250/206.1, 250/216, 250/221, 382/291
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Disclosed is a unique system and method that facilitates cursor control based in part on computer vision activated by a capacitive touch sensor. When turned on, user hand gestures or movements can be tracked by a monitoring component and those movements can be converted in real-time to control or drive cursor movements and/or position on a user interface. The system comprises a monitoring component or camera that can be activated by touch or pressure applied to a capacitive touch sensor. A circuit within the sensor determines when the user is touching a button (e.g., on keyboard or mouse) that activates the monitoring component and cursor control mechanism. Thus, intentional hand movements by the user can readily be determined.

Claims

1. A vision-based navigation control system comprising:

an input detection component that detects user input;

an image monitoring component that receives data from the input detection component which causes the image monitoring component to turn on or off, wherein the data is an amount of capacitive discharge, the image monitoring component is turned on when the amount of capacitive discharge satisfies a threshold and the image monitoring component is turned off when the amount of capacitive discharge does not satisfy the threshold; and

a navigational driving component that receives images captured by the image monitoring component to control position of a cursor;

wherein controlling position of the cursor comprises;

computing a plurality of flow fields based on images captured by the monitoring component to obtain a corresponding plurality of flow field vectors, wherein the images are a temporally consecutive sequence of image frames captured by the image monitoring component; and

performing at least one of the following;

averaging the plurality flow field vectors to ascertain a cursor position relative to a corresponding hand position, or

averaging spatial positions of all non-zero flow field vectors to ascertain an absolute cursor position with respect to the corresponding hand position.

2. The system of claim 1, the input comprises at least one of a touch or pressure applied to the input component by a user.

3. The system of claim 1, the input detection component comprises a capacitive touch sensor that computes an electrical reading based in part on the user-based input.

4. The system of claim 1 further comprises software installed on a host machine that can read a status of the input detection component to determine whether an amount of user-based input satisfies a threshold.

5. The system of claim 1, the input detection component comprises an electrode that can be placed on a surface of an input component.

6. The system of claim 1, further comprising an input component including any one of:

at least one button associated with a touchpad or mouse, and one or more keys of a keyboard.

7. The system of claim 1, the user-based input is convened to an electrical signal that is sensed or read by the detection component.

8. The system of claim 1, the image monitoring component comprises one or more cameras that can capture an image in motion frame by frame.

9. The system of claim 8 further comprising a component that computes flow fields for substantially every frame of an image captured by the image monitoring component in real-time.

10. The system of claim 9, the flow fields are generated to facilitate mapping the user-based input to coincide with controlling position of the cursor on a display screen to navigate through content.

11. The system of claim 9, the navigational driving component recognizes one or more gestures made by a user to determine where to position the cursor in real-time.

12. The system of claim 9, wherein the flow fields are generated by way of block matching to determine vectors that identify one or more directions of movement between at least any two frames of the image.

13. The system of claim 1, the image monitoring component is positioned to monitor the user'"'"'s “

visible”

movement within a viewable space as determined in part by the dimensional limits of the monitoring component as well as in part by user preferences.

14. A computer readable medium having stored thereon the computer executable components of claim 1.

15. A vision-based method that facilitates cursor control comprising:

detecting user-based input via a capacitive touch sensor;

determining whether the input satisfies an amount of discharge threshold;

monitoring user hand gestures occurring within an image monitoring region when the touch threshold is satisfied;

driving cursor position based in part on the user hand gestures, wherein controlling cursor position comprises;

computing a plurality of flow fields based on images of the user hand gestures captured from the image monitoring region to obtain a corresponding plurality of flow field vectors, wherein the images are a temporally consecutive sequence of image frames; and

performing at least one of the following;

averaging the plurality flow field vectors to ascertain a cursor position relative to a corresponding hand position of the user hand gestures, or

averaging spatial positions of all non-zero flow field vectors to ascertain an absolute cursor position with respect to the corresponding hand position of the user hand gestures; and

stopping monitoring of the image monitoring region when the touch threshold is no longer satisfied.

16. The method of claim 15, further comprising maintaining a current monitoring state depending on whether the touch threshold is satisfied.

17. The method of claim 15, the current monitoring state comprises one of:

on or off.

18. The method of claim 15, wherein detecting user-based input and determining whether the touch threshold is satisfied on a continuous basis to ascertain when to monitor user hand gesture activity in the monitoring region.

19. The method of claim 15, the monitoring region comprises an area that is viewable by a monitoring component.

20. The method of claim 19, the monitoring component comprises a camera interfaced to a computing device.

21. The method of claim 15, further comprising converting the user hand gestures into relative cursor movement or position.

22. The method of claim 15, wherein the user hand gestures comprises one or more hand movements.

23. The method of claim 15, wherein determining whether a touch threshold is satisfied comprises analyzing charging and discharging of a capacitive touch sensor to determine if an amount of discharging is sufficient to constitute “

touching”

.

24. A computer-vision based cursor control method comprising:

calculating an amount of discharge from a capacitive touch sensor;

activating an image monitoring component to monitor and detect hand gestures when the discharge satisfies a discharge threshold; and

controlling cursor position with respect to content displayed on a user interface based on detected hand gestures, wherein controlling cursor position comprises;

computing a plurality of flow fields based on images captured by the monitoring component to obtain a corresponding plurality of flow field vectors, wherein the images are a temporally consecutive sequence of image frames captured by the image monitoring component; and

performing at least one of the following;

averaging the plurality flow field vectors to ascertain a cursor position relative to a corresponding hand position;

or

averaging spatial positions of all non-zero flow field vectors to ascertain an absolute cursor position with respect to the corresponding hand position.

25. A vision-based navigation control system comprising:

means for detecting user-based input;

means for determining whether the input satisfies an amount of discharge touch threshold;

means for monitoring user gestures occurring within an image monitoring region when the touch threshold is satisfied; and

means for driving cursor movement based in part on the user hand gestures, wherein driving cursor movement comprises;

computing a plurality of flow fields based on images captured during the monitoring of user gestures to obtain a corresponding plurality of flow field vectors, wherein the images are a temporally consecutive sequence of image frames; and

performing at least one of averaging the plurality of flow field vectors to ascertain a cursor position relative to a corresponding hand position or averaging spatial positions of all non-zero flow field vectors to ascertain an absolute cursor position with respect to the corresponding hand position.

26. The system of claim 25, further comprising means for maintaining a current monitoring state depending on whether the touch threshold is satisfied.

27. The system of claim 25, further comprising means for converting user gestures into relative cursor movement or position, the user gestures comprising one or more hand movements.