Systems and methods using computer vision and capacitive sensing for cursor control

• US 20060033701A1
• Filed: 08/10/2004
• Published: 02/16/2006
• Est. Priority Date: 08/10/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D.
• Granted Patent: US 7,692,627 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 1/1616 with folding flat displays, e.g. laptop computers or notebooks having a clamshell configuration, with body parts pivoting to an open position around an axis parallel to the plane they define in closed position
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Disclosed is a unique system and method that facilitates cursor control based in part on computer vision activated by a capacitive touch sensor. When turned on, user hand gestures or movements can be tracked by a monitoring component and those movements can be converted in real-time to control or drive cursor movements and/or position on a user interface. The system comprises a monitoring component or camera that can be activated by touch or pressure applied to a capacitive touch sensor. A circuit within the sensor determines when the user is touching a button (e.g., on keyboard or mouse) that activates the monitoring component and cursor control mechanism. Thus, intentional hand movements by the user can readily be determined.

Claims

1. A vision-based navigation control system comprising:

an input detection component that detects user input;

a monitoring component that receives data from the input detection component which causes the monitoring component to turn on or off; and

a navigational driving component that receives image data from the monitoring component to facilitate determining where to position or drive a cursor.

2. The system of claim 1, the input comprises at least one of a touch or pressure applied to the input component by a user.

3. The system of claim 1, the input detection component comprises a capacitive touch sensor that computes an electrical reading based in part on the user-based input.

4. The system of claim 1 further comprises software installed on a host machine that can read a status of the input detection component to determine whether an amount of user-based input satisfies a threshold.

5. The system of claim 1, the input detection component comprises an electrode that can be placed on a surface of an input component.

6. The system of claim 1, further comprising an input component including any one of:

at least one button associated with a touchpad or mouse, and one or more keys of a keyboard.

7. The system of claim 1, the user-based input is converted to an electrical signal that is sensed or read by the detection component.

8. The system of claim 1, the monitoring component comprises one or more cameras that can capture an image in motion frame by frame.

9. The system of claim 8 further comprising a component that computes flow fields for substantially every frame of an image captured by the monitoring component in real-time.

10. The system of claim 9, the flow fields are generated to facilitate mapping the user-based input to coincide with driving or controlling a cursor on a display screen to navigate through content.

11. The system of claim 9, the flow field generation component recognizes one or more gestures made by a user to determine where to place a cursor in real-time.

12. The system of claim 9, wherein the flow fields are generated by way of block matching to determine vectors that identify one or more directions of movement between at least any two frames of the image.

13. The system of claim 1, the monitoring component is positioned to monitor the user'"'"'s “

visible”

movement within a viewable space as determined in part by the dimensional limits of the monitoring component as well as in part by user preferences.

28. A computer readable medium having stored thereon the computer executable components of claim 1.

14. A vision-based method that facilitates cursor control comprising:

detecting user-based input via a capacitive touch sensor;

determining whether the input satisfies a touch threshold;

monitoring user gestures occurring within a monitoring region when the touch threshold is satisfied; and

driving cursor movement or position based in part on the user gestures.

15. The method of claim 14, further comprising maintaining a current monitoring state depending on whether the touch threshold is satisfied.

16. The method of claim 14, the current monitoring state comprises one of:

on or off.

17. The method of claim 14, wherein detecting user-based input and determining whether the touch threshold is satisfied on a continuous basis to ascertain when to monitor user gesture activity in the monitoring region.

18. The method of claim 14, the monitoring region comprises an area that is viewable by a monitoring component.

19. The method of claim 18, the monitoring component comprises a camera interfaced to a computing device.

20. The method of claim 14, further comprising converting user gestures into relative cursor movement or position

21. The method of claim 14, wherein the user gestures comprises one or more hand movements.

22. The method of claim 14, wherein determining whether a touch threshold is satisfied comprises analyzing charging and discharging of a capacitive touch sensor to determine if an amount of discharging is sufficient to constitute “

touching”

.

23. A computer-vision based cursor control method comprising:

calculating an amount of discharge from a capacitive touch sensor;

activating a monitoring component to monitor and detect hand gestures when the discharge satisfies a discharge threshold; and

controlling cursor position with respect to content displayed on a user interface based on detected hand gestures.

24. The method of claim 23, wherein controlling cursor position comprises:

computing a plurality of flow fields based on images captured by the monitoring component to obtain corresponding flow field vectors; and

performing at least one of the following;

averaging the flow field vectors to ascertain a cursor position relative to a corresponding hand position; and

averaging spatial positions of substantially all non-zero flow field vectors to ascertain an absolute cursor position with respect to the corresponding hand position.

25. A vision-based navigation control system comprising:

means for detecting user-based input;

means for determining whether the input satisfies a touch threshold;

means for monitoring user gestures occurring within a monitoring region when the touch threshold is satisfied; and

means for driving cursor movement based in part on the user gestures.

26. The system of claim 25, further comprising means for maintaining a current monitoring state depending on whether the touch threshold is satisfied.

27. The system of claim 25, further comprising means for converting user gestures into relative cursor movement or position, the user gestures comprising one or more hand movements.

29. A data packet adapted to be transmitted between two or more computer processes facilitating cursor control based in part on computer vision, the data packet comprising:

information associated with sensing pressure applied to a touch sensor;

determining whether the pressure satisfies a touch threshold;

monitoring user hand gestures; and

driving a cursor position based on the hand gestures.