Virtual hand based on combined data

• US 10,048,779 B2
• Filed: 06/30/2012
• Issued: 08/14/2018
• Est. Priority Date: 06/30/2012
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Thomas, Fred, Baum, Robert Adam, Hanes, David H, Main, John Michael
• Primary Examiner(s): Rayan, Mihir K
• Application Number:
  US14/391,436
• Publication Number:
  US 20150084866A1
• Time in Patent Office: 2,236 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

An example computing system may include a display and an input device. The input device may include a touch sensor to provide touch data, and a contactless sensor to provide contactless data. A field of view of the contactless sensor is directed away from the touch sensor. A controller is to combine the touch data and the contactless data into combined data to generate a virtual hand to be displayed on the display. The virtual hand is to include an unsensed feature.

Claims

1. A computing system comprising:

a display;

an input device including at least one touch sensor to provide touch data, and at least one contactless sensor to provide contactless data, wherein a field of view of the at least one contactless sensor is directed away from the at least one touch sensor; and

a controller to combine the touch data and the contactless data into combined data to generate, based on the combined data, a virtual hand to be displayed on the display including an unsensed feature generated for the virtual hand,

wherein the controller is to generate the unsensed feature of the virtual hand to be displayed on the display from an overhead view relative to the display, wherein the unsensed feature is generated based on the combined data sensed from a non-overhead view relative to the at least one touch sensor, and

wherein the controller is to sense a reposition gesture during a sense session, and initiate a reposition phase of the sense session associated with maintaining an absolute position of the virtual hand on the display, based on the combined data indicating a sensed hand being repositioned relative to the input device during the reposition phase.

2. The computing system of claim 1, wherein the controller is to obtain 3D information for the virtual hand from the at least one contactless sensor.

3. The computing system of claim 1, wherein the at least one touch sensor includes a touch responsive surface smaller than the display, and the controller is to provide 1:

1 absolute mapping between displayed movement of the virtual hand and sensed movement as sensed by the input device.

4. The computing system of claim 1, wherein the at least one contactless sensor is active, including at least one sensor emitter and at least one sensor receiver.

5. The computer system of claim 1, the controller to weight differently the data from the at least one contactless sensor and the at least one touch sensor when generating the virtual hand to be displayed.

6. The computer system of claim 1, wherein the touch sensor has a larger surface area than the display.

7. The computer system of claim 1, the controller to transition between different modalities based on input to the touch sensor.

8. The computer system of claim 1, wherein the at least one contactless is disposed at, and looks outward from, a surface of the at least one touch sensor.

9. The computing system of claim 1, wherein the unsensed feature is a back-side feature of the virtual hand.

10. A method comprising:

receiving touch data from at least one touch sensor of an input device, and contactless data from at least one contactless sensor of the input device, wherein a field of view of the at least one contactless sensor is directed away from the at least one touch sensor;

generating a virtual hand based on combined data including the touch data and the contactless data;

displaying the virtual hand including an unsensed feature generated for the virtual hand, the unsensed feature being unsensed by both the touch sensor and the contactless sensor; and

sensing a reposition gesture during a sense session, and initiating a reposition phase of the sense session associated with maintaining an absolute position of the virtual hand on the display, based on the combined data indicating a sensed hand being repositioned relative to the input device during the reposition phase.

11. The method of claim 10, further comprising displaying the virtual hand, in response to initiating a sense session, at an initial position on the display, wherein the initial position is based on scaling a sensed position, as indicated by the combined data, relative to the input device.

12. The method of claim 10, further comprising updating a position of the virtual hand, during a sense session, based on a non-scaled absolute 1:

1 translation of movement as indicated by the combined data sensed by the input device.

13. The method of claim 10, further comprising terminating a session and removing the virtual hand from the display, based on at least one of i) a duration of non-change in the combined data, and ii) the combined data indicating removal of a sensed hand from an interaction space of the input device.

14. The method of claim 10, further comprising monitoring the combined data for a visual gesture to transition a state of a session, including at least one of:

i) an object entering an interaction space of the input device, ii) an object leaving the interaction space, and iii) an object remaining substantially motionless within the interaction space for a duration of time.

15. The method of claim 10, further comprising adjusting the size and color of the virtual hand to match corresponding combined data as sensed by the input device.

16. The method of claim 10, further comprising deriving orientation and position of virtual fingers for the virtual hand based on the combined data.

17. A non-transitory machine-readable storage medium encoded with instructions executable by a computing device that, when executed, cause the computing device to:

receive touch data from at least one touch sensor of an input device, and contactless data from at least one contactless sensor of the input device, wherein a field of view of the at least one contactless sensor is directed away from the at least one touch sensor;

generate a virtual hand based on combined data including the touch data and the contactless data including weighting differently the data from the at least one contactless sensor and the at least one touch sensor when generating the virtual hand to be displayed; and

display the virtual hand including an unsensed feature generated for the virtual hand, the unsensed feature being unsensed by both the touch sensor and the contactless sensor.

18. The storage medium of claim 17, further comprising instructions that cause the computing device to display the virtual hand translucently overlaid on underlying displayed elements.

19. A computing system comprising:

a display;

an input device including at least one touch sensor to provide touch data, and at least one contactless sensor to provide contactless data, wherein a field of view of the at least one contactless sensor is directed away from the at least one touch sensor; and

a controller to combine the touch data and the contactless data into combined data to generate, based on the combined data, a virtual hand to be displayed on the display, the controller is to weight differently the touch data and the contactless data when generating the virtual hand,

wherein the at least one contactless sensor is disposed at and completely surrounded by the at least one touch sensor.

20. A computing system comprising:

a display;

an input device including at least one touch sensor to provide touch data, and at least one contactless sensor to provide contactless data, wherein a field of view of the at least one contactless sensor is directed away from the at least one touch sensor;

a controller to combine the touch data and the contactless data into combined data to generate, based on the combined data, a virtual hand to be displayed on the display; and

an emitter to emit structured illumination, having a non-uniform pattern, toward the at least one touch sensor, the contactless sensor to use the structured illumination to sense in the field of view.