SYSTEMS AND METHODS FOR MANIPULATING A VIRTUAL ENVIRONMENT

• US 20170031503A1
• Filed: 07/29/2016
• Published: 02/02/2017
• Est. Priority Date: 09/26/2014
• Status: Active Grant

• Current Assignee: Sensel, Inc.
• Original Assignee: Sensel, Inc.
• Inventors: Watson, Charles Robert, Rosenberg, Ilya Daniel, Zarraga, John Aaron, Grau, Alexander Meagher
• Granted Patent: US 9,864,461 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04102 Flexible digitiser, i.e. constructional details for allowing the whole digitising part of a device to be flexed or rolled like a sheet of paper
• CPC Class Codes: G06F 2203/04104 Multi-touch detection in digitiser, i.e. details about the simultaneous detection of a plurality of touching locations, e.g. multiple fingers or pen and finger
• CPC Class Codes: G06F 2203/04105 Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04144 using an array of force sensing means position sensing using the local deformation of sensor cells G06F3/0447
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/045 using resistive elements, e.g. a single continuous surface or two parallel surfaces put in contact
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

One variation of a method for manipulating virtual objects within a virtual environment includes: receiving a touch image from a handheld device, the touch image comprising representations of discrete inputs into a touch sensor integrated into the handheld device; extracting a first force magnitude of a first input at a first location on a first side of the handheld device from the touch image; extracting a second force magnitude of a second input at a second location on a second side of the handheld device from the touch image, the second side of the handheld device opposite the first side of the handheld device; transforming the first input and the second input into a gesture; assigning a magnitude to the gesture based on the first force magnitude; and manipulating a virtual object within a virtual environment based on a type and the magnitude of the gesture.

Claims

1. A system for manipulating virtual objects within a virtual environment comprising:

a housing;

a first touch sensor area comprising;

a first touch sensor surface extending across a first region of the exterior of the housing;

a first array of sense electrodes patterned across a substrate; and

a first resistive layer interposed between the first touch sensor surface and the first array of sense electrodes, in contact with the first array of sense electrodes, and comprising a material exhibiting variations in local electrical resistance responsive to variations in force applied to the first touch sensor surface;

a second touch sensor area comprising;

a second touch sensor surface extending across a second region of the exterior of the housing opposite the first region;

a second array of sense electrodes; and

a second resistive layer interposed between the second touch sensor surface and the second array of sense electrodes, in contact with the second array of sense electrodes, and comprising the material;

a controller coupled to the housing and configured;

to scan the first array of sense electrodes and the second array of sense electrodes during a scan period;

to detect a first input of a first force magnitude at a first location on the first touch sensor surface based on a change in resistance values of a subset of sense electrodes in the first array of sense electrodes;

to detect a second input of a second force magnitude at a second location on the second touch sensor surface based on a change in resistance values of a subset of sense electrodes in the second array of sense electrodes; and

to generate a touch image defining the first location and the first magnitude of the first input and the second location and the second magnitude of the second input for the scan period.

2. The system of claim 1:

wherein the first array of sense electrodes comprises a set of sense electrode and drive electrode pairs;

wherein the first resistive layer exhibits variations in electrical contact resistance over a subset of sense electrode and drive electrode pairs responsive to application of a force to the first touch sensor surface over the subset of sense electrode and drive electrode pairs; and

wherein the controller detects the first input on the first touch sensor surface based on a change in measured resistance across sense electrode and drive electrodes in the subset of sense electrode and drive electrode pairs and determines a force magnitude of the first input based on a magnitude of the change in measured resistance across sense electrode and drive electrodes in the subset of sense electrode and drive electrode pairs.

3. The system of claim 1, wherein the housing defines a cylindrical section;

and wherein the first touch sensor surface and the second touch sensor surface comprise a contiguous touch sensor surface spanning a cylindrical surface of the housing.

4. The system of claim 1, wherein the first touch sensor surface defines a first planar area on a first side of the housing, and wherein the second touch sensor surface defines a second planar area on a second side of the housing opposite the first side of the housing.

5. The system of claim 1, further comprising a computer system configured:

to receive a sequence of touch images corresponding to a sequence of sampling periods from the controller;

to track the first input and the second input across the sequence of touch images; and

to interpret increases in force magnitudes of the first input and the second input, an upward shift in the first location of the first input, and an upward shift in the second location of the second input across the sequence of touch images as a third input on a unsensed surface of the housing above and substantially perpendicular to the first touch sensor surface and to the second touch sensor surface.

6. The system of claim 5, wherein the computer system is remote from the housing;

and further comprising a transmitter arranged within the housing and configured to transmit the sequence of touch images to the computer system.

7. The system of claim 5, wherein the computer system estimates a third force magnitude of the third input on the unsensed surface of the housing based on the force magnitudes of the first input and the second input, a magnitude of the upward shift in the first location of the first input, and a magnitude of the upward shift in the second location of the second input.

8. The system of claim 1, further comprising a sleeve defining a receptacle configured to transiently receive the housing, defining an external surface beyond the receptacle, and configured to communicate a force on the external surface into the first touch sensor surface when the housing is installed in the receptacle.

9. The system of claim 8, wherein the sleeve defines a handed ergonomic overlay;

and

further comprising a second sleeve defining a second receptacle configured to transiently receive the housing, defining a second external surface beyond the second receptacle, configured to communicate a force on the second external surface into the first touch sensor surface when the housing is installed in the second receptacle, and defining a form of a tool corresponding to a virtual tool within a virtual environment.

10. The system of claim 8, wherein the sleeve comprises:

a rigid insert of a first durometer; and

a flexible substrate of a second durometer less than the first durometer, interposed between the receptacle and the rigid insert, configured to deform in response to application of a force on the rigid insert toward the receptacle, and configured to communicate a force applied to the rigid insert into the first touch sensor surface.

11. system of claim 1, wherein the first touch sensor area comprises the first array of sense electrodes and the second touch sensor area comprises the second array of sense electrodes patterned across the substrate;

and wherein the substrate is assembled into a three-dimensional structure within the housing to face the first touch sensor surface and the second touch sensor surface.

12. The system of claim 1, further comprising a strap extending from the housing and configured to couple the housing to a hand of a user.

13. The system of claim 1, further comprising an optical emitter coupled to the housing and configured to broadcast an optical signal detectable by an external tracking system to determine the position and orientation of the housing within a real space.

14. A method for manipulating virtual objects within a virtual environment comprising:

receiving a touch image from a handheld device, the touch image comprising representations of discrete inputs into a touch sensor integrated into the handheld device;

extracting a first force magnitude of a first input at a first location on a first side of the handheld device from the touch image;

extracting a second force magnitude of a second input at a second location on a second side of the handheld device from the touch image, the second side of the handheld device opposite the first side of the handheld device;

transforming the first input and the second input into a gesture;

assigning a magnitude to the gesture based on the first force magnitude; and

manipulating a virtual object within a virtual environment based on a type and the magnitude of the gesture.

15. The system of claim 14:

further comprising extracting a third force magnitude of a third input at a third location on the second side of the handheld device from the touch image;

wherein transforming the first input and the second input into the gesture comprises;

labeling the first input as an intended input based on the first force magnitude approximates a sum of the second force magnitude and the third force magnitude;

discarding the second input and the third input; and

generating the gesture corresponding to a command assigned to a region of the handheld device comprising the first location; and

wherein manipulating the virtual object within the virtual environment comprises transforming the virtual object according to the command to a degree related to the first force magnitude.

16. The system of claim 14,

wherein extracting the first force magnitude and the first location of the first input and the second force magnitude and the second location of the second input comprise tracking the first input and the second input across a sequence of touch images;

wherein transforming the first input and the second input into a gesture comprises

interpreting increases in force magnitudes of the first input and the second input, an upward shift in the first location of the first input, and an upward shift in the second location of the second input as a third input on a unsensed surface of the handheld device above and substantially perpendicular to the first side and to the second side of the handheld device;

labeling the third input as an intended input on the handheld device;

discarding the first input and the second input; and

generating the gesture corresponding to a command assigned to a region of the handheld device comprising the unsensed surface;

wherein assigning the magnitude to the gesture comprises assigning a third force magnitude to the third input on the unsensed surface of the housing based on the force magnitudes of the first input and the second input, a magnitude of the upward shift in the first location of the first input, and a magnitude of the upward shift in the second location of the second input; and

wherein manipulating the virtual object within the virtual environment comprises selecting the virtual object according to the command in response to the third force magnitude exceeding a threshold force magnitude.

17. The system of claim 14:

wherein transforming the first input and the second input into the gesture comprises generating a grasping gesture based on coaxial proximity of the first location and the second location through the handheld device;

wherein assigning the magnitude to the gesture comprises writing a grasping magnitude related to a sum of the first force magnitude and the second force magnitude to the gesture; and

wherein manipulating the virtual object within the virtual environment comprises coupling a virtual tool to the virtual object according to the grasping magnitude of the gesture.

18. The system of claim 17:

further comprising tracking a position of the handheld device within a real space;

wherein manipulating the virtual object within the virtual environment comprises;

moving the virtual object and the virtual tool within the virtual environment according to a change in the position of the handheld device within the real space while the sum of the first force magnitude of the first input and the second force magnitude of the second input exceeds an upper threshold force magnitude;

decoupling the virtual tool from the virtual object within the virtual environment in response to the sum of the first force magnitude of the first input and the second force magnitude of the second input falling below a lower threshold force magnitude; and

slipping the virtual object relative to the virtual tool within the virtual environment according to a friction model in response to the sum of the first force magnitude of the first input and the second force magnitude of the second input falling within a force magnitude range between the upper threshold force magnitude and the lower threshold force magnitude.

19. The system of claim 14,

wherein extracting the first force magnitude and the first location of the first input and the second force magnitude and the second location of the second input comprise:

identifying a set of discrete input areas in the touch image; and

labeling each discrete input area in the set of discrete input areas as one of a palm, a first digit, a second digit, a third digit, a fourth digit, and a fifth digit of a hand;

wherein transforming the first input and the second input into the gesture comprises generating a gesture defining adduction and abduction positions of each virtual finger of a virtual hand model based on locations and labels of each discrete input area in the set of discrete input areas;

wherein assigning the magnitude to the gesture comprises augmenting the gesture with a magnitude of flexion of each virtual finger of the virtual hand model based on a force magnitude of each discrete input area in the set of discrete input areas; and

wherein manipulating the virtual object within the virtual environment comprises transforming the virtual hand model in the virtual environment according to the adduction and abduction positions and magnitudes of flexion defined in the gesture.

20. The system of claim 14, wherein transforming the first input and the second input into the gesture comprises transforming the first input and the second input into the gesture of a type corresponding to a sleeve overlay transiently installed over the handheld device.

21. The system of claim 14,

further comprising extracting a third force magnitude of a third input at a third location on the second side of the handheld device from the touch image;

wherein transforming the first input and the second input into the gesture comprises generating a bending gesture according to lateral separation between the second location and the third location on the second side of the handheld device spanning the first location on the first side of the handheld device;

wherein assigning the magnitude to the gesture comprises assigning a magnitude of a bending moment to the gesture based on the first force magnitude, the second force magnitude, and the third force magnitude; and

wherein manipulating the virtual object within the virtual environment comprises virtually bending the virtual object according to the bending moment assigned to the gesture.