TWO-HANDED OBJECT MANIPULATIONS IN VIRTUAL REALITY

• US 20170329488A1
• Filed: 05/10/2016
• Published: 11/16/2017
• Est. Priority Date: 05/10/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Welker, Stefan, Clement, Manuel Christian
• Granted Patent: US 10,019,131 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/0382 Plural input, i.e. interface arrangements in which a plurality of input device of the same type are in communication with a PC
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

Systems and methods are described for generating a virtual environment including at least one three-dimensional virtual object within a user interface provided in a head mounted display device, detecting a first interaction pattern and a second interaction pattern. In response to detecting the second interaction pattern, a modified version of the three-dimensional virtual object at the first virtual feature is generated according to the first interaction pattern and at the second virtual feature according to the second interaction pattern. The modified version of the three-dimensional virtual object is provided in the user interface in the head mounted display device.

Claims

1. A computer-implemented method comprising:

generating a virtual environment including at least one three-dimensional virtual object within a user interface provided in a head mounted display device;

detecting a first interaction pattern, the first interaction pattern including an indication to modify a first virtual feature associated with the three-dimensional virtual object;

detecting a second interaction pattern, the second interaction pattern including an orientation and movement corresponding to a modification to be performed on a second virtual feature associated with the three-dimensional virtual object;

in response to detecting the second interaction pattern, generating a modified version of the three-dimensional virtual object at the first virtual feature according to the first interaction pattern and at the second virtual feature according to the second interaction pattern; and

providing, in the user interface in the head mounted display device, the modified version of the three-dimensional virtual object.

2. The method of claim 1, wherein the modified version of the three-dimensional virtual object is generated and rendered in the virtual environment while the movement occurs.

3. The method of claim 1, wherein the first interaction pattern defines a location in which to begin modifying the first virtual feature, the second interaction pattern defines a direction and orientation away from the second virtual feature, and the modified version of the three-dimensional virtual object includes a scaled version of the three-dimensional virtual object, the scaled version being scaled from the first virtual feature toward the second virtual feature.

4. The method of claim 1, wherein the modified version of the three-dimensional virtual object is adjusted from the first virtual feature to a new position, orientation, and scale of the three-dimensional virtual object and adjusted from the second virtual feature to a new position, orientation, and scale of the three-dimension virtual object, the modified version being within a user field of view in the virtual environment.

5. The method of claim 1, further comprising, enabling a uniform scaling mode in response to detecting that the first interaction pattern is performed in the same axis of the three dimensions as the second interaction pattern, wherein the uniform scaling mode is configured to scale the three-dimensional virtual object evenly in three dimensions according to the first and second interaction patterns.

6. The method of claim 1, wherein generating a modified version of the three-dimensional virtual object at the first virtual feature according to the first interaction pattern and at the second virtual feature according to the second interaction pattern includes stretching and twisting the three-dimensional virtual object, the stretching including defining a first anchor point and a second anchor point on the three-dimensional virtual object and stretching in a plane formed between the first anchor point and the second anchor point.

7. The method of claim 6, wherein twisting the three-dimensional virtual object includes rotating the three-dimensional virtual object in 3D space at the first virtual feature in a first input plane and rotating the three-dimensional virtual object includes rotating the three-dimensional virtual object in 3D space at the second virtual feature from a second input plane to a third input second plane, wherein the first input plane corresponds to an X-Y coordinate plane, the second input plane corresponds to a Y-Z coordinate plane, and the third input plane corresponds to an X-Z coordinate plane, each plane being of a three-dimensional coordinate space.

8. The method of claim 1, wherein detecting the first interaction pattern includes detecting a first input associated with a first tracked device and detecting the second interaction pattern includes detecting a second input associated with a second tracked device.

9. The method of claim 1, further comprising:

generating a first intersection point at the first virtual feature based at least in part on detecting a first tracked hand movement performed by a left hand of the user accessing the virtual environment;

generating a second intersection point at the second virtual feature, based at least in part on detecting a second tracked hand movement performed by a right hand of the user accessing the virtual environment; and

modifying the three-dimensional virtual object according to the first and second interaction patterns, the modifying being based at least in part on determining that at least one gesture performed in the first or second tracked hand movement is a change in orientation associated with the three-dimensional virtual object, and performing the at least one gesture while modifying the three-dimensional object according to another gesture.

10. The method of claim 9, further comprising:

generating an anchor point configured to freeze, in three-dimensional space, a portion of the three-dimensional virtual object, at the first intersection point while modifying the three-dimensional object at the second virtual feature according to the second interaction pattern.

11. The method of claim 10, wherein generating an anchor point comprises defining at least one location on the three-dimensional virtual object as a selectable scaling location, the selectable scaling location being operable to receive a user-initiated input, the user-initiated input including at least one gesture indicating instructions for scaling the three-dimensional virtual object.

12. A computer-implemented method comprising:

detecting, in a three-dimensional virtual environment, a first interaction pattern performed by a first electronic device, the first interaction pattern including an indication to modify a first virtual feature associated with a three-dimensional virtual object rendered in the virtual environment;

detecting, in the three-dimensional virtual environment, a second interaction pattern performed by a second electronic device, the second interaction pattern including an orientation and movement corresponding to a modification to be performed on a second virtual feature associated with the three-dimensional virtual object;

in response to detecting the second interaction pattern, generating a modified version of the three-dimensional virtual object at the first virtual feature according to the first interaction pattern and at the second virtual feature according to the second interaction pattern; and

rendering, in a user interface of a head mounted display device, the modified version of the three-dimensional virtual object according to the first interaction pattern and the second interaction pattern.

13. The method of claim 12, further comprising, enabling a uniform scaling mode in response to detecting that the first electronic device is oriented to face the second electronic device, wherein the uniform scaling mode is configured to scale the three-dimensional virtual object evenly in three dimensions according to the first interaction pattern and the second interaction pattern.

14. The method of claim 13, wherein detecting that the first electronic device is oriented to face the second electronic device includes:

detecting an initial position and orientation of the second electronic device relative to the first electronic device;

tracking at least one gesture performed by the second electronic device relative to the first electronic device; and

detecting an alignment of the second electronic device to the first electronic device by comparing the initial position and orientation of the second electronic device relative to the first electronic device with a detected updated position and orientation of the first or the second electronic device, the detected updated position being based on the at least one gesture.

15. The method of claim 12, wherein generating a modified version of the three-dimensional virtual object at the first virtual feature includes providing a visual indication, in the virtual environment and affixed to the three-dimensional virtual object, the visual indication including a plurality of anchor points defining snapping frames configured to select at least one coordinate system in which to translate and rotate the virtual object in three dimensions.

16. The method of claim 12, further comprising, in response to detecting the second interaction pattern, defining a rotation path to rotate the three-dimensional virtual object around a location associated with the first virtual feature.

17. A system, comprising:

at least one computing device configured to generate a virtual environment, the at least one computing device including,

a memory storing executable instructions; and

a processor configured to execute the instructions, to cause the at least one computing device to;

generate a virtual environment including at least one three-dimensional virtual object within a user interface provided in a head mounted display device;

detect a first interaction pattern, the first interaction pattern including an indication to modify a first virtual feature associated with the three-dimensional virtual object;

detect a second interaction pattern, the second interaction pattern including an orientation and movement corresponding to a modification to be performed on a second virtual feature associated with the three-dimensional virtual object;

in response to detecting the second interaction pattern, generating a modified version of the three-dimensional virtual object at the first virtual feature according to the first interaction pattern and at the second virtual feature according to the second interaction pattern; and

provide, in the user interface in the head mounted display device, the modified version of the three-dimensional virtual object.

18. The system of claim 17, wherein the modified version of the three-dimensional virtual object is generated and rendered in the virtual environment while the movement occurs.

19. The system of claim 17, wherein the first interaction pattern defines a location in which to begin modifying the first virtual feature, the second interaction pattern defines a direction and orientation away from the second virtual feature, and the modified version of the three-dimensional virtual object includes a scaled version of the three-dimensional virtual object, the scaled version being scaled from the first virtual feature toward the second virtual feature.

20. The system of claim 17, wherein the modified version of the three-dimensional virtual object is adjusted from the first virtual feature to a new position, orientation, and scale of the three-dimensional virtual object and adjusted from the second virtual feature to a new position, orientation, and scale of the three-dimension virtual object, the modified version being within a user field of view in the virtual environment.

21. The system of claim 17, further comprising, enabling a uniform scaling mode in response to detecting that the first interaction pattern is performed in a shared axis plane with the second interaction pattern, wherein the uniform scaling mode is configured to scale the three-dimensional virtual object evenly in three dimensions according to the first and second interaction patterns.

22. A method, comprising:

obtaining, with one or more optical sensors of a computing device, a plurality of characteristics about a virtual object;

generating, by a processor of the computing device, a three-dimensional virtual model of the virtual object based on the plurality of characteristics;

processing, by the processor, the plurality of characteristics and the three-dimensional virtual model to define a plurality of anchor points in the three-dimensional virtual model, the plurality of anchor points being respectively associated with a plurality of selectable regions on the virtual object;

in response to receiving an input selecting at least two of the plurality of regions, correlating the input for each region in the plurality of regions;

modifying a size and orientation of the virtual object, based on the correlation and on the plurality of characteristics associated with the virtual object; and

rendering and displaying a modified virtual object in the three-dimensional virtual model.

23. The method of claim 22, wherein the plurality of characteristics associated with the virtual object include one or more of a planarity, a dimension, an area, an orientation, a corner, a boundary, a contour or a surface texture for the virtual object.

24. The method of claim 22, further comprising generating a virtual reality environment including the three dimensional virtual model of the of the virtual object in the virtual reality environment;

and

automatically scaling the virtual object while rotating the virtual object in three dimensions in the generated virtual reality environment based on a two-hand interaction performed by a user and detected plurality of characteristics associated with the virtual object.