SYSTEM AND METHOD FOR FACILITATING VIRTUAL INTERACTIONS WITH A THREE-DIMENSIONAL VIRTUAL ENVIRONMENT IN RESPONSE TO SENSOR INPUT INTO A CONTROL DEVICE HAVING SENSORS

• US 20180275781A1
• Filed: 05/25/2018
• Published: 09/27/2018
• Est. Priority Date: 05/18/2016
• Status: Active Grant

• Current Assignee: Meta View, Inc.
• Original Assignee: Meta Company
• Inventors: Gribetz, Yishai
• Granted Patent: US 10,521,028 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04892 Arrangements for controlling cursor position based on codes indicative of cursor displacements from one discrete location to another, e.g. using cursor control keys associated to different directions or using the tab key arrangements for controlling cursor position based on coordinate signals G06F3/038
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/04 involving 3D image data

Abstract

Various implementations provide for a three-dimensional trackpad in which sensors and a three-dimensional physical region may be used to interact with a three-dimensional virtual environment. The methods, systems, techniques, and components described herein may facilitate interactions with virtual objects in a three-dimensional virtual environment in response to sensor input into a control device having one or more sensors implemented thereon. The control device may be coupled to a display that may be configured to display the three-dimensional virtual environment. In various implementations, the sensor(s) capture physical movement of a user interaction element (a hand, a stylus, a physical object, etc.) within a specified three-dimensional physical region. The physical movement may be translated into a virtual interaction with the three-dimensional virtual environment. A virtual action in the three-dimensional virtual environment may be identified and displayed.

Claims

1. A method for facilitating one or more virtual interactions with a three-dimensional virtual environment, the method comprising:

controlling a display system to present a three-dimensional virtual environment over views of a real world such that the three-dimensional virtual environment is perceived to be in a first region in the real world;

receiving a sensor signal from one or more sensors, the sensor signal including a representation of physical movement by a user interaction element within a specified three-dimensional physical region, the specified three-dimensional physical region having a fixed spatial relationship with respect to a portable physical object in the real world such that a location of the specified three-dimensional physical region in the real world can be moved by changing a location in the real world of the portable physical object, the specified three-dimensional physical region forming a second region in the real world that is distinct from the first region in the real world, the physical movement represented by the representation of physical movement including physical movement that is devoid of physical contact with the portable physical object;

translating the representation of the physical movement into a virtual interaction by the user with the three-dimensional virtual environment;

identifying a virtual action in the three-dimensional virtual environment, the virtual action corresponding to the virtual interaction; and

providing instructions to the display system to display the virtual action.

2. The method of claim 1, wherein the user interaction element comprises a hand, a stylus, or an object attached to a user.

3. The method of claim 1, further comprising identifying the specified three-dimensional physical region before receiving the sensor signal from the one or more sensors.

4. The method of claim 1, wherein the specified three-dimensional physical region is proximate to the one or more sensors.

5. The method of claim 1, wherein the physical movement comprises one or more of:

a change in horizontal translation of the user interaction element, a change in vertical translation of the user interaction element, a change in pitch of the user interaction element, a change in roll of the user interaction element, and a change in yaw of the user interaction element.

6. The method of claim 1, wherein the one or more sensors comprise a depth camera configured to capture one or more images of the user interaction element, and to sense a depth of the user interaction element in a direction orthogonal to the portable physical object.

7. The method of claim 1, wherein the portable physical object comprises a mobile phone or a tablet computing device.

8. The method of claim 1, wherein translating the representation of the physical movement comprises magnifying the physical movement in the three-dimensional virtual environment.

9. The method of claim 1, wherein the virtual action comprises a movement of a virtual object across the three-dimensional virtual environment.

10. The method of claim 9, wherein the virtual object comprises a three-dimensional virtual cursor.

11. The method of claim 1, wherein the virtual action comprises a first movement of a virtual object across the three-dimensional virtual environment at a first virtual speed, and a second movement of the virtual object across the three-dimensional virtual environment at a second virtual speed.

12. The method of claim 11, further comprising:

determining whether the user interaction element has exceeded a specified movement threshold; and

modifying an object speed of the virtual object from the first virtual speed to the second virtual speed if the user interaction element has exceeded the specified movement threshold.

13. The method of claim 1, wherein the display system comprises a dedicated headset configured to display the three-dimensional virtual environment.

14. The method of claim 13, wherein the control device is coupled to the display system over a computer network.

15. The method of claim 1, further comprising:

determining whether the user interaction element is outside the specified three-dimensional physical region; and

maintaining a state of the three-dimensional virtual environment if the user interaction element is outside the specified three-dimensional physical region.

16. A system for facilitating one or more virtual interactions with a three-dimensional virtual environment, the system comprising:

the display system configured to present a three-dimensional virtual environment over views of a real world such that the three-dimensional virtual environment is perceived to be in a first region in the real world;

a processing unit coupled to the control device and the display system, the processing unit comprising;

memory;

one or more physical computer processors coupled to the memory, the one or more physical computer processors configured by computer readable instructions stored in the memory to;

receive a sensor signal from one or more sensors, the sensor signal including a representation of physical movement by a user interaction element within a specified three-dimensional physical region, the specified three-dimensional physical region having a fixed spatial relationship with respect to a portable physical object in the real world such that a location of the specified three-dimensional physical region in the real world can be moved by changing a location in the real world of the portable physical object, the specified three-dimensional physical region forming a second region in the real world that is distinct from the first region in the real world, the physical movement represented by the representation of physical movement including physical movement that is devoid of physical contact with the portable physical object;

translate the representation of the physical movement into a virtual interaction by the user with the three-dimensional virtual environment;

identify a virtual action in the three-dimensional virtual environment, the virtual action corresponding to the virtual interaction; and

provide instructions to the display system to display the virtual action.

17. The system of claim 16, wherein the user interaction element comprises a hand, a stylus, or an object attached to a user.

18. The system of claim 16, wherein the method further comprises identifying the specified three-dimensional physical region before receiving the sensor signal from the one or more sensors.

19. The system of claim 16, wherein the specified three-dimensional physical region is proximate to the one or more sensors.

20. The system of claim 16, wherein the physical movement comprises one or more of:

a change in horizontal translation of the user interaction element, a change in vertical translation of the user interaction element, a change in pitch of the user interaction element, a change in roll of the user interaction element, and a change in yaw of the user interaction element.