SELECTING VIRTUAL OBJECTS IN A THREE-DIMENSIONAL SPACE

• US 20170109936A1
• Filed: 10/18/2016
• Published: 04/20/2017
• Est. Priority Date: 10/20/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Hamilton, Frank, Fontaine, Marshal A., Naples, Alysha, Abovitz, Rony
• Granted Patent: US 10,521,025 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to allow a user to interact with virtual objects using a user input device and poses. The wearable system can also automatically determine contextual information such as layout of the virtual objects in the user'"'"'s environment and switch the user input mode based on the contextual information.

Claims

1. A system for changing a user input mode for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising interactable objects;

a user input device configured to receive a user input;

a sensor configured to acquire data associated with a pose of the user;

a hardware processor in communication with the user input device, the processor programmed to;

determine whether a current user input mode for interacting with the interactable objects is a first user input mode or a second user input mode,

wherein the first user input mode is based at least partly on the pose of the user and

the second user input mode is based at least partly on the user input from the user input device;

in response to a determination that the current user input mode is the first user input mode;

monitor the pose of the user using the sensor;

present via the display system, based at least partly on the monitored pose, a focus indicator in a first shape associated with the first user input mode in a direction related to the pose of the user;

receive a first indication to switch to the second user input mode; and

switch the current user input mode to the second user input mode in response the first indication; and

in response to a determination that the current user input mode is the second user input mode;

monitor the user input from the user input device;

present via the display system, based at least partly on the monitored input, the focus indicator in the second shape associated with the second user input mode;

receive a second indication to switch to the first user input mode; and

switch the current user input mode to the first user input mode in response the second indication.

2. The system of claim 1, wherein the pose of the user comprises at least one of the following:

a head pose, an eye pose, a foot pose, or a body pose, or wherein the sensor comprises at least one of;

an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or wherein the display system comprises a light field display configured to display one or more of the interactable objects at multiple depth planes.

3. The system of claim 1, wherein the processor is further programmed to:

determine a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of an environment of the user that is perceived at a given time by the user;

determine contextual information associated with the FOV wherein the contextual information comprises at least one of;

a layout of the interactable objects in the FOV, a size of the FOV, a size of one or more of the interactable objects in the user'"'"'s FOV; and

present an option for switching from the first user input mode to the second user input mode or from the second user input mode to the first user input mode based at least partly on the contextual information.

4. The system of claim 3, wherein to present the option, the processor is programmed to present the focus indicator in the second shape when the current user input mode is in the first user input mode, and to present the focus indicator in the first shape when the current user input mode is in the second user input mode.

5. The system of claim 1, wherein the first indication comprises a change in the pose of the user, and wherein the second indication comprises an actuation of the user input device.

6. A method for changing a user input mode for a wearable device, the method comprising:

under control of a wearable device comprising a computer processor, the wearable device configured to permit user interaction with interactable objects in a field of regard (FOR) of a user, the FOR comprising a portion of an environment around the user that is capable of being perceived by the user via a display system of the wearable device;

determining a pose of a user;

displaying, via the display system, a first focus indicator associated with a target interactable object in a direction related to the pose of the user, wherein the target interactable object comprises a plurality of virtual objects;

receiving a selection of the target interactable object;

presenting an option to the user for switching the user input mode from poses to hand gestures on a user input device;

displaying, via the display system, the plurality of virtual objects;

displaying, via the display system, a second focus indicator associated with a target virtual object of the plurality of virtual objects in response to a determination that the user has switched the user input mode from poses to hand gestures on the user input device; and

updating the second focus indicator based at least partly on a user input from the user input device.

7. The method of claim 6, wherein the option is presented in response to the selection of the target interactable object or contextual information associated with the plurality of virtual objects or the target interactable object.

8. The method of claim 7, wherein the contextual information comprises a density of the plurality of virtual objects, wherein the option for switching the user input mode from poses to hand gestures on a user input device comprises is presented in response to a determination that density of the plurality of virtual object exceeds a threshold density.

9. The method of claim 6, wherein the target virtual object is identified based at least partly on the pose of the user.

10. The method of claim 6, wherein updating the second focus indicator comprises transporting the second focus indicator from the target virtual object to another virtual object of the plurality of virtual objects.

11. The method of claim 6, further comprising initiating a selection event on the target virtual object, the selection event comprising at least one of opening a menu associated with the target virtual object or receiving an indication of selecting the target virtual object.

12. The method of claim 6, wherein the plurality of virtual objects comprises at least one of a weather application or an astronomy application, wherein in response to a selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on an object in the environment of the user, and wherein in response to a selection of the astronomy application, the wearable device is programmed to display an interactive planetary system comprising a three-dimensional virtual planet superimposed on the environment of the user.

13. A wearable system for selecting a virtual object located in a three-dimensional (3D) space, the wearable system comprising:

a display system configured to present virtual objects in a 3D space;

a non-transitory data store configured to store interactable objects in the 3D space;

a sensor configured to determine a pose of a user; and

a hardware processor programmed to communicate with the display system, the data store, and the sensor to;

determine the pose of the user based at least partly on data received from the sensor;

determine a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of an environment of the user that is perceived at a given time by the user;

identify a group of interactable objects in the FOV;

identify a target interactable object in the FOV based least partly on the pose of the user; and

initiate a selection event associated with the target interactable object.

14. The wearable system of claim 13, wherein the group of interactable objects is stored in a data structure where an index associated with each interactable object is determined based at least partly on a position of the interactable object in the 3D space.

15. The wearable system of claim 13, wherein in response to initiating a selection event on the target interactable object, the processor of the wearable system is programmed to presenting, in a two-dimensional (2D) interface, of virtual objects within a threshold range of the target interactable object in the 3D space.

16. The wearable system of claim 15, wherein the 2D interface is interactable via a user input device.

17. The wearable system of claim 13, wherein to identify the target interactable object in the FOV, the processor is configured to perform at least one of:

determining a path of a gaze of the user based on the pose of the user and select an interactable object which intersects the path of the gaze as the target interactable object;

or

selecting a leftmost or a rightmost interactable object in the user'"'"'s FOV as the target interactable object, wherein the leftmost or the rightmost interactable object is selected based at least partly on indices associated with the group of interactable objects.

18. The wearable system of claim 13, wherein the processor is configured to initiate the selection event in response to at least one of:

receiving an input from a user input device or detecting a change of the pose of the user.

19. The wearable system of claim 13, wherein the processor is further configured to present a focus indicator associated with the target interactable object.

20. The wearable system of claim 13, further comprising a geolocation sensor configured to acquire data associated with a location of the user, and wherein the target interactable object comprises a weather application which is programmed to:

determine the location of the user based on the data acquired by the geolocation sensor;

communicate with a remote computing device to acquire weather data based on the location of the user;

generate a virtual element associated with the weather data; and

superimpose the virtual element in the 3D space of the user.