Selecting virtual objects in a three-dimensional space

• US 10,521,025 B2
• Filed: 10/18/2016
• Issued: 12/31/2019
• Est. Priority Date: 10/20/2015
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Hamilton, Frank, Fontaine, Marshal A., Abovitz, Rony, Naples, Alysha
• Primary Examiner(s): Silverman, Seth A
• Application Number:
  US15/296,869
• Publication Number:
  US 20170109936A1
• Time in Patent Office: 1,169 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Systems and methods for interacting with virtual objects in a three-dimensional space using a wearable system are disclosed. The wearable system can be programmed to allow a user to interact with virtual objects using a user input device and poses. The wearable system can also automatically determine contextual information such as layout of the virtual objects in the user'"'"'s environment and switch the user input mode based on the contextual information.

Claims

1. A system for changing a user input mode for a wearable device, the system comprising:

a display system of a wearable device configured to present a three-dimensional (3D) view to a user, the 3D view comprising interactable objects;

a user input device configured to receive a user input;

a sensor configured to acquire data associated with a pose of the user;

a hardware processor in communication with the user input device, the processor programmed to;

determine whether a current user input mode for interacting with the interactable objects is a first user input mode or a second user input mode,

wherein the first user input mode is based at least partly on the pose of the user and

the second user input mode is based at least partly on the user input from the user input device;

determine a field of view (FOV) of the user based at least partly on the pose of the user, the FOV comprising a portion of an environment of the user that is perceived at a given time by the user;

determine contextual information associated with the FOV, the contextual information comprising a density of the interactable objects;

in response to a determination that the current user input mode is the first user input mode;

monitor the pose of the user using the sensor;

present via the display system, based at least partly on the monitored pose, a focus indicator in a first shape associated with the first user input mode in a direction related to the pose of the user;

determine that the density of the interactable objects exceeds a first threshold density;

receive a first indication to switch to the second user input mode based on the determination that the density of the interactable objects exceeds the first threshold density; and

switch the current user input mode to the second user input mode in response the first indication; and

in response to a determination that the current user input mode is the second user input mode;

monitor the user input from the user input device;

present via the display system, based at least partly on the monitored input, the focus indicator in the second shape associated with the second user input mode;

determine that the density of the interactable objects is below a second threshold density;

receive a second indication to switch to the first user input mode based on the determination that the density of the interactable objects is below the second threshold density; and

switch the current user input mode to the first user input mode in response the second indication.

2. The system of claim 1, wherein the pose of the user comprises at least one of the following:

a head pose, an eye pose, a foot pose, or a body pose, or wherein the sensor comprises at least one of;

an inward-facing imaging system, an outward-facing imaging system, or an inertial measurement unit, or wherein the display system comprises a light field display configured to display one or more of the interactable objects at multiple depth planes.

3. The system of claim 1,

wherein the contextual information further comprises at least one of:

a layout of the interactable objects in the FOV, a size of the FOV, a size of one or more of the interactable objects in the user'"'"'s FOV; and

wherein the processor is further programmed to present an option for switching from the first user input mode to the second user input mode or from the second user input mode to the first user input mode based at least partly on the contextual information.

4. The system of claim 3, wherein to present the option, the processor is programmed to present the focus indicator in the second shape when the current user input mode is in the first user input mode, and to present the focus indicator in the first shape when the current user input mode is in the second user input mode.

5. The system of claim 1, wherein the first indication comprises a change in the pose of the user, and wherein the second indication comprises an actuation of the user input device.

6. A method for changing a user input mode for a wearable device, the method comprising:

under control of a wearable device comprising a computer processor, the wearable device configured to permit user interaction with interactable objects in a field of regard (FOR) of a user, the FOR comprising a portion of an environment around the user that is capable of being perceived by the user via a display system of the wearable device;

determining a pose of a user;

displaying, via the display system, a first focus indicator associated with a target interactable object in a direction related to the pose of the user, wherein the target interactable object comprises a plurality of virtual objects;

receiving a selection of the target interactable object;

presenting an option to the user for switching the user input mode from poses to hand gestures on a user input device;

displaying, via the display system, the plurality of virtual objects;

displaying, via the display system, a second focus indicator associated with a target virtual object of the plurality of virtual objects in response to a determination that the user has switched the user input mode from poses to hand gestures on the user input device; and

updating the second focus indicator based at least partly on a user input from the user input device,

wherein the option is presented in response to the selection of the target interactable object or contextual information associated with the plurality of virtual objects or the target interactable object,

wherein the contextual information comprises a density of the plurality of virtual objects, wherein the option for switching the user input mode from poses to hand gestures on a user input device is presented in response to a determination that density of the plurality of virtual object exceeds a threshold density.

7. The method of claim 6, wherein the target virtual object is identified based at least partly on the pose of the user.

8. The method of claim 6, wherein updating the second focus indicator comprises transporting the second focus indicator from the target virtual object to another virtual object of the plurality of virtual objects.

9. The method of claim 6, further comprising initiating a selection event on the target virtual object, the selection event comprising at least one of opening a menu associated with the target virtual object or receiving an indication of selecting the target virtual object.

10. The method of claim 6, wherein the plurality of virtual objects comprises at least one of a weather application or an astronomy application, wherein in response to a selection of the weather application, the wearable device is programmed to display virtual weather information superimposed on an object in the environment of the user, and wherein in response to a selection of the astronomy application, the wearable device is programmed to display an interactive planetary system comprising a three-dimensional virtual planet superimposed on the environment of the user.