Web-like hierarchical menu display configuration for a near-eye display

• US 9,563,331 B2
• Filed: 06/28/2013
• Issued: 02/07/2017
• Est. Priority Date: 06/28/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Poulos, Adam G., Ambrus, Anthony J., Brown, Cameron G., Scott, Jason, Mount, Brian J., McCulloch, Daniel J., Bevis, John, Zhang, Wei
• Primary Examiner(s): Kim, Matt
• Assistant Examiner(s): LUU, DAVID V
• Application Number:
  US13/930,379
• Publication Number:
  US 20150007114A1
• Time in Patent Office: 1,320 Days
• Field of Search: 715/852
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

Technology is described for web-like hierarchical menu interface which displays a menu in a web-like hierarchical menu display configuration in a near-eye display (NED). The web-like hierarchical menu display configuration links menu levels and menu items within a menu level with flexible spatial dimensions for menu elements. One or more processors executing the interface select a web-like hierarchical menu display configuration based on the available menu space and user head view direction determined from a 3D mapping of the NED field of view data and stored user head comfort rules. Activation parameters in menu item selection criteria are adjusted to be user specific based on user head motion data tracked based on data from one or more sensors when the user wears the NED. Menu display layout may be triggered by changes in head view direction of the user and available menu space about the user'"'"'s head.

Claims

1. A method for determining a web-like hierarchical display configuration for displaying a menu by a near-eye display (NED) comprising:

identifying by one or more processors data indicating a menu is to be displayed by the near-eye display (NED);

determining an available menu space about a user head and a user head view direction by the one or more processors based on a stored three dimensional (3D) mapping of an environment of the user;

selecting by the one or more processors a position for the menu in a display field of view of the NED based on the user head view direction of a menu activation location;

selecting a web-like hierarchical menu display configuration based on the available menu space, the user head view direction and user head comfort rules, the web-like hierarchical menu display configuration links menu levels and menu items within a menu level with flexible spatial dimensions for menu elements;

adapting menu items to the selected web-like hierarchical menu display configuration based on changes in the available menu space based on objects surrounding the user, the user head view direction and user head comfort rules including head rotation range and head rotation speed as an indicator of a level of head comfort; and

displaying the menu with its menu items connected using a flexible link configurable to be linear and non-linear and is adapted in accordance with the selected web-like hierarchical menu display configuration.

2. The method of claim 1 wherein adapting menu items to the selected web-like hierarchical display configuration based on the available menu space, the user head view direction and user head comfort rules further comprising at least one of:

adapting a size of menu items in at least one menu level displayed simultaneously based on the available menu space; and

adapting a number of menu levels displayed simultaneously based on the available menu space.

3. The method of claim 1 wherein selecting by the one or more processors a position for the menu in a display field of view of the NED based on a user head view direction of a menu activation location further comprises:

selecting the position for the menu in the display field of view based on at least one of a user height or a current user height.

4. The method of claim 1 further comprising:

monitoring head view direction for activating a change in a menu display layout.

5. The method of claim 4 wherein monitoring head view direction for activating a change in a menu display layout activating a change in a menu display layout further comprises:

responsive to monitoring head view direction indicating a user head position is rotating further away from a center head position, decreasing a displayed distance between the menu levels.

6. The method of claim 1 wherein selecting the web-like hierarchical menu display configuration based on the available menu space, the user head view direction and user head comfort rules further comprises:

displaying the web-like hierarchical display configuration with the flexible links for connecting the menu items;

identifying the available menu space has more space in a vertical direction than a horizontal direction with respect to the user head view direction;

selecting the web-like hierarchical menu display configuration including the flexible links for display in which the flexible links connect the menu items within a menu level in a zigzag pattern positioning the menu items back and forth in a first side direction up to a first side width boundary and then in a second side direction up to a second side width boundary within a predetermined vertical height range.

7. The method of claim 4 further comprising:

responsive to a change in the menu display layout being activated due to a user head view direction of the menu item indicating a current user head position does not satisfy user head comfort rules, moving currently unselected menu items in a depth direction away from the NED in a display field of view of the NED and moving display locations of the menu items of a currently user selected menu level to a position in a 3D mapping of the user environment at which the user view direction can be from a center head position.

8. A method for adapting activation of a web-like hierarchical menu interface being executed by one or more processors of a near-eye display (NED) system based on user selection metrics comprising:

adjusting by one or more processors activation parameters in menu item selection criteria for the web-like hierarchical menu interface based on stored user head motion data tracked over time and applying a weighting scheme to later tracked data values of the tracked data;

monitoring head view direction and determining whether a change in the head view direction activates a change in a menu display layout;

determining whether a change in the available menu space activates a change in the menu display layout;

identifying user selection of a menu item and adjusting the menu item selection criteria based on the weighted averages tracked over time for the user head motion data to generate user specific menu item selection criteria; and

displaying feedback confirming user selection of the menu item.

9. The method of claim 8 wherein identifying user selection of a menu item further comprises:

identifying one or more menu items in the head view direction based on a 3D mapping of display field of view of the NED;

detecting one or more user actions of a head based feature while the head view direction stays within a menu selection range;

providing one or more visual aids relating the one or more user actions to a user navigation path within the menu;

determining whether the one or more user actions satisfy the user specific menu item selection criteria for a menu item; and

responsive to the one or more user actions satisfying the user specific menu item selection criteria, outputting feedback by the near-eye display system confirming selection of the menu item.

10. The method of claim 8 wherein the user head motion data comprises a head rotation distance range for an up and down direction, a head rotation distance range for a side to side direction, a speed of head rotation in an up and down direction and a speed of head rotation in a side to side direction.

11. The method of claim 9 wherein the one or more visual aids relating the one or more user actions to a user navigation path within the menu comprise a cursor for navigating the web-like hierarchical menu interface.

12. The method of claim 9 wherein detecting the one or more user actions of a head based feature while the head view direction stays within a menu selection range and determining whether the one or more user actions satisfy the user specific menu item selection criteria for the one or more menu items further comprises:

tracking changes to the user head view direction based on updates to the 3D mapping of the display field of view;

correlating user head view directions to the currently displayed menu items by identifying positions of the menu items in the NED field of view;

determining whether a user head view direction has been maintained with respect to one of the currently displayed menu items satisfying a selection candidate time period stored in user-specific menu item selection criteria;

responsive to the selection candidate time period being satisfied, displaying a visual aid to the user indicating that the one of the currently displayed menu items is a selection candidate; and

based on head orientation data determined from sensor data from an inertial sensor of the NED system, determine whether a selection confirmation head movement indicated by a head position change has occurred within a confirmation time period stored in user-specific menu item selection criteria.

13. The method of claim 8 wherein monitoring head view direction and determining whether a change in the head view direction activates a change in a menu display layout further comprises:

checking view direction of user with respect to menu;

determining whether view direction is within a range for menu selection;

responsive to the view direction not being within the range for menu selection, closing the menu;

responsive to the view direction being within the range for menu selection, determining whether the view direction is within a menu boundary range;

responsive to the view direction being within a menu boundary, adjusting the visual appearance of a cursor to indicate user head view direction is one of moving away from the menu or towards the menu;

determining whether user head position data for the current head view direction satisfies user head comfort rules; and

responsive to the user head comfort rules not being satisfied, repositioning the menu items for satisfying the user head comfort rules.

14. The method of claim 8 wherein determining whether a change in the available menu space activates a change in the menu display layout further comprises:

determining whether the available menu space has changed for the user;

responsive to the available menu space having changed for the user determining whether a minimum display configuration spatial dimensions are met for a current web-like hierarchical display configuration;

responsive to the minimum display configuration spatial dimensions not being met for the current web-like hierarchical display configuration, determining another web-like hierarchical display configuration for the menu and updating the menu with the other web-like hierarchical display configuration;

responsive to the minimum display configuration spatial dimensions being met for the current web-like hierarchical display configuration, determine whether the current spatial dimension requirements are being met for the current web-like hierarchical display configuration; and

responsive to the current spatial dimensions not being met for the current web-like hierarchical display configuration, modifying a layout of menu items for the current web-like hierarchical display configuration based on web-like hierarchical display configuration rules for the current web-like hierarchical display configuration.

15. The method of claim 9 wherein providing one or more visual aids relating the one or more user actions to a user navigation path within the menu further comprises:

the web-like hierarchical display configuration includes flexible links displayed for connecting menu items;

displaying by the near-eye display under control of the one or more processors cursor movement along one of the flexible links indicating progress in selecting one of the menu items with at least one of a head movement or a gaze action.

16. A near-eye display (NED) system for displaying a web-like hierarchical menu display configuration comprising:

a near-eye support structure;

a near-eye display (NED) supported by the near-eye support structure and having a field of view;

an image generator supported by the near-eye support structure to output image data for display by the near-eye display (NED); and

one or more processors having access to a memory storing web-like hierarchical menu display configurations and web-like hierarchical menu display configurations rules, and being communicatively coupled to the image generator to control display of a menu in accordance with one of the web-like hierarchical menu display configurations and user head comfort rules such that the image generator controls the web-like hierarchical menu display configurations to connect menu levels with a flexible link configurable as an arc and a line which are displayed on the NED and navigable using a visual aid to traverse along a link pathway.

17. The near-eye display (NED) system of claim 16 further comprising:

at least one image capture device supported by the near-eye support structure for capturing image data and depth data of real objects in the field of view of the NED;

the one or more processors being communicatively coupled to the at least one image capture device for receiving the image and depth data and determining a three dimensional mapping of the display field of view;

the one or more processors determining a user head view direction of a user when wearing the NED system with respect to a menu position in the display field of view and identifying one or more menu items in the user head view direction;

the one or more processors processing sensor data from one or more sensors supported by the near-eye support for detecting one or more user actions of a head based feature; and

the one or more processors determining which of the one or more menu items in the user head view direction has been selected based on the identified user actions of the head based feature.

18. The near-eye display (NED) system of claim 17 further comprising:

at least one of the user actions of the head based feature is a change in head position; and

the one or more sensors supported by the near-eye support includes an inertial sensor providing orientation data from which head position data is determined.

19. The near-eye display (NED) system of claim 17 further comprising:

the one or more processors identifying available menu space for displaying the menu in the web-like hierarchical menu display configuration based on at least one of the 3D mapping of the display field of view or a 3D mapping of the user environment.

20. The near-eye display (NED) system of claim 18 wherein the user head comfort rules are defined in terms of head position data, and

the stored web-like hierarchical menu display configurations include data identifying size, shape and distance ranges for different display menu elements.