Extramissive spatial imaging digital eye glass apparatuses, methods and systems for virtual or augmediated vision, manipulation, creation, or interaction with objects, materials, or other entities

• US 9,720,505 B2
• Filed: 01/03/2014
• Issued: 08/01/2017
• Est. Priority Date: 01/03/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Meta Company
• Inventors: Gribetz, Meron, Mann, W. Steve G.
• Primary Examiner(s): Sasinowski, Andrew
• Application Number:
  US14/147,199
• Publication Number:
  US 20140184496A1
• Time in Patent Office: 1,306 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

A sensing and display apparatus, comprising: a first phenomenon interface configured to operatively interface with a first augmediated-reality space, and a second phenomenon interface configured to operatively interface with a second augmediated-reality space, is implemented as an extramissive spatial imaging digital eye glass.

Claims

1. An apparatus, comprising:

at least one input tracking sensor;

at least one head-mounted output display;

at least one computational device communicatively coupled to the at least one input tracking sensor and at least one head mounted output display, wherein the at least one computational device is configured to execute processor-readable program instructions, that cause the at least one computational device to;

provide a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receive data from the at least one input tracking sensor including data corresponding to a depth map,

process the depth map to determine spatial coordinates associated with a user gesture;

determine a relationship between the spatial coordinates associated with the user gesture when compared to a position of an icon within the spatial coordinate system indicates interaction with at least one icon from the plurality of icons; and

update the display of the plurality of icons by the at least one head-mounted output display in response to the determined interaction with the at least one icon, wherein the relationship includes a distance measured between spatial coordinates associated with the user gesture that is greater than a first threshold and less than a second threshold at a first instance in time and is greater than the second threshold at a second instance of time, indicating a specific user interaction or gesture related to at least one icon of plurality of icons.

2. The apparatus of claim 1, wherein the head-mounted display provides the plurality of icons for display in response to the at least one input tracking sensor sensing that the user has performed an icon summoning gesture.

3. The apparatus of claim 2, wherein sensing the icon summoning gesture comprises the at least one sensor sensing that the user has an open hand, and wherein providing the plurality of icons comprises determining that the open hand is maintained for at least a threshold period of time.

4. The apparatus of claim 1, wherein providing the plurality of icons for display includes presenting the plurality of icons via at least one cycle bubble animation.

5. The apparatus of claim 1, wherein providing the plurality of icons for display includes responding to the at least one input tracking sensor sensing that the user has performed a hand motion indicating a hover-over to select an icon.

6. The apparatus of claim 1, wherein the updating the plurality of icons by the at least one head-mounted output display in response to the interaction includes:

displaying at least one animation related to the relationship.

7. The apparatus of claim 1, wherein the at least one computational device is further configured to execute processor-readable program instructions, that cause the at least one computational device to:

navigate a hierarchy of icons including a folder level and an application/file level based on depth map data indicating a user interaction with at least one icon in the hierarchy of icons.

8. The apparatus of claim 1, wherein the at least one computational device is further configured to execute processor-readable program instructions, that cause the at least one computational device to:

present a hierarchy of icons in response to the determined interaction, wherein the hierarchy of icons includes at least one settings level icon.

9. The apparatus of claim 8, wherein presenting the hierarchy of icons in response to the determined interaction further includes presenting at least one settings sublevel icon associated with at least one display parameter.

10. The apparatus of claim 1, wherein processing the depth map to determine spatial coordinates associated with a user gesture includes determining a direction of movement of the spatial coordinates associated with the user gesture within the coordinate system over time;

and updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes changing the displayed position of the at least one icon according to the determined direction of movement of the user gesture.

11. The apparatus of 1, wherein updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes placing at least one of the plurality of icons at a new location corresponding to a new position within the spatial coordinate system.

12. An apparatus, comprising:

at least one input tracking sensor;

at least one head-mounted output display;

at least one computational device communicatively coupled to the at least one input tracking sensor and at least one head mounted output display, wherein the at least one computational device is configured to execute processor-readable program instructions, to cause the at least one computational device to;

provide a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receive data from the at least one input tracking sensor including data corresponding to a depth map,

process the depth map to determine spatial coordinates associated with a user gesture;

determine a relationship between the spatial coordinates associated with the user gesture when compared to a position of an icon within the spatial coordinate system indicates interaction with at least one icon from the plurality of icons; and

update the display of the plurality of icons by the at least one head-mounted output display in response to the determined interaction with the at least one icon, wherein the relationship includes a distance measured between spatial coordinates associated with the user gesture that is greater than a first threshold and less than a second threshold at a first instance in time and is less than the first threshold at a second instance of time, indicating a specific user interaction or gesture related to at least one icon of the plurality of icons.

13. The apparatus of claim 12, wherein updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes a transformation of the at least one icon after the second instance of time.

14. The apparatus of 12, wherein the at least one computational device is further configured to execute processor-readable program instructions, that cause the at least one computational device to:

open or execute a file or an application after the second instance of time.

15. An apparatus comprising:

at least one input tracking sensor;

at least one head-mounted output display;

at least one computational device communicatively coupled to the at least one input tracking sensor and at least one head mounted output display, wherein the at least one computational device is configured to execute processor-readable program instructions, that cause the at least one processing device to;

provide a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receive depth map data from the at least one input tracking sensor including spatial coordinates associated with a user gesture, the spatial coordinates associated with the user gesture indicating selection of at least one icon from the plurality of icons associated with at least one settings parameter;

provide, by the at least one head-mounted display, an interface element comprising a line stretching from a first location corresponding to a position of the at least one icon within the spatial coordinate system to a second location corresponding to a position within the spatial coordinate system associated with at least one user body part as determined from the depth map data;

determine a spatial relation between the position of the at least one icon and the position of the at least one user body part; and

adjust the at least one settings parameter and the interface element based on the determined spatial relation.

16. The apparatus of claim 15, wherein processing the depth map includes determining spatial coordinates corresponding to at least one user appendage and tracking movement of the spatial coordinates corresponding to at least one user appendage within the spatial coordinate system over time, and the updating of the display includes updating the display in response to the selection and the movement of the spatial coordinates corresponding to the at least one user appendage.

17. A method of providing an interface for at least one head-mounted output display, comprising:

providing a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receiving data from at least one input tracking sensor including data corresponding to a depth map,

processing the depth map, by at least one computational device, to determine spatial coordinates associated with a user gesture;

determining, by the at least one computational device, that a relationship between the spatial coordinates associated with the user gesture when compared to a position of an icon within the spatial coordinate system indicates interaction with at least one icon from the plurality of icons; and

updating the display of the plurality of icons by the at least one head-mounted output display in response to the determined interaction with the at least one icon, wherein the relationship includes a distance measured between spatial coordinates associated with the user gesture that is greater than a first threshold and less than a second threshold at a first instance in time and is greater than the second threshold at a second instance of time, indicating a specific user interaction or gesture related to at least one icon of the plurality of icons.

18. The method of claim 17, wherein providing the plurality of icons for display includes providing, by the head-mounted display, the plurality of icons for display in response to the at least one input tracking sensor sensing that the user has performed an icon summoning gesture.

19. The method of claim 18, wherein sensing the icon summoning gesture comprises sensing, by the at least one sensor, that the user has an open hand, and wherein providing the plurality of icons comprises determining that the open hand is maintained for at least a threshold period of time.

20. The method of claim 17, wherein providing the plurality of icons for display includes presenting the plurality of icons via at least one cycle bubble animation.

21. The method of claim 17, wherein providing the plurality of icons for display includes responding to the at least one input tracking sensor sensing that the user has performed a hand motion indicating a hover-over to select an icon.

22. The method of claim 17, wherein the updating the plurality of icons by the at least one head-mounted output display in response to the interaction includes:

displaying at least one animation related to the relationship.

23. The method of claim 17, wherein the method further comprises:

navigating a hierarchy of icons including a folder level and an application/file level based on depth map data indicating a user interaction with at least one icon in the hierarchy of icons.

24. The method of claim 17, wherein the method further comprises:

presenting a hierarchy of icons in response to the determined interaction, wherein the hierarchy of icons includes at least one settings level icon.

25. The method of claim 24, wherein presenting the hierarchy of icons in response to the determined interaction further includes presenting at least one settings sublevel icon associated with at least one display parameter.

26. The method of claim 17, wherein processing the depth map to determine spatial coordinates associated with a user gesture includes determining a direction of movement of the spatial coordinates associated with the user gesture within the coordinate system over time;

and updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes changing the displayed position of the at least one icon according to the determined direction of movement of the user gesture.

27. The method of 17, wherein updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes placing at least one of the plurality of icons at a new location corresponding to a new position within the spatial coordinate system.

28. A method of providing an interface for at least one head-mounted output display, comprising:

providing a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receiving data from at least one input tracking sensor including data corresponding to a depth map,

processing the depth map, by at least one computational device, to determine spatial coordinates associated with a user gesture;

determining, by the at least one computational device, that a relationship between the spatial coordinates associated with the user gesture when compared to a position of an icon within the spatial coordinate system indicates interaction with at least one icon from the plurality of icons; and

updating the display of the plurality of icons by the at least one head-mounted output display in response to the determined interaction with the at least one icon, wherein the relationship includes a distance measured between spatial coordinates associated with the user gesture that is greater than a first threshold and less than a second threshold at a first instance in time and is less than the first threshold at a second instance of time, indicating a specific user interaction or gesture related to at least one icon of the plurality of icons.

29. The method of claim 28, wherein updating the display of the plurality of icons by the at least one head-mounted output display in response to the selection includes a transformation of the at least one icon after the second instance of time.

30. The method of 28 further comprises:

opening or executing of a file or an application after the second instance of time.

31. A method of providing an interface for at least one head-mounted output display, comprising:

providing a plurality of icons for display by the at least one head-mounted output display, the plurality of icons associated with positions in a spatial coordinate system and configured for selection;

receiving depth map data from the at least one input tracking sensor including spatial coordinates associated with a user gesture, the spatial coordinates associated with the user gesture indicating selection of at least one icon from the plurality of icons associated with at least one settings parameter;

providing, by the at least one head-mounted display, an interface element comprising a line stretching from a first location corresponding to a position of the at least one icon within the spatial coordinate system to a second location corresponding to a position within the spatial coordinate system associated with at least one user body part as determined from the depth map data;

determining, by at least one computational device, a spatial relation between the position of the at least one icon and the position of the at least one user body part; and

adjusting, by the at least one computational device, the at least one settings parameter and the interface element based on the determined spatial relation.

32. The method of claim 31, wherein processing the depth map to determine spatial coordinates associated with a user gesture includes determining spatial coordinates corresponding to at least one user appendage and tracking movement of the spatial coordinates corresponding to at least one user appendage within the spatial coordinate system over time, and the updating of the display the spatial coordinates corresponding to the at least one user appendage.