KEYBOARDS FOR VIRTUAL, AUGMENTED, AND MIXED REALITY DISPLAY SYSTEMS

• US 20180350150A1
• Filed: 05/18/2018
• Published: 12/06/2018
• Est. Priority Date: 05/19/2017
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Powderly, James M., Niles, Savannah, Awad, Haney, Wheeler, William, Choi, Nari, Stutts, Timothy Michael, Anon, Josh, Sommers, Jeffrey Scott
• Granted Patent: US 11,610,371 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0213 Arrangements providing an integrated pointing device in a keyboard, e.g. trackball, mini-joystick for pointing devices integrated in a laptop computer G06F1/169; joysticks G05G9/047; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 3/0219 Special purpose keyboards
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/0238 Programmable keyboards key guide holders G06F3/0224
• CPC Class Codes: G06F 3/0426 tracking fingers with respect to a virtual keyboard projected or printed on the surface virtual keyboards on touch screens G06F3/04886
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

User interfaces for virtual reality, augmented reality, and mixed reality display systems are disclosed. The user interfaces may be virtual or physical keyboards. Techniques are described for displaying, configuring, and/or interacting with the user interfaces.

Claims

1. A system comprising:

a display system configured to present virtual content in a physical environment of a user;

an outward-facing imaging system configured to image the physical environment of the user;

a hardware processor in communication with the display system and the outward-facing imaging system, the hardware processor programmed to;

analyze an image of the physical environment acquired by the outward-facing imaging system;

recognize a physical keyboard having a plurality of keys based on an analysis of the image;

determine contextual information associated with the physical keyboard;

determine a specification for the physical keyboard based at least in part on the contextual information;

dynamically configure functions of at least a portion of the physical keyboard based at least in part on the specification;

determine a rendering location of a virtual key label based at least in part on the specification; and

instruct the display system to render the virtual key label at the determined rendering location.

2. The system of claim 1, wherein the virtual content comprises at least one of:

augmented or mixed reality content.

3. The system of claim 1, wherein to analyze the image, the hardware processor is programmed to execute an object recognizer to identify a physical keyboard based on visual characteristics of the physical keyboard.

4. The system of claim 3, wherein the visual characteristics comprise at least one of:

a shape of a surface of the physical keyboard or a label of the physical keyboard.

5. The system of claim 1, wherein the outward-facing imaging system comprises a world camera having a FOV, wherein the FOV comprises a portion of the physical environment that is observed by the world camera at a given time, and wherein to recognize a physical keyboard, the hardware processor is programmed to detect at least a portion of the physical keyboard is in the FOV.

6. The system of claim 1, wherein the specification comprises at least one of a layout of the plurality of keys or functions of the plurality of keys.

7. The system of claim 1, wherein to determine the specification for the physical keyboard, the hardware processor is programmed to:

determine a model of the physical keyboard based on the analysis of the image; and

access a database storing the specification of the model to retrieve the specification.

8. The system of claim 1, wherein at least one key of the plurality of keys has a capacitive touch sensor.

9. The system of claim 1, wherein the contextual information comprises at least one of a position of the physical keyboard or an orientation of the physical keyboard.

10. The system of claim 1, wherein the portion of the physical keyboard comprises blank keys and to dynamically configure the functions of at least the portion of the physical keyboard, the hardware processor is programmed to:

assign command functions to the blank keys wherein the command functions cause the system to perform a user interface operation.

11. The system of claim 10, wherein the rendering location of a virtual key label coincides with a location of a blank key in the portion of the physical keyboard, and wherein the hardware processor is configured to instruct the display system to superimpose the virtual key label on the blank key using a pixel stick.

12. A method comprising:

identifying a portion of a keyboard having a plurality of keys in a three-dimensional (3D) physical environment of a user;

determining a specification for the portion of keyboard based at least in part on contextual information;

dynamically configuring functions of the portion of the keyboard based at least in part on the specification;

determining a rendering location of a virtual key label in the portion of the keyboard based at least in part on the specification; and

instructing the display system to render the virtual key label at the determined rendering location.

13. The method of claim 12, wherein the keyboard comprises a physical keyboard.

14. The method of claim 13, wherein the physical keyboard is assembled from a plurality of detachable sections.

15. The method of claim 13, wherein the physical keyboard is associated with a virtual screen in the 3D physical environment of the user and the virtual screen is configured to perform a user interface function in response to an actuation of the physical keyboard.

16. The method of claim 12, wherein the keyboard comprises a virtual keyboard, wherein the virtual keyboard is rendered on a surface of a user'"'"'s environment.

17. The method of claim 16, wherein the surface comprises a surface of a physical keyboard.

18. The method of claim 12, wherein identifying a keyboard comprises at least one of communicating with the keyboard to obtain identifying information of the keyboard, selecting the keyboard among a plurality of keyboards based on the contextual information, recognizing the keyboard based on an image of the keyboard.

19. The method of claim 18, wherein the image is acquired by at least one of:

an outward-facing imaging system of a head-mounted device, a room camera, or a camera coupled to a physical keyboard.

20. The method of claim 12, wherein the portion of the keyboard comprises blank keys.

21. The method of claim 12, wherein the contextual information is associated with at least one of:

the keyboard, the user, or the 3D physical environment.

22. A computer-readable medium with instructions which, when read by a hardware processor, cause the hardware processor to perform a method comprising:

identifying a portion of a keyboard having a plurality of keys in a three-dimensional (3D) physical environment of a user;

determining a specification for the portion of keyboard based at least in part on contextual information;

dynamically configuring functions of the portion of the keyboard based at least in part on the specification;

determining a rendering location of a virtual key label in the portion of the keyboard based at least in part on the specification; and

instructing the display system to render the virtual key label at the determined rendering location.

23-38. -38. (canceled)