Augmented reality display device with deep learning sensors

• US 10,402,649 B2
• Filed: 08/22/2017
• Issued: 09/03/2019
• Est. Priority Date: 08/22/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Rabinovich, Andrew, Malisiewicz, Tomasz Jan, DeTone, Daniel
• Primary Examiner(s): Hicks, Charles V
• Application Number:
  US15/683,664
• Publication Number:
  US 20180053056A1
• Time in Patent Office: 742 Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification

Abstract

A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Claims

1. A head mounted display system comprising:

a plurality of sensors for capturing different types of sensor data, each of the plurality of sensors disposed on a frame of the head mounted display system, the frame configured to be worn on the head of a user and to position a display system in front of the eyes of the user, the plurality of sensors comprising an outward-facing camera configured to obtain face images;

non-transitory memory configured to store

executable instructions, and

a deep neural network for performing face recognition and lighting detection using the sensor data captured by the plurality of sensors,

wherein the deep neural network comprises an input layer for receiving input of the deep neural network, a plurality of lower layers, a plurality of middle layers, and a plurality of head components for outputting results of the deep neural network associated with the face recognition and the lighting detection,

wherein the input layer is connected to a first layer of the plurality lower layers,

wherein a last layer of the plurality of lower layers is connected to a first layer of the middle layers,

wherein a head component of the plurality of head components comprises a head output node, and

wherein the head output node is connected to a last layer of the middle layers through a plurality of head component layers representing a unique pathway from the plurality of middle layers to the head component;

a display configured to display information related to the face recognition and the lighting detection; and

a hardware processor in communication with the plurality of sensors, the non-transitory memory, and the display, the hardware processor programmed by the executable instructions to;

receive the different types of sensor data from the plurality of sensors;

determine the results of the deep neural network using the different types of sensor data; and

cause display of the information related to the face recognition and the lighting detection.

2. The system of claim 1, wherein the plurality of sensors comprises an inertial measurement unit, a depth sensing camera, a microphone, an eye imaging camera, or any combination thereof.

3. The system of claim 1, wherein the plurality of lower layers is trained to extract lower level features from the different types of sensor data.

4. The system of claim 3, wherein the plurality of middle layers is trained to extract higher level features from the lower level features extracted.

5. The system of claim 3, the head component uses a subset of the higher level features to determine the face recognition or the lighting detection.

6. The system of claim 1, the head component is connected to a subset of the plurality of middle layers through the plurality of head component layers.

7. The system of claim 1,

wherein a number of weights associated with the plurality of lower layers is more than 50% of weights associated with the deep neural network, and

wherein a sum of a number of weights associated with the plurality of middle layers and a number of weights associated with the plurality of head components is less than 50% of the weights associated with the deep neural network.

8. The system of claim 1,

wherein a computation associated with the plurality of lower layers is more than 50% of a total computation associated with the deep neural network, and

wherein a computation associated with the plurality of middle layers and the plurality of head components is less than 50% of the total computation associated with the deep neural network.

9. The system of claim 1, wherein the plurality of lower layers, the plurality of middle layers, or the plurality of head component layers comprises a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a concatenation layer, a fully connected layer, a linear fully connected layer, a softsign layer, a recurrent layer, or any combination thereof.

10. The system of claim 1, wherein the plurality of middle layers or the plurality of head component layers comprises a pooling layer.