Augmented reality display device with deep learning sensors

• US 10,733,447 B2
• Filed: 07/18/2019
• Issued: 08/04/2020
• Est. Priority Date: 08/22/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Rabinovich, Andrew, Malisiewicz, Tomasz Jan, DeTone, Daniel
• Primary Examiner(s): Hicks, Charles V
• Application Number:
  US16/515,891
• Publication Number:
  US 20190340435A1
• Time in Patent Office: 383 Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0338 with detection of limited linear or angular displacement of an operating part of the device from a neutral position, e.g. isotonic or isometric joysticks
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 5/01 Dynamic search techniques; Heuristics; Dynamic trees; Branch-and-bound
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification

Abstract

A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).

Claims

1. A system for training a neural network for determining a plurality of different types of events, the system comprising:

non-transitory computer-readable memory storing executable instructions; and

one or more processors programmed by the executable instructions to at least;

receive different types of sensor data,

wherein the sensor data is associated with a plurality of different types of events, and

wherein the different types of sensor data comprise inertial measurement unit data, image data, depth sensor data, sound data, voice data, or any combination thereof;

generate a training set comprising the different types of sensor data as input data and the plurality of different types of events as corresponding target output data; and

train a neural network, for determining a plurality of different types of events, using the training set,

wherein the neural network comprises an input layer for receiving input of the neural network, a plurality of intermediate layers, and a plurality of head components for outputting results of the neural network,

wherein the input layer is connected to a first layer of the plurality intermediate layers,

wherein a head component of the plurality of head components comprises a head output node, and

wherein the head output node is connected to a last layer of the intermediate layers through a plurality of head component layers.

2. The system of claim 1, wherein the plurality of different types of events comprises face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization, or any combination thereof.

3. The system of claim 1, wherein the plurality of intermediate layers comprises a plurality of lower layers and a plurality of middle layers,

wherein the plurality of lower layers is trained to extract lower level features from the different types of sensor data, and

wherein the plurality of middle layers is trained to extract higher level features from the lower level features extracted.

4. The system of claim 1, wherein the plurality of intermediate layers or the plurality of head component layers comprises a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a pooling layer, a concatenation layer, a fully connected layer, a linear fully connected layer, a softsign layer, a recurrent layer, or any combination thereof.

5. The system of claim 1, wherein the one or more processors is further programmed by the executable instructions to at least:

receive a second different type of sensor data,

wherein the second different type of sensor data is associated with a second different type of events, and

wherein the second different type of sensor data comprises inertial measurement unit data, image data, depth sensor data, sound data, voice data, or any combination thereof;

generate a retraining set comprising the second different type of sensor data as input data and the second different type of events as corresponding target output data; and

retrain the neural network, for determining the second different type of events, using the retraining set,

wherein a second head component of the plurality of head components comprises a second head output node for outputting results associated with the second different type of events, and

wherein the head output node is connected to the last layer of the intermediate layers through a plurality of second head component layers.

6. The system of claim 5, wherein to retrain the neural network, the one or more processors are programmed by the executable instructions to at least:

update weights associated with the plurality of second head component layers.

7. The system of claim 5, wherein the neural network is retrained without updating weights associated with the plurality of intermediate layers.

8. The system of claim 1,

wherein the plurality of different types of sensor data is associated with a second different type of events, and

wherein the one or more processors is further programmed by the executable instructions to at least:

generate a retraining set comprising the different types of sensor data as input data and the second different type of events as corresponding target output data; and

retrain the neural network, for determining the second different type of events, using the retraining set.