Threat identification system

• US 9,619,712 B2
• Filed: 05/18/2016
• Issued: 04/11/2017
• Est. Priority Date: 05/18/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew
• Primary Examiner(s): Thompson, James A
• Assistant Examiner(s): Mazumder, Tapas
• Application Number:
  US15/158,498
• Publication Number:
  US 20160342840A1
• Time in Patent Office: 328 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G08B 21/182 Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: G08B 5/36 using visible light sources

Abstract

A head mounted device (HMD) includes a transparent display, sensors to generate sensor data, and a processor. The processor identifies a threat condition based on a threat pattern and the sensor data, and generates a warning notification in response to the identified threat condition. The threat pattern includes preconfigured thresholds for the sensor data. The HMD displays AR content comprising the warning notification in the transparent display.

Claims

1. A head mounted device (HMD) comprising:

a transparent display;

a plurality of sensors configured to measure sensor data; and

a processor comprising an augmented reality (AR) application and a threat application,

the threat application being configured to identify a threat condition based on a threat pattern and the sensor data, and to generate a warning notification in response to the identified threat condition, the threat pattern comprising preconfigured thresholds for the sensor data,

the threat application comprising a threat learning module being configured to access historical sensor data corresponding to user tasks, to identify user tasks with a negative outcome, to filter historical sensor data resulting in the negative outcome from the historical sensor data, and to generate the threat pattern based on the filtered historical sensor data; and

the AR application being configured to cause a display of AR content comprising the warning notification in the transparent display.

2. The HMD of claim 1, wherein the threat application comprises:

the threat learning module configured to receive the threat pattern from a server, the server configured to generate the preconfigured thresholds for the sensor data; and

a threat identification module configured to identify one of the sensor data transgressing one of the preconfigured thresholds, and to generate correction information based on the identified sensor data exceeding one of the preconfigured thresholds, the correction information including instructions to address the threat condition.

3. The HMD of claim 2, wherein the AR content includes the correction information, the correction information including instructions to a user of the HMD and a three-dimensional model including the correction information.

4. The HMD of claim 1, wherein the threat learning module is configured to identify the negative outcome based on preconfigured parameters for the sensor data.

5. The HMD of claim 1, wherein the plurality of sensors includes:

a first set of sensors configured to generate HMD-based sensor data related to the HMD; and

a second set of sensors configured to generate ambient-based sensor data related to an ambient environment of the HMD, the ambient environment including a physical space within a predefined distance of the HMD,

the sensor data comprising the HMD-based sensor data and the ambient-based sensor data.

6. The HMD of claim 5, wherein the threat pattern is generated based on preconfigured parameters for the HMD-based sensor data and the ambient-based sensor data.

7. The HMD of claim 5,

wherein the threat learning module is configured to identify a task provided by the AR application, a physical object associated with the task, historical HMD-based sensor data associated with the task and the physical object, historical ambient-based sensor data associated with the task and the physical object, historical user activities associated with the task, and an outcome of the historical user activities,

wherein the threat learning module is configured to generate the threat pattern based on a negative outcome of the historical user activities, the negative outcome being determined in response to the historical HMD-based sensor data or historical ambient-based sensor data transgressing a predefined threshold,

the threat pattern comprising a series of user activities with respect to the physical object, a series of HMD-based sensor data corresponding to the series of user activities, and a series of historical ambient-based sensor data corresponding to the series of user activities.

8. The HMD of claim 5, wherein the threat application is configured to receive further ambient-based sensor data from a third set of sensors external to the HMD,

the threat pattern comprising a series of historical ambient-based sensor data from the third set of sensors.

9. The HMD of claim 1, wherein the AR application is configured to determine a travel direction of the HMD based on at least one of the plurality of sensors, to determine that the travel direction is outside a field of view of a user of the HMD, to identify a moving physical object outside a field of view of the user of the HMD, to determine a trajectory of the moving physical object based on a travel path of the moving physical object, to determine that the trajectory of the moving physical object intersects the travel direction of the HMD at an estimated intersection located within a threshold distance of the HMD, and to generate the warning notification based on the estimated intersection.

10. A method comprising:

measuring sensor data using a plurality of sensors in a head-mounted device (HMD);

identifying a threat condition based on a comparison of a threat pattern with the sensor data, the threat pattern comprising preconfigured thresholds for the sensor data;

accessing historical sensor data corresponding to user tasks;

identifying user tasks with a negative outcome;

filtering historical sensor data resulting in the negative outcome from the historical sensor data, the threat pattern generated based on the filtered historical sensor data;

generating the threat pattern based on the filtered historical sensor data;

generating a warning notification in response to the identified threat condition; and

displaying, using a hardware processor of the HMD, AR content including the warning notification in a transparent display of the HMD.

11. The method of claim 10, further comprising:

receiving the threat pattern from a server, the server configured to generate the preconfigured thresholds for the sensor data; and

identifying one of the sensor data transgressing one of the preconfigured thresholds; and

generating correction information based on the identified sensor data exceeding one of the preconfigured thresholds, the correction information including instructions to address the threat condition.

12. The method of claim 11, wherein the AR content includes the correction information, the correction information including instructions to a user of the HMD and a three-dimensional model including the correction information.

13. The method of claim 10, further comprising:

identifying the negative outcome based on preconfigured parameters for the sensor data.

14. The method of claim 10, wherein the plurality of sensors includes:

a first set of sensors being configured to generate HMD-based sensor data related to the HMD; and

a second set of sensors being configured to generate ambient-based sensor data related to an ambient environment of the HMD, the ambient environment including a physical space within a predefined distance of the HMD,

the sensor data comprising the HMD-based sensor data and the ambient-based sensor data.

15. The method of claim 14, wherein the threat pattern is generated based on preconfigured parameters for the HMD-based sensor data and the ambient-based sensor data.

16. The method of claim 14, further comprising:

identifying a task provided by an AR application, a physical object associated with the task, historical HMD-based sensor data associated with the task and the physical object, historical ambient-based sensor data associated with the task and the physical object, historical user activities associated with the task, and an outcome of the historical user activities;

generating the threat pattern based on a negative outcome of the historical user activities, the negative outcome being determined in response to the historical HMD-based sensor data or historical ambient-based sensor data transgressing a predefined threshold,

the threat pattern comprising a series of user activities with respect to the physical object, a series of HMD-based sensor data corresponding to the series of user activities, and a series of historical ambient-based sensor data corresponding to the series of user activities.

17. The method of claim 10, further comprising:

determining a travel direction of the HMD based on at least one of the plurality of sensors;

determining that the travel direction is outside a field of view of a user of the HMD;

identifying a moving physical object outside a field of view of the user of the HMD;

determining a trajectory of the moving physical object based on a travel path of the moving physical object;

determining that the trajectory of the moving physical object intersects the travel direction of the HMD at an estimated intersection located within a threshold distance of the HMD; and

generating the warning notification based on the estimated intersection.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

measuring sensor data using a plurality of sensors in a head-mounted device (HMD);

identifying a threat condition based on a comparison of a threat pattern with the sensor data, the threat pattern comprising preconfigured thresholds for the sensor data;

accessing historical sensor data corresponding to user tasks;

identifying user tasks with a negative outcome;

filtering historical sensor data resulting in the negative outcome from the historical sensor data, the threat pattern generated based on the filtered historical sensor data;

generating the threat pattern based on the filtered historical sensor data;

generating a warning notification in response to the identified threat condition; and

displaying, using a hardware processor of the HMD, AR content including the warning notification in a transparent display of the HMD.