Threat identification system

• US 9,864,910 B2
• Filed: 03/02/2017
• Issued: 01/09/2018
• Est. Priority Date: 05/18/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew
• Primary Examiner(s): Mazumder, Tapas
• Application Number:
  US15/448,285
• Publication Number:
  US 20170177941A1
• Time in Patent Office: 313 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G08B 21/182 Level alarms, e.g. alarms responsive to variables exceeding a threshold
• CPC Class Codes: G08B 5/36 using visible light sources

Abstract

A head mounted device (HMD) includes a transparent display, sensors to generate sensor data, and a processor. The processor identifies a threat condition based on a threat pattern and the sensor data, and generates a warning notification in response to the identified threat condition. The threat pattern includes preconfigured thresholds for the sensor data. The HMD displays AR content comprising the warning notification in the transparent display.

Claims

1. A server comprising:

a storage device configured to store historical sensor data and task data, the historical sensor data generated from a first plurality of sensors of a plurality of augmented-reality devices; and

a hardware processor configured to execute a threat application, the threat application configured to perform operations comprising;

receiving sensor data from a second plurality of sensors of an augmented-reality device;

retrieving the historical sensor data and the task data from the storage device;

identifying a threat condition based on a comparison of a threat pattern with a combination of the sensor data, the historical sensor data, and the task data, the threat pattern comprising preconfigured thresholds for the sensor data based cm the historical sensor data and for the task data based on parameters of the task data;

generating a notification in response to the identified threat condition; and

communicating the notification to the augmented-reality device having a display, the display configured to display the notification.

2. The server of claim 1, wherein the operations further comprise:

generating the preconfigured thresholds for the sensor data based on the historical sensor data and for the task data based on the parameters of the task data;

identifying one of the sensor data and the task data transgressing one of the preconfigured thresholds;

generating correction information based on one of the identified sensor data and the identified task data exceeding one of the preconfigured thresholds, the correction information including instructions to address the threat condition.

3. The server of claim 2, wherein the operations further comprise:

accessing the sensor data and the task data corresponding to user tasks;

identifying user tasks with a negative outcome;

filtering the sensor data and the task data resulting in the negative outcome from the sensor data and the task data; and

generating the threat pattern based on the filtered sensor data and the filtered task data.

4. The server of claim 1, wherein the sensor data comprises real-time sensor data, and the task data comprises historical task data.

5. The server of claim 1, wherein the sensors comprise a combination of internal sensors and external sensors.

6. The server of claim 1, wherein the augmented-reality device includes a head mounted device (HMD).

7. The server of claim 6, wherein the second plurality of sensors includes:

a first set of sensors being configured to generate HMD-based sensor data related to the HMD; and

a second set of sensors being configured to generate ambient-based sensor data related to an ambient environment of the HMD, the ambient environment including a physical space within a predefined distance of the HMD.

8. The server of claim 7, wherein the operations further comprise:

generating the threat pattern based on preconfigured parameters for the HMD-based sensor data and for the ambient-based sensor data.

9. The server of claim 7, wherein the operations further comprise:

receiving further ambient-based sensor data from a third set of sensors external to the HMD,

the threat pattern comprising a series of historical ambient-based sensor data from the third set of sensors.

10. The server of claim 7, wherein the operations further comprise:

identifying a task retrieved from the storage device, a physical object associated with the task, historical HMD-based sensor data associated with the task and the physical object, historical ambient-based sensor data associated with the task and the physical object, historical user activities associated with the task, and an outcome of the historical user activities; and

generating the threat pattern based on a negative outcome of the historical user activities, the negative outcome being determined in response to the historical HMD-based sensor data or historical ambient-based sensor data transgressing a predefined threshold,

the threat pattern comprising a series of user activities with respect to the physical object, a series of HMD-based sensor data corresponding to the series of user activities, and a series of historical ambient-based sensor data corresponding to the series of user activities.

11. A method comprising:

receiving sensor data from a second plurality of sensors of an augmented-reality device;

retrieving historical sensor data and task data from a storage device of a server, the historical sensor data generated from a first plurality of sensors of a plurality of augmented-reality devices;

identifying a threat condition based on a comparison of a threat pattern with a combination of the sensor data, the historical sensor data, and the task data, the threat pattern comprising preconfigured thresholds for the sensor data based on the historical sensor data and for the task data based on parameters of the task data; and

generating a notification in response to the identified threat condition; and

communicating the notification to the augmented-reality device having a display, the display configured to display the notification.

12. The method of claim 11, further comprising:

generating the preconfigured thresholds for the sensor data based on the historical sensor data and for the task data based on the parameters of the task data;

identifying one of the sensor data and the task data transgressing one of the preconfigured thresholds; and

generating correction information based on one of the identified sensor data and the identified task data exceeding one of the preconfigured thresholds, the correction information including instructions to address the threat condition.

13. The method of claim 12, further comprising:

accessing the sensor data and the task data corresponding to user tasks;

identifying user tasks with a negative outcome;

filtering the sensor data and the task data resulting in the negative outcome from the sensor data and the task data; and

generating the threat pattern based on the filtered the sensor data and the filtered task data.

14. The method of claim 11, wherein the sensor data comprises real-time sensor data, and the task data comprises historical task data.

15. The method of claim 11, wherein the sensors comprise a combination of internal sensors and external sensors.

16. The method of claim 11, wherein the augmented-reality device includes a head mounted device (HMD).

17. The method of claim 16, wherein the second plurality of sensors includes:

a first set of sensors being configured to generate HMD-based sensor data related to the HMD; and

a second set of sensors being configured to generate ambient-based sensor data related to an ambient environment of the HMD, the ambient environment including a physical space within a predefined distance of the HMD.

18. The method of claim 17, further comprising:

generating the threat pattern based on preconfigured parameters for the HMD-based sensor data and for the ambient-based sensor data.

19. The method of claim 17, further comprising:

identifying a task retrieved from the storage device, a physical object associated with the task, historical HMD-based sensor data associated with the task and the physical object, historical ambient-based sensor data associated with the task and the physical object, historical user activities associated with the task, and an outcome of the historical user activities,

generating the threat pattern based on a negative outcome of the historical user activities, the negative outcome being determined in response to the historical HMD-based sensor data or historical ambient-based sensor data transgressing a predefined threshold; and

the threat pattern comprising a series of user activities with respect to the physical object, a series of HMD-based sensor data corresponding to the series of user activities, and a series of historical ambient-based sensor data corresponding to the series of user activities.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

receiving sensor data from a second plurality of sensors of an augmented-reality device;

retrieving historical sensor data and task data from a storage device of a server, the historical sensor data generated from the plurality of sensors of a plurality of augmented-reality devices;

identifying a threat condition based on a comparison of a threat pattern with a combination of the sensor data, the historical sensor data, and the task data, in which the threat pattern comprising preconfigured thresholds for the sensor data based on the historical sensor data and for the task data based on parameters of the task data;

generating a notification in response to the identified threat condition; and

communicating the notification to the augmented-reality device having a display, the display configured to display the notification.