Thermal detection in an augmented reality system

• US 9,536,355 B1
• Filed: 03/24/2016
• Issued: 01/03/2017
• Est. Priority Date: 03/24/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Kumar, Rajay, Sararu, Mark Anthony
• Primary Examiner(s): Hoang, Phi
• Application Number:
  US15/080,414
• Time in Patent Office: 285 Days
• US Class Current: 1/1
• CPC Class Codes: G01B 21/18 for measuring depth
• CPC Class Codes: G01J 5/00 Radiation pyrometry, e.g. infrared or optical thermometry
• CPC Class Codes: G01J 5/0025 Living bodies ear thermometers G01J5/0011; detecting, measuring or recording for diagnostic purposes A61B5/00
• CPC Class Codes: G01J 5/0066 for hot spots detection
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An augmented reality (AR) device comprises a transparent display, a thermal sensor, a depth sensor, and a processor. The thermal sensor generates thermal data related to physical objects detected by the AR device. A thermal baseline identification engine forms a thermal baseline using data from infrared imaging and depth profiling sensors, thermal data of the physical objects in a mapped 3D space, thermal context of the objects both spatially and based on their operating status, and their thermal parameters. A thermal anomaly identification engine identifies a thermal anomaly based on the established thermal baseline and the real time measurements, and generates a warning notification in response to the identified thermal anomaly.

Claims

1. An augmented reality (AR) device comprising:

a transparent display;

a thermal sensor configured to generate thermal data related to physical objects detected by the AR device;

a depth sensor configured to generate depth data of the physical objects and map the physical objects in a three-dimensional space based on the depth data;

a processor comprising an augmented reality (AR) application, a thermal anomaly detection engine, a thermal baseline identification module, and a thermal anomaly identification module,

the thermal anomaly detection engine configured to access a thermal baseline established based on a combination of depth data of the physical objects, thermal data of the physical objects in the mapped three-dimensional space, and thermal context of the physical objects based on their operating status and thermal parameters, to identify a thermal anomaly based on a context of the physical objects, the thermal data of the physical objects, and the established thermal baseline, and to generate a warning notification in response to the identified thermal anomaly,

the thermal baseline identification module configured to identify operating thermal ranges of the physical objects, and to generate the thermal parameters based on the operating thermal ranges of the physical objects,

the thermal anomaly identification module configured to identify a physical object with a temperature exceeding the thermal parameters, and to generate correction information based on the context of the physical objects, and

the AR application configured to cause a display of AR content comprising identifying the physical object associated with the thermal anomaly in the transparent display, and to display the correction information in the transparent display.

2. The AR device of claim 1, wherein the context of the physical objects includes thermal ranges for each physical object based on a combination of at least an operating status of each physical object, the thermal ranges of one physical object dependent on the operating status of the other physical objects in the three-dimensional space, and operating temperatures of other physical objects in contact with or nearby the physical objects.

3. The AR device of claim 2, wherein the thermal baseline identification module is configured to sample thermal data on the mapped physical objects after determining that the physical objects are operational, and to send the sample thermal data to a server.

4. The AR device of claim 3, wherein the server is configured to receive a plurality of sample thermal data related to the three-dimensional space from a plurality of AR devices, to identify a context and operating status of the physical objects in the three-dimensional space, and to generate thermal parameters based on the plurality of sample thermal data and the context and operating status of the physical objects.

5. The AR device of claim 4, wherein the thermal baseline identification module is configured to receive the thermal parameters from the server.

6. The AR device of claim 2, wherein the thermal anomaly identification module is configured to identify a part of the physical object with corresponding thermal data exceeding the thermal parameters for the context of the physical objects.

7. The AR device of claim 1, wherein the AR content includes instructions to be displayed in the transparent display, the AR content comprising a three-dimensional model representing the instructions.

8. The AR device of claim 1, wherein the thermal anomaly identification module is configured to identify a part of the physical objects causing the thermal anomaly based on the context, the AR content comprising instructions to repair the identified part of the physical objects.

9. The AR device of claim 1, wherein the thermal anomaly identification module is configured to identify a part of the mapped three-dimensional space causing the thermal anomaly, the AR content comprising instructions related to the part of the mapped space.

10. A method comprising:

determining thermal data related to physical objects detected by a thermal sensor of an Augmented Reality (AR) device;

generating depth data of the physical objects;

mapping the physical objects in a three-dimensional space based on the depth data;

accessing a thermal baseline established based on a combination of the depth data of the physical objects, the thermal data of the physical objects in the mapped three-dimensional space, and a thermal context of the physical objects based on operating status and thermal parameters;

identifying a thermal anomaly based on a context of the physical objects, the thermal data of the physical objects, and the thermal parameters;

identifying operating thermal ranges of the physical objects;

generating the thermal parameters based on the operating thermal ranges of the physical objects;

identifying a physical object with a temperature exceeding the thermal parameters;

generating a warning notification in response to the identified thermal anomaly;

generating correction information based on the context of the physical objects;

causing a display of AR content comprising the warning notification and an identification of the physical object associated with the thermal anomaly in a transparent display of the AR device; and

displaying the correction information in the transparent display.

11. The method of claim 10, wherein the context of the physical objects includes thermal ranges for each physical object based on at least a combination of an operating status of each physical object, the thermal ranges of one physical object dependent on the operating status of the other physical objects in the three-dimensional space, and operating temperatures of other physical objects in contact with or nearby the physical objects.

12. The method of claim 11, further comprising:

sampling thermal data on the mapped physical objects after determining that the physical objects are operational; and

sending the sample thermal data to a server.

13. The method of claim 12, wherein the server is configured to receive a plurality of sample thermal data related to the three-dimensional space from a plurality of AR devices, to identify the context and operating status of the physical objects in the three-dimensional space, and to generate the thermal parameters based on the plurality of sample thermal data and the context and operating status of the physical objects.

14. The method of claim 13, further comprising:

receiving the thermal parameters from the server.

15. The method of claim 11, further comprising:

identifying a part of the physical object with corresponding thermal data exceeding the thermal parameters for the context of the physical objects.

16. The method of claim 10, wherein the AR content includes instructions to be displayed in the transparent display, the AR content comprising a three-dimensional model representing the instructions.

17. The method of claim 10, further comprising:

identifying a part of the physical objects causing the thermal anomaly based on the context, the AR content comprising instructions to repair the identified part of the physical objects.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of an Augmented Reality (AR) device, cause the AR device to perform operations comprising:

generating thermal data related to physical objects detected by a thermal sensor of the AR device;

generating depth data of the physical objects;

mapping the physical objects in a three-dimensional space based on the depth data;

accessing a thermal baseline established based on a combination of the depth data of the physical objects, the thermal data of the physical objects in the mapped three-dimensional space, and a thermal context of the physical objects based on operating status and thermal parameters;

identifying a thermal anomaly based on a context of the physical objects, the thermal data of the physical objects, and the established thermal baseline;

identifying operating thermal ranges of the physical objects;

generating the thermal parameters based on the operating thermal ranges of the physical objects;

identifying a physical object with a temperature exceeding the thermal parameters;

generating a warning notification in response to the identified thermal anomaly; and

generating correction information based on the context of the physical objects;

causing a display of AR content comprising the warning notification and an identification of the physical object associated with the thermal anomaly in a transparent display of the AR device; and

displaying the correction information in the transparent display.