AUGMENTED REALITY SENSOR

• US 20190035152A1
• Filed: 07/26/2017
• Published: 01/31/2019
• Est. Priority Date: 07/26/2017
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: DAQRI, LLC
• Inventors: Kazansky, Lucas
• Granted Patent: US 10,733,799 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/252 between a Database Management System and a front-end application
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal

Abstract

An augmented-reality (AR) device comprises a physical sensor and a display. The AR device identifies, using the physical sensor, a location and orientation of the AR device within a physical environment. A selection of a type of a virtual sensor and a placement location of the virtual sensor within the physical environment is received at the AR device. The placement location of the virtual sensor is determined relative to the location and orientation of the AR device within the physical environment. The AR device provides the type of the virtual sensor and the placement location of the virtual sensor to a server that generates virtual content for the virtual sensor. The AR device displays the virtual content at the placement location of the virtual sensor in the display.

Claims

1. An augmented-reality (AR) device comprising:

a physical sensor;

a display; and

one or more hardware processors comprising an AR sensor application, the AR sensor application configured to perform operations comprising;

identifying, using the physical sensor, a location and orientation of the AR device within a physical environment;

receiving a selection of a type of a virtual sensor and a placement location of the virtual sensor within the physical environment, the placement location of the virtual sensor determined relative to the location and orientation of the AR device within the physical environment; and

providing the type of the virtual sensor and the placement location of the virtual sensor to a server, the server configured to generate virtual content for the virtual sensor displayed at the placement location of the virtual sensor in the display.

2. The AR device of claim 1, wherein the server is further configured to:

identify a sensor data source for the virtual sensor based on the type and placement location of the virtual sensor;

receive sensor data from the sensor data source; and

generate the virtual sensor as coupled to a physical object in the physical environment at the placement location of the virtual sensor,

wherein the virtual content is generated based on the sensor data from the sensor data source and the type of the virtual sensor.

3. The AR device of claim 2, wherein the server is further configured to:

update the virtual content based on the sensor data from the sensor data source, the sensor data source including a combination of a first physical sensor of the AR device, a second physical sensor coupled to the server, a third physical sensor coupled to the physical object at the placement location of the virtual sensor, and a fourth physical sensor of another AR device.

4. The AR device of claim 2, wherein the operations further comprise:

displaying, in the display, the virtual content as an overlay to the physical object at the placement location of the virtual sensor.

5. The AR device of claim 2, wherein the physical sensor comprises:

a location sensor configured to determine the location of the AR device;

an orientation sensor configured to determine the orientation of the AR device; and

an attribute sensor configured to measure an attribute related to one of the physical object and the location of the AR device.

6. The AR device of claim 1, wherein the operations further comprise:

accessing first sensor data from the physical sensor of the AR device located at the location of the AR device;

receiving second sensor data from a second physical sensor external to the AR device; and

forming third sensor data based on the first and second sensor data,

wherein the virtual content is generated based on the third sensor data.

7. The AR device of claim 1, wherein the operations further comprise:

providing the location and orientation of the AR device to the server;

receiving second virtual content related to a second virtual sensor based on the location and orientation of the AR device, the second virtual content based on second sensor data associated with the second virtual sensor; and

displaying, in the display, the second virtual content as an overlay to a second physical object associated with a location of the second virtual sensor.

8. The AR device of claim 1, wherein the operations further comprise:

determining that the location of the AR device is within a predefined distance of a second virtual sensor, the second virtual sensor placed by a second virtual device;

receiving second virtual content related to the second virtual sensor in response to determining that the location of the AR device is within the predefined distance of the second virtual sensor, the second virtual content based on second sensor data associated with the second virtual sensor; and

displaying, in the display, the second virtual content as an overlay to a second physical object associated with a location of the second virtual sensor.

9. The AR device of claim 1, wherein the operations further comprise:

forming a relational database identifying the virtual sensor, the type of the virtual sensor, the placement location of the virtual sensor, a physical object associated with the placement location of the virtual sensor, a sensor data source associated with the type and placement location of the virtual sensor and the physical object, and virtual content for the virtual sensor based on sensor data from the sensor data source.

10. The AR device of claim 1, wherein the virtual sensor includes an AR sensor associated with the selected placement location, the AR sensor configured to appear as an overlay to a physical object associated with the placement location of the AR sensor within the physical environment.

11. A method comprising:

identifying, using a physical sensor of an augmented-reality (AR) device, a location and orientation of the AR device within a physical environment, the AR device comprising a display;

receiving, at the AR device, a selection of a type of a virtual sensor and a placement location of the virtual sensor within the physical environment, the placement location of the virtual sensor determined relative to the location and orientation of the AR device within the physical environment; and

providing the type of the virtual sensor and the placement location of the virtual sensor to a server, the server configured to generate virtual content for the virtual sensor displayed at the placement location of the virtual sensor in the display.

12. The method of claim 11, further comprising:

identifying a sensor data source for the virtual sensor based on the type and placement location of the virtual sensor;

receiving sensor data from the sensor data source; and

generating the virtual sensor as coupled to a physical object in the physical environment at the placement location of the virtual sensor,

wherein the virtual content is generated based on the sensor data from the sensor data source and the type of the virtual sensor.

13. The method of claim 12, further comprising:

updating the virtual content based on the sensor data from the sensor data source, the sensor data source including a combination of a first physical sensor of the AR device, a second physical sensor coupled to the server, a third physical sensor coupled to the physical object at the placement location of the virtual sensor, and a fourth physical sensor of another AR device.

14. The method of claim 12, further comprising:

displaying, in the display, the virtual content as an overlay to the physical object at the placement location of the virtual sensor.

15. The method of claim 12, further comprising:

determining the location of the AR device with a location sensor of the AR device;

determining the orientation of the AR device with an orientation sensor of the AR device; and

measuring an attribute related to one of the physical object and the location of the AR device with an attribute sensor of the AR device.

16. The method of claim 11, further comprising:

accessing first sensor data from the physical sensor of the AR device located at the location of the AR device;

receiving second sensor data from a second physical sensor external to the AR device; and

forming third sensor data based on the first and second sensor data,

wherein the virtual content is generated based on the third sensor data.

17. The method of claim 11, further comprising:

providing the location and orientation of the AR device to the server;

receiving second virtual content related to a second virtual sensor based on the location and orientation of the AR device, the second virtual content based on second sensor data associated with the second virtual sensor; and

displaying, in the display, the second virtual content as an overlay to a second physical object associated with a location of the second virtual sensor.

18. The method of claim 11, further comprising:

determining that the location of the AR device is within a predefined distance of a second virtual sensor, the second virtual sensor placed by a second AR device;

receiving second virtual content related to the second virtual sensor in response to determining that the location of the AR device is within the predefined distance of the second virtual sensor, the second virtual content based on second sensor data associated with the second virtual sensor; and

displaying, in the display, the second virtual content as an overlay to a second physical object associated with a location of the second virtual sensor.

19. The method of claim 11, further comprising:

forming a relational database identifying the virtual sensor, the type of the virtual sensor, the placement location of the virtual sensor, a physical object associated with the placement location of the virtual sensor, a sensor data source associated with the type and placement location of the virtual sensor and the physical object, and virtual content for the virtual sensor based on sensor data from the sensor data source.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying, using a physical sensor of an augmented-reality (AR) device, a location and orientation of the AR device within a physical environment, the AR device comprising a display;

receiving, at the AR device, a selection of a type of a virtual sensor and a placement location of the virtual sensor within the physical environment, the placement location of the virtual sensor determined relative to the location and orientation of the AR device within the physical environment; and

providing the type of the virtual sensor and the placement location of the virtual sensor to a server, the server configured to generate virtual content for the virtual sensor displayed at the placement location of the virtual sensor in the display.