Extracting sensor data for augmented reality content

• US 9,412,205 B2
• Filed: 08/25/2014
• Issued: 08/09/2016
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew, Broaddus, Christopher
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US14/467,428
• Publication Number:
  US 20160055674A1
• Time in Patent Office: 715 Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for extracting data for augmented reality content are described. A device identifies a sensing device using an image captured with at least one camera of the device. Visual data are extracted from the sensing device. The device generates an AR content based on the extracted visual data and maps and displays the AR content in the display to form a layer on the sensing device.

Claims

1. A device comprising:

at least one camera;

a display; and

a hardware processor comprising an augmented reality (AR) application, the AR application configured to include;

an object recognition module to identify a sensing device, an image of which is captured with the at least one camera of the device;

a data extraction module to extract visual data from the sensing device;

an AR content generator to generate AR content based on the extracted visual data; and

an AR content mapping module to map the AR content in the display to form a layer visually perceived on the sensing device.

2. The device of claim 1, wherein the object recognition module further comprises:

a feature points module to identify feature points of objects in pictures received from the at least one camera, and to recognize the sensing device from the feature points of the sensing device in pictures received from the at least one camera.

3. The device of claim 1, wherein the data extraction module further comprises:

a units identifier module to identify measuring units associated with the sensing device;

a range identifier module to identify a range of the sensing device; and

a level identifier module to identify a measured level from the sensing device.

4. The device of claim 3, wherein the AR content generator further comprises:

an AR range identifier module to generate a virtual range indicator for the range of the sensing device;

an AR level identifier module to generate a virtual measured level indicator for the measured level from the sensing device; and

a digital gauge module to generate a virtual digital gauge based on the measured level from the sensing device.

5. The device of claim 4, wherein the AR content mapping module further comprises:

an AR range mapping module to map a first location in the display to display the virtual range indicator as a layer on the sensing device;

an AR level identifier module to map a second location in the display to display the virtual measured level indicator as a layer on the sensing device; and

a digital gauge module to map a third location in the display to display the virtual digital gauge in the display.

6. The device of claim 3, wherein the sensing device comprises a gauge with a dial, and further:

the units identifier module to identify measuring units displayed in a center of the dial;

the range identifier module to identify a range of the gauge based on a minimum range level and a maximum range level, both displayed in the dial; and

the level identifier module to identify the measured level from the sensing device based on a dial position relative to the gauge.

7. The device of claim 1, wherein the AR content generator is to access notification parameters related to the sensing device, and to generate an alert notification in the display in response to a measured level exceeding a threshold of the notification parameters.

8. The device of claim 1, wherein the AR application is to communicate the image of the sensing device to a server, to receive AR server content from the server, the server to extract visual data from the image of the sensing device, and generate the AR server content based on the extracted visual data, the server not in communication with the sensing device.

9. The device of claim 1, wherein the AR application is to communicate the image of the sensing device to a server, the server to identify the sensing device from the image, to extract visual data from the image, to access notification parameters associated with the identified sensing device, to generate an alert notification in the display in response to one of the visual data exceeding a threshold of the notification parameters.

10. The device of claim 1, further comprising:

a geolocation sensor to identify a geographic location of the device, wherein the object recognition module is to identify the sensing device based on the image of the sensing device and the geographic location of the device.

11. A method comprising:

identifying a sensing device using an image captured with at least one camera of a device;

extracting, by a hardware processor, visual data from the sensing device;

generating augmented reality (AR) content based on the extracted visual data; and

mapping and displaying the AR content in a display to form a layer visually perceived on the sensing device.

12. The method of claim 11, further comprising:

identifying feature points of objects in pictures received from the at least one camera; and

recognizing the sensing device from the features points of the sensing device in pictures received from the at least one camera.

13. The method of claim 11, further comprising:

identifying measuring units associated with the sensing device;

identifying a range of the sensing device; and

identifying a measured level from the sensing device.

14. The method of claim 13, further comprising:

generating a virtual range indicator for the range of the sensing device;

generating a virtual measured level indicator for the measured level from the sensing device; and

generating a virtual digital gauge based on the measured level from the sensing device.

15. The method of claim 14, further comprising:

mapping a first location in the display to display the virtual range indicator as a layer on the sensing device;

mapping a second location in the display to display the virtual measured level indicator as a layer visually perceived on the sensing device; and

mapping a third location in the display to display the virtual digital gauge in the display.

16. The method of claim 14, further comprising:

accessing notification parameters related to the sensing device; and

generating an alert notification in the display in response to the measured level exceeding a threshold of the notification parameters.

17. The method of claim 11, further comprising:

communicating the image of the sensing device to a server; and

receiving an AR server content from the server, the server to extract visual data from the image of the sensing device, and generate the AR server content based on the extracted visual data, the server not in communication with the sensing device.

18. The method of claim 11, further comprising:

communicating the image of the sensing device to a server, the server to identify the sensing device from the image, to extract visual data from the image, to access notification parameters associated with the identified sensing device, to generate an alert notification in the display in response to one of the visual data exceeding a threshold of the notification parameters; and

receiving the alert notification from the server.

19. The method of claim 11, further comprising:

identifying a geographic location of the device using a geolocation sensor in the device; and

identifying the sensing device based on the image of the sensing device and the geographic location of the device.

20. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

identifying a sensing device using an image captured with at least one camera of a device;

extracting visual data from the sensing device;

generating AR content based on the extracted visual data; and

mapping and displaying the AR content in a display to form a layer visually perceived on the sensing device.