Spatial data collection

• US 9,799,142 B2
• Filed: 08/15/2014
• Issued: 10/24/2017
• Est. Priority Date: 08/15/2014
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew, Irving, Jr., Frank Chester
• Primary Examiner(s): He, Yingchun
• Application Number:
  US14/461,264
• Publication Number:
  US 20160049006A1
• Time in Patent Office: 1,166 Days
• Field of Search: None
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for spatial data collection are described. Sensor data related to a position and an orientation of a device are generated over time using sensors of the device. Augmented reality content is generated based on a physical object captured by the device. A path bundle data package identifying a user interaction of the device with the augmented reality content relative to the physical object is generated. The user interaction identifies a spatial path of an interaction with the augmented reality content. The path bundle data package is generated based on the sensor data using a data model comprising a data header and a data payload. The data header comprises a contextual header having data identifying the device and a user of the device. A path header includes data identifying the path of the interaction with the augmented reality content. A sensor header includes data identifying the sensors. The data payload comprises dynamically sized sampling data from the sensor data.

Claims

1. A device comprising:

a plurality of sensors configured to generate sensor data related to a position and an orientation of the device;

a hardware processor comprising an augmented reality application, the augmented reality application including;

an augmented reality processing module configured to generate augmented reality content based on a physical object captured by the device;

a data aggregation module configured to generate a path bundle data package describing user interaction of the device with the augmented reality content relative to the physical object,

the user interaction identifying a spatial path of an interaction with the augmented reality content,

the path bundle data package generated based on the sensor data using a data model comprising a data header and a data payload,

the data header comprising a contextual header having data identifying the device and a user of the device, a path header having data identifying the spatial path of the interaction with the augmented reality content, and a sensor header having data identifying the plurality of sensors, the data payload comprising dynamically sized sampling data from the sensor data, and

the path header comprising a path start time, a path end time, a type of sensor identifier for each corresponding sensor, a total duration of the path, and a total distance of the path.

2. The device of claim 1, wherein the plurality of sensors comprises optical sensors and inertial sensors.

3. The device of claim 1, further comprising a display configured to display the augmented reality content,

wherein the augmented reality content comprises a virtual object generated based on the physical object captured with at least one of the sensors of the device, the virtual object comprising a three-dimensional model, a view of the three-dimensional model dynamically adjusted based on the position and orientation of the device relative to the physical object.

4. The device of claim 1, wherein the contextual header comprises a device identifier, a session identifier, a physical location, and a device type.

5. The device of claim 1, wherein the sensor header comprises, for each sensor, a sensor identifier, a sensor position identifier, three dimensional position data, orientation data, and a number of data samples.

6. The device of claim 1, wherein the dynamically sized sampling data comprises a sensor identifier, a data sample number, sensor data, a relative time, a duration percentage through the path, and a time percentage through the path.

7. The device of claim 1, wherein the data aggregation module is configured to aggregate sensor data from the plurality of sensors using the data model.

8. The device of claim 1, wherein the path of the interaction comprises a spatial path relative to the augmented reality content.

9. The device of claim 1, further comprising:

the data aggregation module configured to generate a path bundle streaming data package, the path bundle streaming data package comprising the path bundle data package without data points where information is generated after the data point is generated.

10. A method comprising:

generating sensor data related to a position and an orientation of a device over time using a plurality of sensors of the device;

generating, using a processor of a machine, augmented reality content based on a physical object captured by the device; and

generating a path bundle data package describing a user interaction of the device with the augmented reality content relative to the physical object,

the user interaction identifying a spatial path of an interaction with the augmented reality content,

the path bundle data package generated based on the sensor data using a data model comprising a data header and a data payload,

the data header comprising a contextual header having data identifying the device and a user of the device, a path header having data identifying the path of the interaction with the augmented reality content, and a sensor header having data identifying the plurality of sensors, the data payload comprising dynamically sized sampling data from the sensor data, and

the path header comprising a path start time, a path end time, a type of sensor identifier for each corresponding sensor, a total duration of the path, and a total distance of the path.

11. The method of claim 10, wherein the plurality of sensors comprises optical sensors and inertial sensors.

12. The method of claim 10, further comprising:

displaying the augmented reality content,

wherein the augmented reality content comprises a virtual object generated based on the physical object captured with at least one of the sensors of the device, the virtual object comprising a three-dimensional model, a view of the three-dimensional model dynamically adjusted based on the position and orientation of the device relative to the physical object.

13. The method of claim 10, wherein the contextual header comprises a device identifier, a session identifier, a physical location, and a device type.

14. The method of claim 10, wherein the sensor header comprises, for each sensor, a sensor identifier, a sensor position identifier, three dimensional position data, orientation data, and a number of data samples.

15. The method of claim 10, wherein the dynamically sized sampling data comprises a sensor identifier, a data sample number, sensor data, a relative time, a duration percentage through the path, and a time percentage through the path.

16. The method of claim 10, further comprising aggregating sensor data from the plurality of sensors using the data model, wherein the path of the interaction comprises a spatial path relative to the augmented reality content.

17. The method of claim 10, further comprising:

generating a path bundle streaming data package, the path bundle streaming data package comprising the path bundle data package without data points where information is generated after the data point is generated; and

displaying, in a transparent display of a head-mounted wearable device, the augmented reality content, a position and size of the augmented reality content in the transparent display based on a position and orientation of the device relative to the physical object.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

generating sensor data related to a position and an orientation of a device over time using a plurality of sensors of the device;

generating augmented reality content based on a physical object captured by the device;

generating a path bundle data package describing a user interaction of the device with the augmented reality content relative to the physical object,

the user interaction identifying a spatial path of an interaction with the augmented reality content,

the path bundle data package generated based on the sensor data using a data model comprising a data header and a data payload,

the data header comprising a contextual header having data identifying the device and a user of the device, a path header having data identifying the path of the interaction with the augmented reality content, and a sensor header having data identifying the plurality of sensors, the data payload comprising dynamically sized sampling data from the sensor data, and

the path header comprising a path start time, a path end time, a type of sensor identifier for each corresponding sensor, a total duration of the path, and a total distance of the path.