Remote object detection and local tracking using visual odometry

• US 10,586,395 B2
• Filed: 05/09/2018
• Issued: 03/10/2020
• Est. Priority Date: 12/30/2013
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Wolf, Daniel, Zillner, Jakob, Micusik, Branislav, Hoff, William
• Primary Examiner(s): Doan, Duyen M
• Application Number:
  US15/974,998
• Publication Number:
  US 20180261012A1
• Time in Patent Office: 671 Days
• Field of Search: 709203, 709219
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: H04L 67/04 specially adapted for terminals or networks with limited capabilities; specially adapted for terminal portability
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 67/52 specially adapted for the location of the user terminal
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass

Abstract

A system and method for offloading object detection are described. A server receives first sensor data from a first sensor of an augmented reality (AR) display device. The first sensor data indicates a pose of the AR display device relative to a first reference coordinate system. The server detects a physical object using second sensor data received from a second sensor of the AR display device. The server determines, based on the second sensor data, a pose of the physical object relative to the AR display device. The server then determines the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the AR display device and the pose of the AR display device relative to the first reference coordinate system.

Claims

1. A method comprising:

receiving, by a server system, first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;

receiving, by the server system, second sensor data from a second sensor of the augmented reality display device;

detecting, by the server system, a physical object using the second sensor data;

determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;

determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and

providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.

2. The method of claim 1, wherein the augmented reality display device is configured to render a virtual object corresponding to the physical object, and display the virtual object in a display of the augmented reality display device using the pose of the physical object relative to the first reference coordinate system.

3. The method of claim 2, wherein the virtual object is displayed in the display of the augmented reality display device independently from the pose of the physical object relative to the augmented reality display device.

4. The method of claim 1, further comprising:

receiving, by the server system, third sensor data from a third sensor external to the augmented reality display device, the third sensor data indicating the pose of the augmented reality display device relative to the first reference coordinate system,

wherein determining the pose of the physical object relative to the first reference coordinate system is based on the pose of the physical object relative to the augmented reality display device and, using the third sensor data, the pose of the augmented reality display device relative to the first reference coordinate system.

5. The method of claim 4, further comprising:

rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and

providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.

6. The method of claim 1, further comprising:

rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and

providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.

7. The method of claim 1, wherein the first sensor includes a visual odometry system configured to identify a position and an orientation of the augmented reality display device relative to the first reference coordinate system.

8. The method of claim 1, wherein the second sensor includes an optical sensor configured to generate second sensor data based on an image of the physical object captured by the optical sensor.

9. The method of claim 1, further comprising:

receiving, by the server system, a request from the augmented reality display device to offload a tracking process of the augmented reality display device to the server system;

in response to receiving the request, generating offloaded processed data based on the first sensor data and second sensor data; and

providing the offloaded processed data to the augmented reality display device, the augmented reality display device configured to present, in a display of the augmented reality display device, virtual objects based on the offloaded processed data.

10. The method of claim 9, wherein third sensor data includes externally tracked data based on a location and an orientation of the AR display device relative to a third sensor that is external to the augmented reality display device, wherein the offloaded processed data is based on a combination of the first, second, and third sensor data.

11. A server system comprising:

one or more computer processors; and

one or more non-transitory computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the server system to perform operations comprising;

receiving first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;

receiving second sensor data from a second sensor of the augmented reality display device;

detecting a physical object using the second sensor data;

determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;

determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and

providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.

12. The server system of claim 11, wherein the augmented reality display device is configured to render a virtual object corresponding to the physical object, and display the virtual object in a display of the augmented reality display device using the pose of the physical object relative to the first reference coordinate system.

13. The server system of claim 12, wherein the virtual object is displayed in the display of the augmented reality display device independently from the pose of the physical object relative to the augmented reality display device.

14. The server system of claim 11, wherein the operations further comprise:

receiving third sensor data from a third sensor external to the augmented reality display device, the third sensor data indicating the pose of the augmented reality display device relative to the first reference coordinate system,

wherein determining the pose of the physical object relative to the first reference coordinate system is based on the pose of the physical object relative to the augmented reality display device and, using the third sensor data, the pose of the augmented reality display device relative to the first reference coordinate system.

15. The server system of claim 14, wherein the operations further comprise:

rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and

providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.

16. The server system of claim 11, wherein the operations further comprise:

rendering a virtual object based on the pose of the physical object relative to the first reference coordinate system; and

providing the rendered virtual object to the augmented reality display device, wherein the augmented reality display device is configured to display the rendered virtual object based on the pose of the physical object relative to the first reference coordinate system.

17. The server system of claim 11, wherein the first sensor includes a visual odometry system configured to identify a position and an orientation of the augmented reality display device relative to the first reference coordinate system.

18. The server system of claim 11, wherein the second sensor includes an optical sensor configured to generate second sensor data based on an image of the physical object captured by the optical sensor.

19. The server system of claim 11, wherein the operations further comprise:

receiving a request from the augmented reality display device to offload a tracking process of the augmented reality display device to the server system;

in response to receiving the request, generating offloaded processed data based on the first sensor data and second sensor data; and

providing the offloaded processed data to the augmented reality display device, the augmented reality display device configured to present, in a display of the augmented reality display device, virtual objects based on the offloaded processed data.

20. A non-transitory computer-readable medium storing instructions that, when executed by one or more computer processors of a server system, cause the server system to perform operations comprising:

receiving first sensor data from a first sensor of an augmented reality display device, the first sensor data indicating a pose of the augmented reality display device relative to a first reference coordinate system;

receiving second sensor data from a second sensor of the augmented reality display device;

detecting a physical object using the second sensor data;

determining, based on the second sensor data, a pose of the physical object relative to the augmented reality display device;

determining the pose of the physical object relative to the first reference coordinate system based on the pose of the physical object relative to the augmented reality display device and the pose of the augmented reality display device relative to the first reference coordinate system; and

providing the pose of the physical object relative to the first reference coordinate system to the augmented reality display device.