Multi-spectrum segmentation for computer vision

• US 10,366,495 B2
• Filed: 01/26/2018
• Issued: 07/30/2019
• Est. Priority Date: 02/09/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Senthamil, Nalin, Lundquist, Eric Douglas
• Primary Examiner(s): Strege, John B
• Application Number:
  US15/880,886
• Publication Number:
  US 20180150957A1
• Time in Patent Office: 550 Days
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A device for multi-spectrum segmentation for computer vision is described. A first optical sensor operates within a first spectrum range and generates first image data corresponding to a first image captured by the first optical sensor. A second optical sensor operates within a second spectrum range different from the first spectrum range and generates second image data corresponding to a second image captured by the second optical sensor. The device identifies a first region in the first image, maps a first portion of the first image to a second portion of the second image data, provides the second portion of the second image data to a server that generates augmented reality content based on the second portion of the second image data. The device displays the augmented reality content.

Claims

1. A device comprising:

a first optical sensor configured to operate within a first spectrum range and to generate first image data corresponding to a first image captured by the first optical sensor, the first optical sensor comprising a depth sensor configured to generate depth image data for the first image;

a second optical sensor configured to operate within a second spectrum range different from the first spectrum range and to generate second image data corresponding to a second image captured by the second optical sensor;

one or more hardware processor configured to perform operations comprising;

identifying, from the first image data, a first region in the first image, the first region of the first image comprising corresponding depth image data that exceed a depth threshold,

mapping a first portion of the first image data corresponding to the first region in the first image to a second portion of the second image data corresponding to a second region in the second image,

providing the second portion of the second image data to a server configured to generate augmented reality content based on the second portion of the second image data, and

receiving the augmented reality content from the server; and

a display configured to display the augmented reality content.

2. The device of claim 1, wherein the identifying the first region in the first image further comprises determining that the first portion of the first image data transgress a first spectrum threshold.

3. The device of claim 2, wherein the server is configured to identify a physical object within the second region of the second image, and to generate the generated augmented reality content based on the identified physical object.

4. The device of claim 1, wherein the first optical sensor comprises an infrared sensor configured to generate infrared image data for the first image, and

wherein the first region of the first image includes corresponding infrared image data that exceed an infrared threshold.

5. The device of claim 1, wherein the operations further comprise:

mapping the first portion of the first image data to the second portion of the second image data based on a mapping between the first optical sensor and the second sensor, wherein the second region depicts an image of a physical object, the first region not depicting the image of the physical object; and

identifying and tracking the physical object based on the second image data.

6. The device of claim 1, wherein the first optical sensor and the second optical sensor are configured to capture a substantially similar field of view from the device.

7. The device of claim 1, wherein the operations further comprise:

receiving third image data from a remote device, the third image data generated with a third optical sensor of the remote device, the third optical sensor having a third spectrum range with frequencies lower than the frequencies of the second spectrum range of the second optical sensor, the third image data corresponding to a third image captured by the third optical sensor;

identifying, from the third image data, a third region of the third image, the third region including corresponding third image data that exceed a third optical sensor threshold;

mapping the third image data to the second image data based on relative locations between the remote device and the device, and orientation data of the remote device and the device; and

identifying a second region of the second image based on the mapping between the third image data and the second image data, the second region of the second image corresponding to the third region of the third image.

8. The device of claim 7, wherein the operations further comprise:

identifying and tracking a physical object depicted in the second region of the second image.

9. The device of claim 1, wherein the augmented reality content is displayed as an overlay to the physical object.

10. A method comprising:

accessing, at a device, first image data corresponding to a first image captured by a first optical sensor of the device, the first optical sensor configured to operate within a first spectrum range, the first optical sensor comprising a depth sensor configured to generate depth image data for the first image;

accessing at the device, second image data corresponding to a second image captured by a second optical sensor of the device, the second optical sensor configured to operate within a second spectrum range different from the first spectrum range;

identifying, from the first image data, a first region in the first image, the first region of the first image comprising corresponding depth image data that exceed a depth threshold;

mapping a first portion of the first image data corresponding to the first region in the first image to a second portion of the second image data corresponding to a second region in the second image;

providing the second portion of the second image data to a server configured to generate augmented reality content based on the second portion of the second image data;

receiving the augmented reality content from the server; and

displaying the augmented reality content in a display of the device.

11. The method of claim 10, wherein identifying the first region in the first image further comprises determining that the first portion of the first image data transgress a first spectrum threshold.

12. The method of claim 11, further comprising:

identifying a physical object within the second region of the second image; and

generating the generated augmented reality content based on the identified physical object.

13. The method of claim 10, wherein the first optical sensor comprises an infrared sensor configured to generate infrared image data for the first image, and

wherein the first region of the first image includes corresponding infrared image data that exceed an infrared threshold.

14. The method of claim 10, further comprising:

mapping the first image data to the second image data based on a mapping between the first sensor and the second sensor, wherein the second region depicts an image of a physical object, the first region not depicting the image of the physical object; and

identifying and tracking the physical object based on the second image data.

15. The method of claim 10, wherein the first optical sensor and the second optical sensor are configured to capture a substantially similar field of view from the device.

16. The method of claim 10, further comprising:

receiving third image data from a remote device, the third image data generated with a third optical sensor of the remote device, the third optical sensor having a third spectrum range with frequencies lower than the frequencies of the second spectrum range of the second optical sensor, the third image data corresponding to a third image captured by the third optical sensor;

identifying, from the third image data, a third region of the third image, the third region including corresponding third image data that exceed a third optical sensor threshold;

mapping the third image data to the second image data based on relative locations between the remote device and the device, and orientation data of the remote device and the device; and

identifying a second region of the second image based on the mapping between the third image data and the second image data, the second region of the second image corresponding to the third region of the third image.

17. The method of claim 16, further comprising:

identifying and tracking a physical object depicted in the second region of the second image, wherein the augmented reality content is displayed as an overlay to the physical object.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

accessing at a device, first image data corresponding to a first image captured by a first optical sensor of the device, the first optical sensor configured to operate within a first spectrum range, the first optical sensor comprising a depth sensor configured to generate depth image data for the first image;

accessing at the device, second image data corresponding to a second image captured by a second optical sensor of the device, the second optical sensor configured to operate within a second spectrum range different from the first spectrum range;

identifying, from the first image data, a first region in the first image, the first region of the first image comprising corresponding depth image data that exceed a depth threshold;

mapping a first portion of the first image data corresponding to the first region in the first image to a second portion of the second image data corresponding to a second region in the second image;

providing the second portion of the second image data to a server configured to generate augmented reality content based on the second portion of the second image data;

receiving the augmented reality content from the server; and

displaying the augmented reality content in a display of the device.