Multi-spectrum segmentation for computer vision

• US 9,916,664 B2
• Filed: 02/09/2016
• Issued: 03/13/2018
• Est. Priority Date: 02/09/2016
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Senthamil, Nalin, Lundquist, Eric Douglas
• Primary Examiner(s): Do, Anh H
• Application Number:
  US15/019,382
• Publication Number:
  US 20170228869A1
• Time in Patent Office: 763 Days
• Field of Search: 382312, 382173, 382103, 382128, 3482221, 348585, 348591, 348592, 348589, 348600, 348 E5031, 348 E5051, 362227, 362231, 362125, 362127, 362236, 362240, 362255, 36224902, 362276, 362800, 702 85, 702 92, 702 93
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20112 Image segmentation details
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A system and method for multi-spectrum segmentation for computer vision is described. A first sensor captures an image within a first spectrum range and generates first sensor data. A second sensor captures an image within a second spectrum range different than the first spectrum range and generates second sensor data. A multi-spectrum segmentation module identifies a segmented portion of the image within the second spectrum range based on: the second sensor data, a subset of the first sensor data corresponding to the segmented portion of the image within the second spectrum range, and a segmented portion of the image at the second spectrum range corresponding to the subset of the first sensor data. The multi-spectrum segmentation module identifies a physical object in the segmented portion of the image within the second spectrum range, and a device generates augmented reality content based on the identified physical object.

Claims

1. A device comprising:

a first sensor configured to operate within a first spectrum range and generate first sensor data;

a second sensor configured to operate within a second spectrum range different from the first spectrum range and to generate second sensor data;

one or more processors comprising an application and a multi-spectrum segmentation module,

the multi-spectrum segmentation module being configured to;

map the first sensor data to the second sensor data based on a mapping of the first and second sensors,

identify a first segmented portion of an image within the second spectrum range based on the second sensor data,

identify a subset of the second sensor data corresponding to the first segmented portion of the image,

identify a subset of the first sensor data corresponding to the subset of the second sensor data based on the mapping between the first sensor data and the second sensor data,

identify a second segmented portion of the image within the second spectrum range corresponding to the subset of the first sensor data, and

identify and track an image of a physical object in the second segmented portion of the image within the second spectrum range, and

the application being configured to generate content based on the identified physical object in the second segmented portion of the image; and

a display configured to display the content.

2. The device of claim 1, wherein the multi-spectrum segmentation module is configured to:

identify a first portion and a second portion of the image based on the first and second sensor data, the first portion depicting the image of the physical object, the second portion not depicting the image of the physical object;

identify a subset of the first and second sensor data, the subset corresponding to the first portion of the image; and

identify and track the physical object based on the subset of the first and second sensor data.

3. The device of claim 1, wherein the multi-spectrum segmentation module comprises:

a first sensor computation module configured to;

perform a first computation on the second sensor data, to identify the first segmented portion of the image within the second spectrum range based on the first computation, and to generate first segmentation information based on the identified first segmented portion of the image;

an application logic configured to receive the first segmentation information, to map the first segmentation information to a subset of the second sensor data to the first sensor data;

a second sensor computation module configured to receive the subset of the second sensor data from the application logic, to perform a second computation on the subset of the second sensor data, to track and identify the physical object based on the second computation, and to provide the application logic with tracking and identification of the physical object to the application logic.

4. The device of claim 3, wherein the multi-spectrum segmentation module further comprises:

a sensor mapping module configured to map the first sensor computation module with the second sensor computation module based on the first and second spectrum ranges of the first and second sensors, a first data input of the first sensor computation module for the first spectrum range corresponding to a second data input of the second sensor computation module for the second spectrum range.

5. The device of claim 1, wherein the multi-spectrum segmentation module further comprises:

a first sensor computation module configured to identify the second segmented portion of the image within the first spectrum range; and

a second sensor computation module configured to track the first segmented portion of the image within the second spectrum range,

wherein the first sensor and the second sensor include optical sensors that capture a substantially similar field of view from the device.

6. The device of claim 1, wherein the multi-spectrum segmentation module is configured to:

receive a third sensor data from a remote device, the third sensor data generated with a third sensor of the remote device, the third sensor having a third spectrum range with frequencies lower than the frequencies of the second spectrum range of the second sensor;

identify a third segmented portion of the image based on the third sensor data;

identify a subset of the second sensor data corresponding to the third segmented portion of the image; and

identify and track a physical object in the subset of the second sensor data.

7. The device of claim 6, wherein the multi-spectrum segmentation module is configured to:

map the third sensor data to the second sensor data based on relative locations between the remote device and the device, and orientation data of the remote device and the device; and

identify the subset of the second sensor data corresponding to the third segmented portion of the image based on the mapping between the third sensor data and the second sensor data.

8. The device of claim 1, wherein the multi-spectrum segmentation module is configured to:

receive a third sensor data from a first remote device, the third sensor data generated with a third sensor of the first remote device;

receive a fourth sensor data from a second remote device, the fourth sensor data generated with a fourth sensor of the second remote device, the third sensor having a spectrum range with frequencies lower than frequencies of the spectrum range of the fourth sensor;

identify a third segmented portion of the image based on the third sensor data;

identify a subset of the fourth sensor data corresponding to the third segmented portion of the image; and

identify and track the physical object in the subset of the fourth sensor data.

9. The device of claim 8, wherein the multi-spectrum segmentation module is configured to:

map the third sensor data to the fourth sensor data based on relative locations between the device, the first remote device, and the second remote device, and orientation data of the device, the first remote device, and the second remote device; and

identify the subset of the fourth sensor data corresponding to the third segmented portion of the image based on the mapping between the third sensor data and the fourth sensor data.

10. A method comprising:

operating a first sensor within a first spectrum range with a first sensor of a device;

generating first sensor data with the first sensor;

operating a second sensor within a second spectrum range different from the first spectrum range with a second sensor of the device;

generating second sensor data with the second sensor;

mapping the first sensor data to the second sensor data based on a mapping between the first and second sensors;

identifying a first segmented portion of an image within the second spectrum range based on the second sensor data;

identify a subset of the second sensor data corresponding to the first segmented portion of the image;

identify a subset of the first sensor data corresponding to the subset of the second sensor data based on the mapping between the first sensor data and the second sensor data;

identifying a second segmented portion of the image within the second spectrum range corresponding to the subset of the first sensor data;

identifying and tracking an image of a physical object in the second segmented portion of the image within the second spectrum range;

generating content based on the identified physical object in the second segmented portion of the image; and

causing a display of the content in a display of the device.

11. The method of claim 10, further comprising:

identifying a first portion and a second portion of the image based on the first and second sensor data, the first portion depicting the image of the physical object, the second portion not depicting the image of the physical object;

identifying a subset of the first and second sensor data corresponding to the first portion of the image; and

identifying and tracking the physical object based on the subset of the first and second sensor data.

12. The method of claim 10, further comprising:

performing a first computation on the second sensor data;

identifying the first segmented portion of the image within the second spectrum range based on the first computation;

generating first segmentation information based on the identified first segmented portion of the image;

receiving the first segmentation information;

mapping the first segmentation information to a subset of the second sensor data to the first sensor data;

receiving the subset of the second sensor data from the application logic;

performing a second computation on the subset of the second sensor data;

tracking and identifying the physical object based on the second computation; and

providing the application logic with tracking and identification of the physical object to the application logic.

13. The method of claim 12, further comprising:

mapping the first sensor data with the second sensor data based on the first and second spectrum ranges of the first and second sensors, a first data input of the first spectrum range corresponding to a second data input of the second spectrum range.

14. The method of claim 10, further comprising:

identifying the second segmented portion of the image within the first spectrum range;

tracking the first segmented portion of the image within the second spectrum range; and

capturing a substantially similar field of view with optical sensors of the first sensor and the second sensor.

15. The method of claim 10, further comprising:

receiving a third sensor data from a remote device, the third sensor data generated with a third sensor of the remote device, the third sensor having a third spectrum range with frequencies lower than the frequencies of the second spectrum range of the second sensor;

identifying a third segmented portion of the image based on the third sensor data;

identifying a subset of the second sensor data corresponding to the third segmented portion of the image; and

identifying and tracking a physical object in the subset of the second sensor data.

16. The method of claim 15, further comprising:

mapping the third sensor data to the second sensor data based on relative locations between the remote device and the device, and orientation data of the remote device and the device; and

identifying the subset of the second sensor data corresponding to the third segmented portion of the image based on the mapping between the third sensor data and the second sensor data.

17. The method of claim 10, further comprising:

receiving a third sensor data from a first remote device, the third sensor data generated with a third sensor of the first remote device;

receiving a fourth sensor data from a second remote device, the fourth sensor data generated with a fourth sensor of the second remote device, the third sensor having a spectrum range with frequencies lower than the frequencies of the spectrum range of the fourth sensor;

identifying a third segmented portion of the image based on the third sensor data;

mapping the third sensor data to the fourth sensor data based on relative locations between the device, the first remote device, and the second remote device, and orientation data of the device, the first remote device, and the second remote device;

identifying the subset of the fourth sensor data corresponding to the third segmented portion of the image based on the mapping between the third sensor data and the fourth sensor data; and

identifying and tracking the physical object in the subset of the fourth sensor data.

18. A non-transitory machine-readable medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

operating a first sensor within a first spectrum range with a first sensor of a device;

generating first sensor data with the first sensor;

operating a second sensor within a second spectrum range different from the first spectrum range with a second sensor of the device;

generating second sensor data with the second sensor;

mapping the first sensor data to the second sensor data based on a mapping between the first and second sensors;

identifying a first segmented portion of an image within the second spectrum range based on the second sensor data;

identify a subset of the second sensor data corresponding to the first segmented portion of the image;

identify a subset of the first sensor data corresponding to the subset of the second sensor data based on the mapping between the first sensor data and the second sensor data;

identifying a second segmented portion of the image within the second spectrum range corresponding to the subset of the first sensor data;

identifying and tracking an image of a physical object in the second segmented portion of the image within the second spectrum range;

generating content based on the identified physical object in the second segmented portion of the image; and

causing a display of the content in a display of the device.