ARCHITECTURE FOR REAL-TIME EXTRACTION OF EXTENDED MAXIMALLY STABLE EXTREMAL REGIONS (X-MSERs)

• US 20160070975A1
• Filed: 04/15/2015
• Published: 03/10/2016
• Est. Priority Date: 09/10/2014
• Status: Active Grant

• Current Assignee: Khalifa University Of Science, Technology & Research
• Original Assignee: Khalifa University Of Science, Technology & Research
• Inventors: Salahat, Ehab Najeh, Saleh, Hani Hasan Mustafa, Sluzek, Andrzej Stefan, Elnaggar, Mohammed Ismail
• Granted Patent: US 9,600,739 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Architecture for real-time extraction of maximally stable extremal regions (MSERs) is disclosed. The architecture includes communication interface and processing circuitry that is adapted in hardware to receive a data streams of an intensity image and a depth image in real-time and provide intensity labels for image regions within the intensity image that match a given intensity threshold and provide depth labels for image regions within the depth image that match a given depth threshold. The processing circuitry is also adapted in hardware to find intensity extremal regions within the intensity image based upon the intensity labels and to find depth extremal regions within the depth image based upon the depth labels. The processing circuitry determines strong extremal regions based upon significant overlap between the intensity extremal regions and depth extremal regions. The processing circuitry then determines X-MSER ellipses parameters based upon the strong extremal regions and X-MSER criteria.

Claims

1. An architecture for real-time extraction of extended maximally stable extremal regions (X-MSERs) comprising a communication interface and processing circuitry adapted in hardware to in real-time:

receive a data stream of an intensity image via the communication interface;

receive a data stream of a depth image via the communication interface;

find intensity extremal regions within the intensity image;

find depth extremal regions within the depth image;

determine strong extremal regions based upon significant overlap between the intensity extremal regions and the depth extremal regions; and

determine X-MSER ellipses parameters based upon the strong extremal regions and X-MSER criteria.

2. The architecture of claim 1 wherein the X-MSER criteria include a nested X-MSER tolerance value.

3. The architecture of claim 2 wherein the X-MSER criteria further include a minimum X-MSER area, a maximum X-MSER area, and an acceptable growth rate value for X-MSER area.

4. The architecture of claim 1 wherein the X-MSER ellipses parameters include a center of gravity, a major axis length, a minor axis length, and an angle of the major axis length with respect to a horizontal axis.

5. The architecture of claim 1 wherein the processing circuitry includes X-MSER moments calculator hardware adapted to calculate X-MSER moments.

6. The architecture of claim 5 wherein the processing circuitry further includes elliptical fit approximator hardware adapted to receive X-MSER moments from the X-MSER moments calculator hardware and fit an X-MSER ellipse to an extremal region based upon the X-MSER moments.

7. The architecture of claim 1 wherein the processing circuitry includes union-find hardware adapted to provide intensity labels for image regions within the intensity image that match a given intensity threshold and adapted to provide depth labels for the image regions that match a given depth threshold.

8. The architecture of claim 7 wherein the processing circuitry includes extremal region find hardware that is adapted to receive the intensity labels and the depth labels for the image regions and find extremal regions based upon the intensity labels and the depth labels for the image regions.

9. The architecture of claim 8 wherein the extremal region find hardware is adapted to find extremal regions using a mathematical relationship q(t)=|Q(t+Δ

)|\Q(t−

Δ

)|/|Q(t)|, wherein each extremal region'"'"'s cardinality, |Q(t)| is a function of an intensity threshold t that is the given intensity threshold to find intensity extremal regions and the given depth threshold to find depth extremal regions.

10. The architecture of claim 1 wherein the processing circuitry includes X-MSER selector hardware adapted to automatically select X-MSERs based upon the X-MSER criteria.

11. An architecture for real-time extraction of extended maximally stable extremal regions (X-MSERs) comprising:

image process hardware adapted to receive a data stream of an intensity image and output intensity labels for image regions within the intensity image that match a given intensity threshold and to receive a data stream of a depth image and output depth labels for image regions within the depth image that match a given depth threshold;

extremal regions find hardware adapted to receive the intensity labels for the intensity image and find intensity extremal regions within the intensity image and to receive the depth labels for the depth image and find depth extremal regions within the depth image; and

X-MSER process hardware adapted to determine strong extremal regions based upon significant overlap between the intensity extremal regions and the depth extremal regions and to receive X-MSER criteria and output X-MSER ellipses parameters based upon the strong extremal regions.

12. The architecture of claim 11 wherein the X-MSER criteria include a nested MSER tolerance value.

13. The architecture of claim 12 wherein the X-MSER criteria further include a minimum X-MSER area value, a maximum X-MSER area value, and an acceptable growth rate value for MSER areas.

14. The architecture of claim 11 wherein the X-MSER ellipses parameters include a center of gravity, a major axis length, a minor axis length, and an angle of the major axis length with respect to a horizontal axis.

15. The architecture of claim 11 wherein the image process hardware includes union-find hardware adapted to label region seeds.

16. The architecture of claim 11 wherein the image process hardware includes union-find hardware adapted to provide the intensity labels for the image regions within the intensity image that match a given intensity threshold and to provide the depth labels for the image regions within the intensity image that match a given depth threshold.

17. The architecture of claim 16 wherein the image process hardware further includes labeled region seeds updater/unifier hardware adapted to prevent a seed that is a first pixel location within the intensity image from being stored in a seed list, if the seed is presently stored in the seed list.

18. The architecture of claim 17 further including region map updater hardware adapted to store a value of Q(t+Δ

), Q(t), and Q(t−

Δ

) for each seed, where t is an intensity threshold and Δ

is an increment of the intensity threshold t when processing an intensity image, and where t is a depth threshold and Δ

is an increment of the depth threshold t when processing a depth image.

19. The architecture of claim 16 wherein the extremal regions find hardware is adapted to find extremal regions using a mathematical relationship q(t)=|Q(t+Δ

)\|Q(t−

Δ

)|/|Q(t)|, where each extremal region'"'"'s cardinality, |Q(t)| is a function of an intensity threshold t that is the given intensity threshold to find intensity extremal regions and the given depth threshold to find depth extremal regions.

20. The architecture of claim 11 wherein the image process hardware, the extremal regions find hardware, and the X-MSER process hardware are fabricated on a single application specific integrated circuit (ASIC).

21. The architecture of claim 11 wherein the image process hardware, the extremal regions find hardware and the X-MSER process hardware are implemented on a single field programmable gate array (FPGA).

22. A method for real-time extraction of extended maximally stable extremal regions (X-MSERs) via processing circuitry comprising:

receiving a data stream of an intensity image via a communication interface in communication with the processing circuitry;

receiving a data stream of a depth image via the communication interface in communication with the processing circuitry;

generating intensity labels for image regions within the intensity image that match a given intensity threshold in real-time via the processing circuitry;

generating depth labels for the image regions within the depth image that match a given depth intensity threshold in real-time via the processing circuitry;

finding intensity extremal regions within the intensity image based upon the intensity labels in real-time via the processing circuitry;

finding depth extremal regions within the depth image based upon the depth labels in real-time via the processing circuitry;

determining strong extremal regions based upon significant overlap between the intensity extremal regions and the depth extremal regions; and

determining X-MSER ellipses parameters based on the strong extremal regions and X-MSER criteria in real-time via the processing circuitry.

23. The method for real-time extraction of X-MSERs via the processing circuitry of claim 22 wherein the X-MSER criteria include a nested X-MSER tolerance value.

24. The method for real-time extraction of MSERs via the processing circuitry of claim 23 wherein the X-MSER criteria further include a minimum X-MSER area, a maximum X-MSER area, and an acceptable growth rate value for X-MSER areas.

25. The method for real-time extraction of MSERs via the processing circuitry of claim 22 wherein the X-MSER ellipses parameters include a center of gravity, a major axis length, a minor axis length, and an angle of the major axis length with respect to a horizontal axis.

26. The method for real-time extraction of X-MSERs via the processing circuitry of claim 22 wherein the processing circuitry includes X-MSER moments calculator hardware adapted to calculate X-MSER moments.

27. The method for real-time extraction of X-MSERs via the processing circuitry of claim 26 wherein the processing circuitry further includes elliptical fit approximator hardware adapted to receive X-MSER moments from the X-MSER moments calculator hardware and fit an X-MSER ellipse to an extremal region based upon the X-MSER moments.

28. The method for real-time extraction of X-MSERs via the processing circuitry of claim 22 wherein the processing circuitry includes union-find hardware adapted to provide the intensity labels for the image regions within the intensity image that match a given intensity threshold and to provide the depth labels for the image regions within the depth image that match a given depth threshold.

29. The method for real-time extraction of X-MSERs via the processing circuitry of claim 28 wherein the processing circuitry includes extremal region find hardware that is adapted to receive the intensity labels and depth labels for the image regions and find the strong extremal regions based upon the intensity labels and the depth labels for the image regions.

30. The method for real-time extraction of X-MSERs via the processing circuitry of claim 29 wherein the extremal region find hardware is adapted to find the extremal regions using a mathematical relationship q(t)=|Q(t+Δ

)\|Q(t−

Δ

)|/|Q(t)|, where each extremal region'"'"'s cardinality, |Q(t)| is a function of an intensity threshold t that is the given intensity threshold to find the intensity extremal regions and the given depth threshold to find the depth extremal regions.

31. The method for real-time extraction of X-MSERs via the processing circuitry of claim 22 wherein the processing circuitry includes X-MSER selector hardware adapted to automatically select the X-MSERs based upon the X-MSER criteria.