Method and apparatus for determining disparity

• US 9,842,400 B2
• Filed: 08/25/2015
• Issued: 12/12/2017
• Est. Priority Date: 01/27/2015
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Shin, Changwoo, Lee, Kyoobin, Lee, Jun Haeng, Ryu, Eric Hyunsurk, Woo, Jooyeon
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US14/835,078
• Publication Number:
  US 20160217584A1
• Time in Patent Office: 840 Days
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals

Abstract

A disparity determination method and apparatus are provided. The disparity determination method includes receiving first signals of an event from a first sensor disposed at a first location and second signals of the event from a second sensor disposed at a second location that is different than the first location, and extracting a movement direction of the event, based on at least one among the first signals and the second signals. The disparity determination method further includes determining a disparity between the first sensor and the second sensor, based on the movement direction, a difference between times at which the event is sensed by corresponding pixels in the first sensor, and a difference between times at which the event is sensed by corresponding pixels in the first sensor and the second sensor.

Claims

1. A disparity determination method of a processing device, the method comprising:

receiving first signals of an event from a first sensor disposed at a first location and second signals of the event from a second sensor disposed at a second location that is different than the first location;

extracting a movement direction of the event, based on at least one among the first signals and the second signals;

determining a disparity between the first sensor and the second sensor, based on the movement direction, a first difference between first times at which the event is sensed by corresponding first pixels in the first sensor, and a second difference between second times at which the event is sensed by corresponding second pixels in the first sensor and the second sensor;

generating depth information, based on the disparity; and

outputting the depth information.

2. The disparity determination method of claim 1, wherein a resolution of the disparity is less than a size of pixels in each of the first sensor and the second sensor.

3. The disparity determination method of claim 1, wherein the corresponding first pixels sense the event corresponding to a point of an object at the first times that are different.

4. The disparity determination method of claim 1, further comprising setting, as the corresponding first pixels in the first sensor, a first pixel and a second pixel in the first sensor that sense the event, based on the movement direction.

5. The disparity determination method of claim 1, wherein the extracting comprises:

comparing a time of an element corresponding to one of the first signals to a time of another element neighboring the element in an event map of the first sensor; and

extracting the movement direction based on a result of the comparing.

6. The disparity determination method of claim 1, wherein the corresponding second pixels in the first sensor and the second sensor sense the event corresponding to a point of an object at the second times that are different.

7. The disparity determination method of claim 1, further comprising setting, as the corresponding second pixels in the first sensor and the second sensor, a first pixel in the first sensor and a second pixel in the second sensor that sense the event, based on the movement direction, an arrangement of pixels in each of the first sensor and the second sensor, and the disparity having a resolution equal to a size of each of the corresponding second pixels in the first sensor and the second sensor.

8. The disparity determination method of claim 1, further comprising:

determining whether the movement direction is matched to a virtual direction from the second location of the second sensor to the first location of the first sensor; and

setting third pixels corresponding to a same coordinate in the first sensor and the second sensor as the corresponding second pixels in the first sensor and the second sensor, in response to the determining that the movement direction is matched to the virtual direction.

9. The disparity determination method of claim 1, wherein a resolution of the disparity is equal to a size of pixels in each of the first sensor and the second sensor.

10. The disparity determination method of claim 1, wherein the determining comprises:

calculating an acceleration of the event, based on a third difference between third times at which the event is sensed by neighboring third pixels in the first sensor, and a fourth difference between fourth times at which the event is sensed by neighboring fourth pixels in the second sensor; and

determining the disparity, based on the acceleration, the first difference between the first times at which the event is sensed by the corresponding first pixels in the first sensor, and the second difference between the second times at which the event is sensed by the corresponding second pixels in the first sensor and the second sensor.

11. The disparity determination method of claim 1, wherein the event comprises a same change of light incident on the first sensor being previously detected by the second sensor.

12. The disparity determination method of claim 1, further comprising performing stereo matching between a first image generated based on the first signals and a second image generated based on the second signals, to generate the disparity having a resolution equal to a size of pixels in each of the first sensor and the second sensor.

13. The disparity determination method of claim 1, wherein the determining comprises:

calculating the first difference between the first times at which the event is sensed by the corresponding first pixels in the first sensor, based on a first event map of the first sensor; and

calculating the second difference between the second times at which the event is sensed by the corresponding second pixels in the first sensor and the second sensor, based on the first event map and a second event map of the second sensor.

14. The disparity determination method of claim 1, further comprising:

updating first timestamps of first elements corresponding to the first signals in a first event map of the first sensor, in response to the receiving the first signals; and

updating second timestamps of second elements corresponding to the second signals in a second event map of the second sensor, in response to the receiving the second signals.

15. A non-transitory computer-readable storage medium storing a program comprising instructions to cause a computer to implement the method of claim 1.

16. A disparity determination apparatus comprising:

an extractor configured to;

receive first signals of an event from a first sensor disposed at a first location and second signals of the event from a second sensor disposed at a second location that is different than the first location, and

extract a movement direction of the event, based on at least one among the first signals and the second signals; and

a processor configured to;

determine a disparity between the first sensor and the second sensor, based on the movement direction, a first difference between first times at which the event is sensed by corresponding first pixels in the first sensor, and a second difference between second times at which the event is sensed by corresponding second pixels in the first sensor and the second sensor;

generate depth information, based on the disparity; and

outputting the depth information.

17. The disparity determination apparatus of claim 16, wherein at least one among the first sensor and the second sensor comprises an event-based vision sensor configured to:

sense the event based on a change in incident light; and

output an event signal of the event that is sensed.

18. The disparity determination apparatus of claim 16, wherein at least one among the first sensor and the second sensor comprises an event-based vision sensor comprising pixels, each of the pixels configured to:

sense a movement; and

output an event signal of the movement that is sensed.

19. The disparity determination apparatus of claim 16, wherein at least one among the first sensor and the second sensor comprises:

a frame-based vision sensor configured to capture consecutive images for each frame; and

an output module configured to

sense the event based on a third difference between the consecutive images, and

output an event signal of the event.

20. The disparity determination apparatus of claim 16, wherein the extractor is further configured to

receive third signals of the event from a third sensor disposed at a third location that is different than the first location and the second location, and

extract the movement direction, based on at least one among the first signals, the second signals, and the third signals, and

the processor is further configured to determine the disparity, based on a third difference between third times at which the event is sensed by corresponding third pixels in the first sensor and the third sensor, the first difference between the first times at which the event is sensed by the corresponding first pixels in the first sensor, and the second difference between the second times at which the event is sensed by the corresponding second pixels in the first sensor and the second sensor.

21. The disparity determination apparatus of claim 16, further comprising a light emitter configured to project a moving light pattern.

22. The disparity determination apparatus of claim 16, further comprising:

a setter configured to set, as the corresponding first pixels in the first sensor, a first pixel and a second pixel in the first sensor that sense the event, based on the movement direction.

23. The disparity determination apparatus of claim 16, wherein the extractor is further configured to:

compare a first time of a first element corresponding to one of the first signals to a second time of a second element neighboring the first element in an event map of the first sensor; and

extract the movement direction based on a result of the first time of the first element being compared to the second time of the second element.

24. The disparity determination apparatus of claim 16, further comprising:

a setter configured to set, as the corresponding second pixels in the first sensor and the second sensor, a first pixel in the first sensor and a second pixel in the second sensor that sense the event, based on the movement direction, an arrangement of pixels in each of the first sensor and the second sensor, and the disparity having a resolution equal to a size of the corresponding second pixels in each of the first sensor and the second sensor.

25. The disparity determination apparatus of claim 16, further comprising:

a determiner configured to determine whether the movement direction is matched to a virtual direction from the second location of the second sensor to the first location of the first sensor; and

a setter configured to set pixels corresponding to a same coordinate in the first sensor and the second sensor as the corresponding second pixels in the first sensor and the second sensor, in response to the determiner determining that the movement direction is matched to the virtual direction.

26. The disparity determination apparatus of claim 16, wherein a resolution of the disparity is equal to a size of pixels in each of the first sensor and the second sensor.

27. The disparity determination apparatus of claim 16, wherein the processor is further configured to:

calculate an acceleration of the event, based on a third difference between third times at which the event is sensed by neighboring third pixels in the first sensor, and a fourth difference between fourth times at which the event is sensed by neighboring fourth pixels in the second sensor; and

determine the disparity, based on the acceleration, the first difference between the first times at which the event is sensed by the corresponding first pixels in the first sensor, and the second difference between the second times at which the event is sensed by the corresponding second pixels in the first sensor and the second sensor.

28. The disparity determination apparatus of claim 16, further comprising a matcher configured to perform stereo matching between a first image generated based on the first signals and a second image generated based on the second signals, to generate the disparity having a resolution equal to a size of pixels in each of the first sensor and the second sensor.

29. The disparity determination apparatus of claim 16, wherein the processor is further configured to:

calculate the first difference between the first times at which the event is sensed by the corresponding first pixels in the first sensor, based on a first event map of the first sensor; and

calculate the second difference between the second times at which the event is sensed by the corresponding second pixels in the first sensor and the second sensor, based on the first event map and a second event map of the second sensor.

30. The disparity determination apparatus of claim 16, further comprising an updater configured to:

update first timestamps of first elements corresponding to the first signals in a first event map of the first sensor, in response to the extractor receiving the first signals; and

update second timestamps of second elements corresponding to the second signals in a second event map of the second sensor, in response to the extractor receiving the second signals.