Machine vision for ego-motion, segmenting, and classifying objects

• US 10,043,282 B2
• Filed: 04/13/2016
• Issued: 08/07/2018
• Est. Priority Date: 04/13/2015
• Status: Active Grant

• Current Assignee: Gerard Dirk Smits
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Smits, Gerard Dirk
• Primary Examiner(s): Wang, Carol
• Application Number:
  US15/098,285
• Publication Number:
  US 20160335778A1
• Time in Patent Office: 846 Days
• CPC Class Codes: G01P 3/36 Devices characterised by the use of optical means, e.g. using infrared, visible, or ultraviolet light G01P3/68 takes precedence; gyrometers using the Sagnac effect, i.e. rotation-induced shifts between counter-rotating electromagnetic beams G01C19/64
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56

Abstract

Systems and methods for machine vision are presented. Such machine vision includes ego-motion, as well as the segmentation and/or classification of image data of one or more targets of interest. The projection and detection of scanning light beams that generate a pattern are employed. Real-time continuous and accurate spatial-temporal 3D sensing is achieved. The relative motion between an observer and a projection surface is determined. A combination of visible and non-visible patterns, as well as a combination of visible and non-visible sensor arrays is employed to sense 3D coordinates of target features, as well as acquire color image data to generate 3D color images of targets. Stereoscopic pairs of cameras are employed to generate 3D image data. Such cameras are dynamically aligned and calibrated. Information may be encoded in the transmitted patterns. The information is decoded upon detection of the pattern and employed to determine features of the reflecting surface.

Claims

1. A system for sensing relative motion of a target, comprising:

one or more transmitters that transmit one or more light patterns;

one or more receivers that detect the one or more light patterns;

one or more memory devices for storing instructions; and

one or more processor devices that execute the instructions that enable actions, including;

employing the one or more transmitters to transmit the one or more light patterns to the target, wherein the one or more transmitters scan the one or more light patterns across the target in one or more directions;

employing the one or more light patterns to generate a pattern on the target, wherein a rotation of the pattern is controlled by modulating the one or more light patterns;

employing the one or more receivers to detect one or more reflected light patterns from the target and generate one or more signals that correspond to the one or more reflected light patterns;

employing the one or more signals to determine at least one of a spatial elongation or contraction of the one or more reflected light patterns in the one or more directions;

providing one or more characteristics of a relative velocity between the target and the one or more receivers based on at least one of the determined spatial elongation or the determined contraction of the one or more reflected light patterns; and

providing a magnitude of the relative velocity based on one or more portions of the rotation of the pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

2. The system of claim 1, wherein the one or more light patterns are scanned across the target to generate a two-dimensional (2D) pattern on the target and the one or more characteristics of the relative velocity includes a direction based on one or more portions of the 2D pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

3. The system of claim 1, the actions further including:

scanning the one or more light patterns across the target such that a first row of a plurality of rows of pixels included in the one or more receivers is illuminated by the one or more reflected light patterns during a first time period;

employing a feedback loop between the one or more receivers and the one or more transmitters to synchronize a shutter included in the one or more receivers such that the shutter exposes the first row of the plurality of rows during the first time period;

scanning the one or more light patterns along the target such that a second row of the plurality of rows of a sensor array is illuminated by the one or more reflected light patterns during a second time period that is later than the first time period; and

employing the feedback loop to synchronize the shutter such that the shutter exposes the second row of the plurality of rows during the second time period.

4. The system of claim 1, wherein at least one of the one or more receivers includes a first sensor array sensitive to photons within a visible portion of an electromagnetic (EM) spectrum and a second sensor array sensitive to photons within a non-visible portion of the EM spectrum, wherein the first and second sensor arrays are stacked such that at least a portion of the one or more reflected patterns are transmitted through the first sensor array prior to detection by the second sensor array.

5. The system of claim 1, wherein the actions further comprise:

encoding one or more transmission angles of the one or more light patterns in the one or more light patterns by modulating a combination of a plurality of light sources included in the one or more transmitters, wherein the combination of the plurality of light sources generate the one or more light patterns;

decoding the one or more transmission angles based on the one or more signals; and

providing at least a position of the target based on at least the decoded one or more transmission angles.

6. The system of claim 1, wherein at least one of the one or more receivers includes one or more sensors that include a plurality of sensor pixels, wherein a first portion of the plurality of sensor pixels are sensitive to photons within a visible portion of an electromagnetic (EM) spectrum and a second portion of the plurality of sensor pixels are sensitive to photons within a non-visible portion of the EM spectrum and the first and second portions of the plurality of sensor pixels are spatially multiplexed.

7. The system of claim 1, wherein at least one of the one or more receivers includes a plurality of photodiodes, wherein each of the plurality of photodiodes is uniquely coupled to one of a plurality of row sense lines and to one of a plurality of column sense lines.

8. A method for sensing relative motion of a target, comprising:

employing the one or more transmitters to transmit the one or more light patterns to the target, wherein the one or more transmitters scan the one or more light patterns across the target in one or more directions;

employing the one or more light patterns to generate a pattern on the target, wherein a rotation of the pattern is controlled by modulating the one or more light patterns;

employing the one or more receivers to detect one or more reflected light patterns from the target and generate one or more signals that correspond to the one or more reflected light patterns;

employing the one or more signals to determine at least one of a spatial elongation or contraction of the one or more reflected light patterns in the one or more directions;

providing one or more characteristics of a relative velocity between the target and the one or more receivers based on at least one of the determined spatial elongation or the determined contraction of the one or more reflected light patterns; and

providing a magnitude of the relative velocity based on one or more portions of the rotation of the pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

9. The method of claim 8, wherein the one or more light patterns are scanned across the target to generate a two-dimensional (2D) pattern on the target and the one or more characteristics of the relative velocity includes a direction based on one or more portions of the 2D pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

10. The method of claim 8, further comprising:

scanning the one or more light patterns across the target such that a first row of a plurality of rows of pixels included in the one or more receivers is illuminated by the one or more reflected light patterns during a first time period;

employing a feedback loop between the one or more receivers and the one or more transmitters to synchronize a shutter included in the one or more receivers such that the shutter exposes the first row of the plurality of rows during the first time period;

scanning the one or more light patterns across the target such that a second row of the plurality of rows of a sensor array is illuminated by the one or more reflected light patterns during a second time period that is later than the first time period; and

employing the feedback loop to synchronize the shutter such that the shutter exposes the second row of the plurality of rows during the second time period.

11. The method of claim 8, wherein at least one of the one or more receivers includes a first sensor array sensitive to photons within a visible portion of an electromagnetic (EM) spectrum and a second sensor array sensitive to photons within a non-visible portion of the EM spectrum, wherein the first and second sensor arrays are stacked such that at least a portion of the one or more reflected patterns are transmitted through the first sensor array prior to detection by the second sensor array.

12. The method of claim 8, further comprising:

encoding one or more transmission angles of the one or more light patterns in the one or more light patterns by modulating a combination of a plurality of light sources included in the one or more transmitters, wherein the combination of the plurality of light sources generate the one or more light patterns;

decoding the one or more transmission angles based on the one or more signals; and

providing at least a position of the target based on at least the decoded one or more transmission angles.

13. A processor readable non-transitory storage media that includes instructions for sensing relative motion of a target, wherein execution of the instructions by one or more processors enable actions, comprising:

employing the one or more transmitters to transmit the one or more light patterns to the target, wherein the one or more transmitters scan the one or more light patterns across the target in one or more directions;

employing the one or more light patterns to generate a pattern on the target, wherein a rotation of the pattern is controlled by modulating the one or more light patterns;

employing the one or more receivers to detect one or more reflected light patterns from the target and generate one or more signals that correspond to the one or more reflected light patterns;

employing the one or more signals to determine at least one of a spatial elongation or contraction of the one or more reflected light patterns in the one or more directions;

providing one or more characteristics of a relative velocity between the target and the one or more receivers based on at least one of the determined spatial elongation or the determined contraction of the one or more reflected light patterns; and

providing a magnitude of the relative velocity based on one or more portions of the rotation of the pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

14. The media of claim 13, wherein the one or more light patterns are scanned across the target to generate a two-dimensional (2D) pattern on the target and the one or more characteristics of the relative velocity includes a direction based on one or more portions of the 2D pattern that are associated with one or more maximums of the spatial elongation or contraction of the one or more reflected light patterns.

15. The media of claim 13, the actions further including:

scanning the one or more light patterns across the target such that a first row of a plurality of rows of pixels included in the one or more receivers is illuminated by the one or more reflected light patterns during a first time period;

employing a feedback loop between the one or more receivers and the one or more transmitters to synchronize a shutter included in the one or more receivers such that the shutter exposes the first row of the plurality of rows during the first time period;

scanning the one or more light patterns along the target such that a second row of the plurality of rows of a sensor array is illuminated by the one or more reflected light patterns during a second time period that is later than the first time period; and

employing the feedback loop to synchronize the shutter such that the shutter exposes the second row of the plurality of rows during the second time period.

16. The media of claim 13, wherein at least one of the one or more receivers includes a first sensor array sensitive to photons within a visible portion of an electromagnetic (EM) spectrum and a second sensor array sensitive to photons within a non-visible portion of the EM spectrum, wherein the first and second sensor arrays are stacked such that at least a portion of the one or more reflected patterns are transmitted through the first sensor array prior to detection by the second sensor array.

17. The media of claim 13, wherein the actions further comprise:

encoding one or more transmission angles of the one or more light patterns in the one or more light patterns by modulating a combination of a plurality of light sources included in the one or more transmitters, wherein the combination of the plurality of light sources generate the one or more light patterns;

decoding the one or more transmission angles based on the one or more signals; and

providing at least a position of the target based on at least the decoded one or more transmission angles.