Image processing system, image processing apparatus, workpiece pickup method, and workpiece pickup program

• US 10,407,250 B2
• Filed: 10/19/2017
• Issued: 09/10/2019
• Est. Priority Date: 01/10/2017
• Status: Active Grant

• Current Assignee: Omron Corporation
• Original Assignee: Omron Corporation
• Inventors: Iwai, Yosuke, Iida, Toyoo, Shimamura, Junji
• Primary Examiner(s): Logan, Kyle O
• Application Number:
  US15/787,788
• Publication Number:
  US 20180194573A1
• Time in Patent Office: 691 Days
• CPC Class Codes: B25J 9/0093 co-operating with conveyor means
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 15/12 with two or more endless belts
• CPC Class Codes: B65G 47/32 Applications of transfer devices such devices per se B65G47/74
• CPC Class Codes: G05B 19/4182 manipulators and conveyor only
• CPC Class Codes: G05B 19/4189 characterised by the transport system
• CPC Class Codes: G05B 2219/39102 Manipulator cooperating with conveyor
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/75 involving models

Abstract

An imaging unit has its imaging field of view containing the conveyor, and images a subject in the imaging field. The imaging unit generates a luminance image representing the subject and a range image including height information indicating a height of at least one point on the subject from the conveyor by imaging the subject. A measurement unit measures a position of each workpiece in the luminance image. An obtaining unit obtains an overlap order of the workpieces based on the height information in the range image corresponding to the position of each workpiece in the luminance image. A determination unit determines the workpiece pickup order to allow a workpiece having a highest place in the overlap order to be picked up with a higher priority than other workpieces.

Claims

1. An image processing system for determining a workpiece pickup order in which overlapping workpieces being transported on a conveyor are picked up by a robot, the system comprising:

an imaging unit comprising a projector and arranged to have an imaging field of view containing the conveyor, and configured to;

image a subject comprising a plurality of the overlapping workpieces included in the imaging field of view; and

generate, based on linear light projected from the projector and reflected from the subject and natural light reflected from the subject, a luminance image representing the subject and a range image comprising height information indicating a height of at least one point on the subject; and

a processor configured with a program to perform operations comprising;

operation as a measurement unit configured to measure a position of each of the plurality of workpieces in the subject, based on the luminance image;

operation as an obtaining unit configured to obtain an overlap order in which the plurality of workpieces overlap one another based on the height information in the range image corresponding to the position of each of the plurality of workpieces in the luminance image; and

operation as a determination unit configured to;

determine the workpiece pickup order for the plurality of workpieces such that a workpiece having a highest place in the overlap order is picked up with a higher priority than other workpieces; and

determine, in response to the luminance image comprising at least one workpiece that does not overlap another workpiece and a plurality of overlapping workpieces, the workpiece pickup order for the at least one workpiece that does not overlap another workpiece and a workpiece having a highest place in the overlap order among the plurality of overlapping workpieces such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.

2. The image processing system according to claim 1, wherein

the projector is configured to project, onto the plurality of workpieces transported on the conveyor, the linear light such that the linear light diverges in a direction orthogonal to a transportation direction of the plurality of workpieces, and

the imaging unit comprises a photoreceiver configured to successively receive the reflected linear light from the plurality of workpieces transported on the conveyor to generate the range image based on a position of the received reflected light upon every reception of light, and receive reflected light from the subject included in the imaging field of view to generate the luminance image.

3. The image processing system according to claim 1, wherein

the projector is configured to project, onto the plurality of workpieces transported on the conveyor, the linear light such that the linear light diverges in a direction orthogonal to a transportation direction of the plurality of workpieces, and

the imaging unit comprises:

a first photoreceiver configured to successively receive the reflected linear light from the plurality of workpieces transported on the conveyor to generate the range image based on a position of the received reflected light upon every reception of light; and

a second photoreceiver configured to receive the natural light reflected from the subject included in the imaging field of view to generate the luminance image.

4. The image processing system according to claim 1, wherein the processor is configured with the program such that operation as the determination unit comprises operation as the determination unit that determines the workpiece pickup order for such that a workpiece having a higher place in the overlap order is picked up earlier than a workpiece having a lower place in the overlap order.

5. The image processing system according to claim 4, wherein

the projector is configured to project, onto the plurality of workpieces transported on the conveyor, the linear light such that the linear light diverges in a direction orthogonal to a transportation direction of the plurality of workpieces, and

the imaging unit comprises a photoreceiver configured to successively receive the reflected linear light from the plurality of workpieces transported on the conveyor to generate the range image based on a position of the received reflected light upon every reception of light, and receive reflected light from the subject included in the imaging field of view to generate the luminance image.

6. The image processing system according to claim 4, wherein the processor is configured with the program such that:

operation as the obtaining unit comprises operation as the obtaining unit that, in response to the luminance image comprising a plurality of workpiece groups, each of the plurality of workpiece groups comprising a plurality of overlapping workpieces, obtains the overlap order for each workpiece group; and

operation as the determination unit comprises operation as the determination unit that determines the workpiece pickup order for a workpiece having a highest place in the overlap order in each workpiece group such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.

7. The image processing system according to claim 6, wherein

the projector is configured to project, onto the plurality of workpieces transported on the conveyor, the linear light such that the linear light diverges in a direction orthogonal to a transportation direction of the plurality of workpieces, and

the imaging unit comprises a photoreceiver configured to successively receive the reflected linear light from the plurality of workpieces transported on the conveyor to generate the range image based on a position of the received reflected light upon every reception of light, and receive reflected light from the subject included in the imaging field of view to generate the luminance image.

8. The image processing system according to claim 1, wherein the processor is configured with the program to perform operations such that:

operation as the obtaining unit comprises operation as the obtaining unit that, in response to the luminance image comprising a plurality of workpiece groups, each of the plurality of workpiece groups comprising a plurality of overlapping workpieces, obtains the overlap order for each workpiece group; and

operation as the determination unit comprises operation as the determination unit that determines the workpiece pickup order for a workpiece having a highest place in the overlap order in each workpiece group such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.

9. The image processing system according to claim 3, wherein

the projector is configured to project, onto the plurality of workpieces transported on the conveyor, the linear light such that the linear light diverges in a direction orthogonal to a transportation direction of the plurality of workpieces, and

the imaging unit comprises a photoreceiver configured to successively receive the reflected light of the linear light from the plurality of workpieces transported on the conveyor to generate the range image based on a position of the received reflected light upon every reception of light, and receive reflected light from the subject included in the imaging field of view to generate the luminance image.

10. An image processing apparatus for determining a workpiece pickup order in which overlapping workpieces being transported on a conveyor are picked up by a robot, the apparatus comprising:

an imaging unit comprising a projector and arranged to have an imaging field of view containing the conveyor, and configured to;

image a subject comprising a plurality of the overlapping workpieces and included in the imaging field of view; and

generate, based on linear light projected from the projector and reflected from the subject and natural light reflected from the subject, a luminance image representing the subject and a range image comprising height information indicating a height of at least one point on the subject; and

a processor configured with a program to perform operations comprising;

operation as a measurement unit configured to measure a position of each of the plurality of workpieces in the subject, based on the luminance image;

operation as an obtaining unit configured to obtain an overlap order in which the plurality of workpieces overlap one another based on the height information in the range image corresponding to the position of each of the plurality of workpieces in the luminance image; and

operation as a determination unit configured to;

determine the workpiece pickup order for the plurality of workpieces such that a workpiece having a highest place in the overlap order is picked up with a higher priority than other workpieces; and

determine, in response to the luminance image comprising at least one workpiece that does not overlap another workpiece and a plurality of overlapping workpieces, the workpiece pickup order for the at least one workpiece that does not overlap another workpiece and a workpiece having a highest place in the overlap order among the plurality of overlapping workpieces such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.

11. A pickup method for determining a workpiece pickup order in which overlapping workpieces being transported on a conveyor are picked up by a robot, the method comprising:

imaging a subject comprising a plurality of the overlapping workpieces included in an imaging field of view using an imaging unit comprising a projector and arranged to have the imaging field of view containing the conveyor by generating, based on linear light projected from the projector and reflected from the subject and natural light reflected from the subject, a luminance image representing the subject and a range image comprising height information indicating a height of at least one point on the subject from the conveyor by imaging the subject;

measuring a position of each of the plurality of workpieces in the subject, based on the luminance image;

obtaining an overlap order in which the plurality of workpieces overlap one another based on the height information in the range image corresponding to the position of each of the plurality of workpieces in the luminance image;

determining the workpiece pickup order for the plurality of workpieces such that a workpiece having a highest place in the overlap order is picked up with a higher priority than other workpieces; and

determining, in response to the luminance image comprising at least one workpiece that does not overlap another workpiece and a plurality of overlapping workpieces, the workpiece pickup order for the at least one workpiece that does not overlap another workpiece and a workpiece having a highest place in the overlap order among the plurality of overlapping workpieces such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.

12. A non-transitory computer-readable recording medium storing a pickup program for determining a workpiece pickup order in which overlapping workpieces being transported on a conveyor are picked up by a robot, the program causing a computer to perform operations comprising:

imaging a subject comprising a plurality of overlapping workpieces included in an imaging field of view using an imaging unit comprising a projector and arranged to have the imaging field of view containing the conveyor by generating, based on linear light projected from the projector and reflected from the subject and natural light reflected from the subject, a luminance image representing the subject and a range image comprising height information indicating a height of at least one point on the subject from the conveyor by imaging the subject;

measuring a position of each of the plurality of workpieces in the subject, based on the luminance image;

obtaining an overlap order in which the plurality of workpieces overlap one another based on the height information in the range image corresponding to the position of each of the plurality of workpieces in the luminance image;

determining the workpiece pickup order for the plurality of workpieces such that a workpiece having a highest place in the overlap order is picked up with a higher priority than other workpieces; and

determining, in response to the luminance image comprising at least one workpiece that does not overlap another workpiece and a plurality of overlapping workpieces, the workpiece pickup order for the at least one workpiece that does not overlap another workpiece and a workpiece having a highest place in the overlap order among the plurality of overlapping workpieces such that a workpiece more downstream in a transport path on the conveyor is picked up with a higher priority than other workpieces.