IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD

• US 20200134801A1
• Filed: 10/23/2019
• Published: 04/30/2020
• Est. Priority Date: 10/30/2018
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Moteki, Atsunori, Yoshitake, Toshiyuki, Ishikawa, Yu, Watanabe, Yasuto
• Granted Patent: US 11,176,661 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/0006 using a design-rule based approach
• CPC Class Codes: G06T 7/001 using an image reference approach
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/457 by analysing connectivity, e.g. edge linking, connected component analysis or slices
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 2201/06 Recognition of objects for industrial automation
• CPC Class Codes: G06V 2201/12 Acquisition of 3D measurements of objects

Abstract

A memory stores shape information representing a shape of an object. A processor detects a plurality of feature lines from an image of an object that has been captured by an image capturing device in a manufacturing process of manufacturing the object, and produces a plurality of combinations by correlating each of a plurality of line segments in the shape information and each of the plurality of feature lines with each other. Next, the processor performs classification of each of the combinations into an identified combination that has a certain correlation result or an unidentified combination that has an uncertain correlation result. The processor also changes a result of the classification for the identified combination and the unidentified combination according to a reliability of each of the plurality of combinations and determines a degree of progress of the manufacturing process by using the identified combination included in the changed result.

Claims

1. An image processing apparatus comprising:

a memory that stores shape information representing a shape of an object;

a processor coupled to the memory and the processor configured to;

detect a plurality of feature lines from an image of the object that has been captured by an image capturing device in a manufacturing process of manufacturing the object;

produce a plurality of combinations by correlating each of a plurality of line segments included in the shape information and each of the plurality of feature lines with each other;

perform classification of each of the plurality of combinations into an identified combination that has a certain correlation result or an unidentified combination that has an uncertain correlation result;

changes a result of the classification for the identified combination and the unidentified combination according to a reliability of each of the plurality of combinations; and

determine a degree of progress of the manufacturing process by using the identified combination included in the changed result of the classification.

2. The image processing apparatus according to claim 1, wherein

the processor

produces a plurality of projection lines by projecting the plurality of line segments onto the image,

determines an error between a position of a projection line corresponding to a line segment included in each of the plurality of combinations and a position of a feature line included in each of the plurality of combinations,

classifies, into the identified combination, a combination that has an error less than a threshold from among the plurality of combinations,

extracts a combination that has an error greater than the threshold from among the plurality of combinations, and

classifies, according to a state of connection between a line segment included in an extracted combination and a line segment included in a different combination, the extracted combination into the identified combination or the unidentified combination.

3. The image processing apparatus according to claim 2, wherein

the processor

classifies, according to the state of connection between the line segment included in the extracted combination and the line segment included in the different combination, the extracted combination into a defective portion of the shape of the object, an incorrectly correlated combination, or the unidentified combination, and

defines, as the identified combination, a combination classified into the defective portion of the shape of the object or the incorrectly correlated combination.

4. The image processing apparatus according to claim 1, wherein

the processor

performs a classification process a plurality of times in the manufacturing process, the classification process being a process of classifying a combination obtained by correlating each of the plurality of line segments included in the shape information and each of the plurality of feature lines detected from the image of the object with each other, and

determines the reliability according to a first number of times each of the plurality of combinations has been classified into the identified combination and a second number of times each of the plurality of combinations has been classified into the unidentified combination.

5. The image processing apparatus according to claim 1, wherein:

the image of the object has been captured at a first position; and

the processor

detects a plurality of feature lines from a different image of the object that has been captured at a second position different from the first position,

determines a presence range in which a feature line that corresponds to a line segment included in each of the plurality of combinations is present within the different image, and

determines the reliability according to whether the feature line that corresponds to the line segment included in each of the plurality of combinations has been detected from the presence range.

6. The image processing apparatus according to claim 1, wherein:

the object includes a plurality of parts; and

the processor

specifies a part among the plurality of parts to which a line segment included in each of the plurality of combinations belongs, and

determines the reliability according to a number of identified combinations that include a line segment belonging to the specified part.

7. The image processing apparatus according to claim 1, wherein:

the object includes a plurality of parts each having a rank that depends on an assembling order; and

the processor

determines a provisional degree of progress by using the identified combination included in the result of the classification before changing the result of the classification,

specifies a part group that includes a part within a range from an initial rank to a rank that corresponds to the provisional degree of progress, and

determines the reliability according to whether a line segment included in each of the plurality of combinations belongs to the part group.

8. The image processing apparatus according to claim 1, wherein

the processor determines the degree of progress of the manufacturing process according to a ratio of a total of lengths of line segments included in identified combinations included in the changed result of the classification to a total of lengths of line segments included in the plurality of combinations.

15. The image processing method according to claim 3, wherein

the object includes a plurality of parts each having a rank that depends on an assembling order, and

the image processing method further comprises:

determining a provisional degree of progress by using the identified combination included in the result of the classification before changing the result of the classification;

specifying a part group that includes a part within a range from an initial rank to a rank that corresponds to the provisional degree of progress; and

determining the reliability according to whether a line segment included in each of the plurality of combinations belongs to the part group.

9. An image processing method comprising:

detecting, by a processor, a plurality of feature lines from an image of an object that has been captured by an image capturing device in a manufacturing process of manufacturing the object;

producing, by the processor, a plurality of combinations by correlating each of a plurality of line segments included in shape Information representing a shape of the object and each of the plurality of feature lines with each other;

performing, by the processor, classification of each of the plurality of combinations into an identified combination that has a certain correlation result or an unidentified combination that has an uncertain correlation result;

changing, by the processor, a result of the classification for the identified combination and the unidentified combination according to a reliability of each of the plurality of combinations; and

determining, by the processor, a degree of progress of the manufacturing process by using the identified combination included in the changed result of the classification.

10. The image processing method according to claim 9, wherein

the performing classification of each of the plurality of combinations

produces a plurality of projection lines by projecting the plurality of line segments onto the image,

determines an error between a position of a projection line corresponding to a line segment included in each of the plurality of combinations and a position of a feature line included in each of the plurality of combinations,

classifies, into the identified combination, a combination that has an error less than a threshold from among the plurality of combinations,

extracts a combination that has an error greater than the threshold from among the plurality of combinations, and

classifies, according to a state of connection between a line segment included in an extracted combination and a line segment included in a different combination, the extracted combination into the identified combination or the unidentified combination.

11. The image processing method according to claim 10, wherein

the performing classification of each of the plurality of combinations

classifies, according to the state of connection between the line segment included in the extracted combination and the line segment included in the different combination, the extracted combination into a defective portion of the shape of the object, an incorrectly correlated combination, or the unidentified combination, and

defines, as the identified combination, a combination classified into the defective portion of the shape of the object or the incorrectly correlated combination.

12. The image processing method according to claim 9, wherein

the performing classification of each of the plurality of combinations performs a classification process a plurality of times in the manufacturing process, the classification process being a process of classifying a combination obtained by correlating each of the plurality of line segments included in the shape information and each of the plurality of feature lines detected from the image of the object with each other, and

the image processing method further comprises determining the reliability according to a first number of times each of the plurality of combinations has been classified into the identified combination and a second number of times each of the plurality of combinations has been classified into the unidentified combination.

13. The image processing method according to claim 9, wherein

the image of the object has been captured at a first position, and

the image processing method further comprises:

detecting a plurality of feature lines from a different image of the object that has been captured at a second position different from the first position; and

determining a presence range in which a feature line that corresponds to a line segment included in each of the plurality of combinations is present within the different image; and

determining the reliability according to whether the feature line that corresponds to the line segment included in each of the plurality of combinations has been detected from the presence range.

14. The image processing method according to claim 9, wherein

the object includes a plurality of parts, and

the image processing method further comprises:

specifying a part among the plurality of parts to which a line segment included in each of the plurality of combinations belongs; and

determining the reliability according to a number of identified combinations that include a line segment belonging to the specified part.