Method for on-line machine vision measurement, monitoring and control of product features during on-line manufacturing processes

• US 7,068,817 B2
• Filed: 11/07/2002
• Issued: 06/27/2006
• Est. Priority Date: 11/07/2002
• Status: Expired due to Fees

• Current Assignee: Kocam International Company Limited, Mcmaster University, Frito-Lay North America Incorporated (PepsiCo Incorporated)
• Original Assignee: Mcmaster University
• Inventors: Martin, Paul Allan, Yu, Honglu, MacGregor, John F., Bresnahan, Steven A., Bourg, Wilfred M. Jr., Haarsma, Gabe Jan
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): CARTER, AARON W
• Application Number:
  US10/289,592
• Publication Number:
  US 20040091135A1
• Time in Patent Office: 1,328 Days
• Field of Search: 382/110, 382/170, 382190-208
• US Class Current: 382/110
• CPC Class Codes: G01N 21/8851 Scan or image signal processing specially adapted therefor, e.g. for scan signal adjustment, for detecting different kinds of defects, for compensating for structures, markings, edges G01N21/8806 and G01N21/93 - G01N21/95692 take precedence; optical measurement of dimensions G01B11/00; optical scanning G02B26/10; image transformation G06T3/00; computerised image enhancement G06T5/00; image processing per se for flaw detection G06T7/0002
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06T 2207/30128 Food products
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06V 10/40 Extraction of image or video features

Abstract

A method for extracting feature information from product images using multivariate image analysis based on Principal Component Analysis (PCA) which is used to develop predictive models for feature content and distribution on the imaged product. The imaging system is used to monitor product quality variables in an on-line manufacturing environment. It may also be integrated into a closed-loop feedback control system in automated systems.

Claims

1. A method of monitoring a process producing a characterizing product, the method comprising the steps of:

sequentially capturing multivariate images of the characterizing product, each image consisting of an image array of pixel elements of measured intensity values in at least three wavelength ranges defining the dimensions for the image array;

creating a feature vector from said image array;

performing a regression method to correlate the feature vector with a characterizing feature of the product; and

,

creating an output of said characterizing feature for continuous monitoring of the process.

2. A method according to claim 1 in which the multivariate image of the characterizing product is captured in the visible spectrum.

3. A method according to claim 2 in which the pixel elements of the image array have varying intensities of the colours red, green, and blue.

4. A method according to claim 1 in which the multivariate image of the characterizing product is captured in the near infra-red spectrum.

5. A method according to claim 1 in which the feature vector consists of average values for each of the said at least three wavelength ranges.

6. A method according to claim 1 in which a multivariate statistical projection method is applied to the image array to reduce the dimensions to a low dimensional score space image data defined by a small number of score vectors t_a and said feature vector is created from said low dimensional score space image data.

7. A method according to claim 6 in which the multivariate statistical projection method is selected from the group comprising:

multi-way principal component analysis (PCA);

multi-way partial least squares (PLS);

projection pursuit;

independent component analysis.

8. A method according to claim 6 in which the multivariate statistical projection method applied to the image array reduces the dimensions of the image array to a t₁–

t₂ score space.

9. A method according to claim 8 in which the feature vector is a first loading vector p₁.

10. A method according to claim 8 in which the feature vector is obtained by dividing the t₁–

t₂ score space into a number of blocks and summing the intensity values within each block.

11. A method according to claim 8 in which the feature vector is created, comprising the steps of:

applying a multivariate statistical projection method to an image array for a control process to reduce the dimensions to a low dimensional score space having a significant loading vector t_s;

projecting the t₁–

t₂ score space along a direction defined by said significant loading vector t_s to create a projection vector V; and

,

creating a histogram from said projection vector V to define the feature vector.

12. A method according to claim 8 in which the feature vector is created comprising the steps of:

applying a multivariate statistical projection method to an image array for a control process to reduce the dimensions to a low dimensional score space having a significant loading vector t_s;

projecting the t₁–

t₂ score space along a direction defined by said significant loading vector t_s to create a projection vector V; and

,

creating a cumulative histogram from said projection vector V to define the feature vector.

13. A method according to claim 8 in which said feature vector is created comprising the steps of:

calculating covariance matrices COV₁ and COV₂ between at least one histogram elements from the t₁–

t₂ score space image data for two respective features Z_{1 and Z2};

calculating an angle matrix θ

from said covariance matrices COV₁ and COV₂;

superimposing the t₁–

t₂ score space image data on said angle matrix θ

; and

,

calculating a cumulative histogram to define the feature vector.

14. A method according to claim 8 in which said feature vector is created comprising the steps of:

calculating covariance matrices COV₃ and COV₄ between at least one histogram elements from a t₁–

t₂ score space image data for two respective features Z₁ and Z₂ in a control process;

adding covariance matrices COV₃ and COV₄ and displaying the sum as a colour coded image sum display in a first t₁–

t₂ score space;

selecting a mask to inscribe an area A in said sum display in said first t₁–

t₂ space corresponding to a product region;

projecting the multivariate images to said first t₁–

t₂ score space and removing pixels which lie outside said area A to create pure product images;

applying a multivariate statistical projection method to the pure product images to reduce the dimensions of the pure product images to a second t₁–

t₂ score space;

calculating covariance matrices COV₅ and COV₆ between histogram elements from the second t₁–

t₂ score space for said two respective features Z₁ and Z₂;

calculating an angle matrix θ

from said covariance matrices COV₅ and COV₆;

superimposing the second t₁–

t₂ score space image data on said angle matrix θ

; and

,

calculating a cumulative histogram to define the feature vector.

15. A method according to claim 1 in which the regression method is selected from the group comprising principal component regression (PCR);

partial least squares (PLS);

multivariate linear regression (MLR);

artificial neural networks (ANN).

16. A method according to claim 1 in which the output is displayed as a colour coded image for visual monitoring of the process.

17. A method of monitoring a process in which a seasoning is added to a food product under variable operating conditions, comprising the steps of:

shining a light onto the food product;

sequentially capturing multivariate images of the food product in the visible spectrum, each image consisting of an image array having a plurality of pixel elements, each pixel element having intensity values of the colours red, green, and blue defining three dimensions for the image array;

applying a multi-way principal component analysis to the image array to reduce the dimensions of the image array to a t₁–

t₂ score space;

creating a feature vector from said t₁–

t₂ score space;

performing a regression method to correlate the feature vector with a coating property on the food product; and

,

creating an output of the coating property for continuous monitoring and feed back control of the operating conditions of the process.

18. A method of monitoring a process in which a seasoning is added to a snack food product under variable operating conditions, the method comprising the steps of:

shining a light onto a food product;

sequentially capturing multivariate images of the food product in the visible spectrum, each image consisting of an image array having a plurality of pixel elements, each pixel element having intensity values of the colours red, green, and blue defining three dimensions for the image array;

applying a multi-way principal component analysis to the image array to reduce the dimensions of the image array to a first t₁–

t₂ score space;

selecting a mask to inscribe an area A in said first t₁–

t₂ score space which excludes background from the image array;

projecting the multivariate images to said first t₁–

t₂ score space and removing pixels which lie outside said area A to create pure product images;

applying a multi-way principal component analysis to the pure product images to reduce the dimensions of the pure product images to a second t₁–

t₂ score space;

creating a feature vector from said second t₁–

t₂ score space;

performing a regression method to correlate the feature vector with a coating property of the food product; and

,

creating an output of the coating property for continuous monitoring and feed back control of the operating conditions of the process.

19. A method according to claim 18 in which the regression method is performed on the feature vector and selected process measurements to create the output of the coating property.

20. A method according to claim 18 in which the coating property is selected from the group comprising predicted seasoning concentration, seasoning distribution plot, seasoning distribution variance and color-coded seasoning concentration image.

21. A method of monitoring a process in which a fuel is consumed to produce a flame under changing operating conditions, comprising the following steps of:

sequentially capturing multivariate images of the flame in the visible spectrum, each image consisting of an image array having a plurality of pixel elements, each pixel element having intensity values of the colours red, green, and blue defining three dimensions for the image array;

applying a multi-way principal component analysis to the image array to reduce the dimensions of the image array to a t₁–

t₂ score space;

selecting a mask to inscribe an area A in said t₁–

t₂ space corresponding to a flame luminous region;

calculating statistical features of said flame luminous region to create a feature vector;

performing a regression method to correlate the feature vector and at least one selected process measurements with a characterizing feature of the process; and

,

creating an output of said characterizing feature for continuous monitoring of the operating conditions of the process.

22. A method according to claim 21 in which said statistical feature is selected from the group containing flame area, flame brightness, flame luminous uniformity, average brightness of flame region, number of colours appearing in the flame region.

23. A method according to claim 21 in which said regression method is selected from principal component regression (PCR), partial least squares (PLS), multivariate linear regression (MLR) and artificial neural network (ANN).

24. A method according to claim 21 in which the characterizing feature is selected from the group comprising steam flow, concentration of pollutants and reactants in the process.

25. A method for product feature monitoring and quality control in an on-line environment comprising the steps of:

exposing a food product to an electromagnetic radiation source;

imaging at least one product feature of the food product;

extracting background features from the product feature image;

creating an image score space summarizing information in the product feature image;

correlating the product feature image score space to a training sample score space by statistical regression; and

,

predicting the occurrence of the product feature on the food product.

26. The method of claim 25 wherein the predicted food product feature is at least one of the quantity, characteristic, level, or degree of the product feature on the food product.

27. The method of claim 25 wherein the product feature is the coating coverage on the food product.

28. The method of claim 25 wherein the product feature is one of toast points, seasoning concentration, seasoning coverage, ingredient concentration, surface blisters, surface visual defects, or texture of the food product.

29. The method of claim 25 wherein the prediction of the occurrence of the product feature on the product is calculated within subsections of the product feature image.

30. The method of claim 29 wherein a product feature variance is calculated based on at least one of the values generated from the prediction phase within the subsections of the image of the product feature and the product feature variance across the image is reported for monitoring and controlling online product feature variance.

31. The method of claim 25 wherein the occurrence prediction is for the presence or absence of the product feature.

32. The method of claim 25 wherein the product feature is imaged by at least one of visual light spectrum imaging, infrared spectrum imaging, ultraviolet spectrum imaging, audible sound wave detection, inaudible sound wave detection, mass spectroscopy and chromatography.

33. The method of claim 25 wherein the product feature is a food product or an ingredient of a food product.

34. The method of claim 25 wherein an image acquisition apparatus, a computer processor and computer program for image processing are utilized to reduce the methodology to on-line practice for a product manufacturing process.

35. The method of claim 25 further comprising automatically adjusting an on-line manufacturing process with closed loop control.

36. The method of claim 25 further comprising providing quality control images with visualization software to provide process monitoring, control, alarms and defective product aborting for a product manufacturing process.

37. The method of claim 25 wherein the food product is either bedded or not bedded.

38. The method of claim 25 further comprising the step of:

developing feature correlation models through statistical regression between a predicted value for a product coating coverage, the amount of coating, the product density, surface area, and product depth or distance from the imaging system.

39. The method of claim 38 wherein at least one of the correlation models are used to predict or augment a measurement value of at least one product feature variable using measurements of at least one other feature variable.

40. The method of claim 25 wherein the predicted occurrence of a product feature is the product bulk density and specific bulk density.

41. The method of claim 25 wherein the correlation of the product features includes at least one of product appearance, seasoning concentration, seasoning coverage, product density, bedded product depth and product distance from the imaging apparatus.

42. The method of claim 41 wherein a specific bed depth model is used to predict the bed depth of the product.

43. The method of claim 42 wherein the predicted specific bed depth is used to augment the measurement of other product features which are affected by variable bed depth and distance from the imaging device.