COMPUTER VISION SYSTEM AND METHOD

• US 20200134772A1
• Filed: 10/31/2018
• Published: 04/30/2020
• Est. Priority Date: 10/31/2018
• Status: Active Grant

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: POUDEL KARMATHA, Rudra Prasad, BONDE, Ujwal, LIWICKI, Stephan, ZACH, Christopher
• Granted Patent: US 10,769,744 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

An image processing method for segmenting an image, the method comprising:

• • receiving first image;
  • producing a second image from said first image, wherein said second image is a lower resolution representation of said first image;
  • processing said first image with a first processing stage to produce a first feature map;
  • processing said second image with a second processing stage to produce a second feature map; and
  • combining the first feature map with the second feature map to produce a semantic segmented image;
  • wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution; the number of layers in the first neural network being smaller than the number of layers in the second neural network.

Claims

1. An image processing method for segmenting an image, the method comprising:

receiving first image;

producing a second image from said first image, wherein said second image is a lower resolution representation of said first image;

processing said first image with a first processing stage to produce a first feature map;

processing said second image with a second processing stage to produce a second feature map; and

combining the first feature map with the second feature map to produce a semantic segmented image;

wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution;

the number of layers in the first neural network being smaller than the number of layers in the second neural network.

2. An image processing method according to claim 1, wherein the first feature map and the second feature map are combined at just one stage.

3. An image processing method according to claim 2, wherein the first feature map and the second feature map are combined by adding.

4. An image processing method according to claim 2, wherein the first feature map and the second feature map are combined at a fusion stage, wherein said fusion stage comprises upsampling the second feature map and adding the first feature map to the second feature map wherein adding comprises adding corresponding values of the upsampled second feature map and the first feature map.

5. An image processing method according to claim 4, wherein upsampled second feature map is subjected to depthwise convolution prior to adding to the first feature map, wherein the depthwise convolution is performed with a dilation factor that is greater than 1.

6. An image processing method according to claim 5, wherein upsampled second feature map has been subjected to depthwise convolution with a dilation factor that is greater than 1 and the first feature map are subjected to two dimensional convolution prior to adding.

7. An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are depthwise convolution modules.

8. An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are bottleneck architecture modules.

9. An image processing method according to claim 1, wherein separable convolution modules in the second processing stage are bottleneck residual architecture modules.

10. An image processing method according to claim 1, wherein the second image is subjected to standard convolution prior to be processed by said separable convolution modules.

11. An image processing method, according to claim 9, wherein there are a plurality of residual bottleneck architecture modules.

12. An image processing method according to claim 11, wherein a final standard convolution module is provided after the residual bottleneck architecture modules.

13. An image processing method according to claim 12, wherein a pyramid pooling module is provided between the final standard convolution module and the residual bottleneck architecture modules.

14. An image processing method according to claim 1, wherein the first processing stage comprises a plurality of depthwise separable convolution modules.

20. A non-transitory carrier medium carrying computer readable instructions adapted to cause a computer to perform the method of claim 1.

15. A method of training a model, said model for segmenting an image, the model comprising:

a first neural network having a plurality of layers comprising at least one separable convolution module configured to perform separable convolutions;

a second neural network with a plurality of layers, comprising at least one separable convolution module configured to perform separable convolutions;

the number of layers in the first neural network being smaller than the number of layers in the second neural network,

the first neural network being configured to process an image at a first resolution and the second neural network being configured to process the same image at a lower resolution,

the feature map of the first and second neural networks being combined by addition at a single stage;

the training method comprising;

providing training data, the training data comprising images and semantic segmented information concerning said images;

training said model using said images as the input and the semantic segmented info as the output, wherein the two stages are trained together.

16. A method according to claim 15, further comprising increasing the number of filters during training, and reducing their number to discard filters of lower importance.

17. A method according to claim 15, further comprising adapting the model during training to add a second output on said second processing stage, the method further comprising training using the images as input and determining the loss by comparing with both the semantic segmented information at both the output and at the second output and updating the weights during training by using the determined losses from both outputs.

18. An image processing system for segmenting an image, the system comprising:

an interface and a processor,

said interface having an image input and being adapted to receive a first image, said processor being adapted to;

produce a second image from said first image, wherein said second image is a lower resolution representation of said first image;

process said first image with a first processing stage to produce a first feature map;

process said second image with a second processing stage to produce a second feature map; and

combine the first feature map with the second feature map to produce a semantic segmented image;

wherein the first processing stage comprises a first neural network comprising at least one separable convolution module configured to perform separable convolution and said second processing stage comprises a second neural network comprising at least one separable convolution module configured to perform separable convolution;

the number of layers in the first neural network being smaller than the number of layers in the second neural network.

19. A detection system for a vehicle, said detection system, the detection system comprising the image processing system of claim 18, being adapted to receive an image and determine objects from said image by segmenting said image.