DEVICES AND METHODS FOR IDENTIFYING AN OBJECT IN AN IMAGE

• US 20190311493A1
• Filed: 04/06/2018
• Published: 10/10/2019
• Est. Priority Date: 04/06/2018
• Status: Active Grant

• Current Assignee: Elekta AB
• Original Assignee: Elekta AB
• Inventors: Hillborg, Mikael
• Granted Patent: US 10,546,389 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 2210/12 Bounding box
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/225 based on a marking or identifier characterising the area
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images

Abstract

Methods, devices, and computer-readable storage media for identifying an object in an image, the method including using a first neural network to identify an approximate position of an object of interest in an image and identifying based on the approximate position, a section of the image that includes the object of interest. A plurality of sub-images corresponding to the identified section of the image are applied to a plurality of second neural networks to determine a plurality of second positions of the object of interest. The plurality of second positions are statistically analyzed to determine an output position of the object of interest.

Claims

1. A computer-implemented method for identifying an object in an image, the method comprising:

receiving, by one or more processors, an image having one or more objects of interest;

using a first neural network to identify an approximate position of an object of interest in the received image;

identifying, by the one or more processors, based on the identified approximate position, a section of the received image that includes the object of interest;

obtaining, by the one or more processors, a plurality of sub-images corresponding to the identified section of the received image;

applying the plurality of sub-images to a plurality of second neural networks, each of the second neural networks determining a respective position of the object of interest, such that the plurality of second neural networks determine a respective plurality of second positions of the object of interest; and

statistically analyzing the plurality of second positions to determine an output position of the object of interest.

2. The method of claim 1, wherein obtaining the plurality of sub-images corresponding to the identified section of the received image includes:

creating a first sub-image by adding a first margin to the identified section of the image including the object of interest; and

creating a second sub-image by adding a second margin of a different amount from the first margin to the identified section of the image including the object of interest.

3. The method of claim 1, further comprising:

identifying, by the one or more processors, based on each determined second position in each of the sub-images a corresponding section of the sub-image that includes the object of interest; and

determining bounding rectangles matching edges of each of the identified sections.

4. The method of claim 3, further comprising:

determining average top, left, right, and bottom sides based on sides of the bounding rectangles;

constructing an average rectangle using the average sides;

calculating a center of the average rectangle;

translating the center of the average rectangle to a position in the received image; and

identifying the translated position as a position of the object of interest in the received image.

5. The method of claim 4, further comprising:

determining standard deviations of top, left, right, and bottom sides based on the sides of the bounding rectangles of the identified sections of the sub-images; and

removing sides of the bounding rectangles which are more than one standard deviation of the respective side.

6. The method of claim 1, wherein prior to receiving the image having one or more objects of interest, the method further includes:

training, by one or more processors, the first neural network to learn to identify the approximate position of the object of interest in a training image; and

training, by one or more processors, the plurality of second neural networks to learn to identify the position of the object of interest in the training image.

7. The method of claim 6, wherein the training of the first neural network to learn to identify the approximate position of the object of interest includes:

pre-processing the training image by adjusting one or more color characteristics of the image.

8. The method of claim 6, wherein training the first neural network to learn to identify the approximate position of the object of interest includes:

generating one or more images of the training image by;

cropping a section of the training image which includes the object of interest,

flipping the training image,

rotating the training image, or

adjusting color characteristics of the training image; and

applying the generated image to the first neural network to identify the position of the object of interest in the generated image.

9. The method of claim 6, wherein training the second neural network to learn to identify the position of the object of interest includes:

obtaining, by the one or more processors, a training sub-image including the position of the object of interest identified by the first neural network; and

applying the training sub-image to the second neural network to determine a position of the object of interest.

10. A device for identifying objects in an image, the device comprising:

an input interface that receives an image having one or more objects of interest;

at least one storage device configured to store the received image; and

one or more processors configured to;

use a first neural network to identify an approximate position of an object of interest;

identify, based on the identified approximate position, a section of the received image that includes the object of interest;

obtain a plurality of sub-images corresponding to the identified section of the received image;

apply the plurality of sub-images to a plurality of second neural networks, each of the second neural networks determining a respective position of the object of interest, such that the plurality of second neural networks determine a respective plurality of second positions of first object of interest; and

statistically analyze the plurality of second positions to determine an output position of the object of interest.

11. The device of claim 10, wherein the one or more processors being further configured to:

create a first sub-image by adding a first margin to the identified section of the image including the object of interest; and

create a second sub-image by adding a second margin of a different amount from the first margin to the identified section of the image including the object of interest.

12. The device of claim 10, the one or more processors being further configured to:

identify, based on each determined second position in each of the sub-images a corresponding section of the sub-image that includes the object of interest; and

determine bounding rectangles matching edges of each of the each identified sections.

13. The device of claim 12, the one or more processors being further configured to:

determine average top, left, right, and bottom sides based on sides of the bounding rectangles;

construct an average rectangle using the average sides; and

calculate center of the average rectangle; and

translate a center of the average rectangle to a position in the received image; and

identify the translated position as a position of the object of interest in the received image.

14. The device of claim 13, wherein the one or more processors being further configured to:

determine standard deviation of top, left, right, and bottom sides based on the sides of the bounding rectangles of the identified sections of the sub-images; and

remove sides of the bounding rectangles which are more than one standard deviation of the respective side.

15. The device of claim 10, wherein prior to receiving the image having one or more objects of interest the one or more processors are further configured to:

train the first neural network to learn to identify the approximate position of the object of interest in a training image; and

train the plurality of second neural networks to learn to identify the position of the object of interest in the training image.

16. The device of claim 15, wherein the one or more processors being further configured to:

pre-process the training image by adjusting one or more color characteristics of the image.

17. The device of claim 15, wherein the one or more processors being further configured to:

generate one or more images of the training image by;

cropping a section of the training image which includes the object of interest,

flipping the training image,

rotating the training image, or

adjusting color characteristics of the training image; and

apply the generated image to the first neural network to identify the position of the object of interest in the generated image.

18. The device of claim 15, wherein the one or more processors being further configured to:

obtain a training sub-image including the position of the object of interest identified by the first neural network; and

apply the training sub-image to the second neural network to determine a position of the object of interest.

19. A non-transitory computer-readable medium storing instructions that, when executable by one or more processors, cause the one or more processors to perform a method for identifying an object in an image, the method comprising:

receiving an image having one or more objects of interest;

using a first neural network to identify an approximate position of an object of interest the received image;

identifying based on the identified approximate position, a section of the received image that includes the object of interest;

obtaining a plurality of sub-images corresponding to the identified section of the received image;

applying the plurality of sub-images to a plurality of second neural networks, each of the second neural networks determining a respective position of the object of interest, such that the plurality of second neural networks determine a respective plurality of second positions of the object of interest; and

statistically analyzing the plurality of second positions to determine an output position of the object of interest.

20. The non-transitory computer-readable medium of claim 19, wherein obtaining the plurality of sub-images corresponding to the identified section of the received image includes:

creating a first sub-image by adding a first margin to the identified section of the image including the object of interest; and

creating a second sub-image by adding a second margin of a different amount from the first margin to the identified section of the image including the object of interest.

21. The non-transitory computer-readable medium of claim 19, wherein the method further comprises:

identifying, based on each determined second position in each of the sub-images a corresponding section of the sub-image that includes the object of interest; and

determining bounding rectangle matching edges of each of the identified sections.

22. The non-transitory computer-readable medium of claim 21, wherein the method further comprises:

determining average top, left, right, and bottom sides based on sides of the bounding rectangles;

constructing an average rectangle using the average sides; and

calculating center of the average rectangle; and

translating the center of the average rectangle to a position in the received image; and

identifying the translated position as a position of the object of interest in the received image.

23. The non-transitory computer-readable medium of claim 22, wherein the method further comprises:

determining standard deviation of top, left, right, and bottom sides based on the sides of the bounding rectangles of the identified sections of the sub-images; and

removing sides of the bounding rectangles which are located more than one standard deviation of the respective side.

24. The non-transitory computer-readable medium of claim 20 containing instructions that, when executable by the one or more processors, cause the one or more processors to perform a method prior to receiving the image having one or more objects of interest the one or more processors, the method comprising:

training, by one or more processors, the first neural network to learn to identify the approximate position of the object of interest in a training image; and

training, by one or more processors, the plurality of second neural networks to learn to identify the position of the object of interest in the training image.

25. The method of claim 24, wherein training of the first neural network to learn to identify the position of the object of interest includes:

pre-processing the training image by adjusting one or more color characteristics of the image.

26. The method of claim 24, wherein training the first neural network to learn to identify the position of the object of interest includes:

generating one or more images of the original image by;

cropping a section of the training image which includes the object of interest,

flipping the training image,

rotating the training image, or

adjusting color characteristics of the training image, and

applying the generated image to the first neural network to identify the position of the object of interest in the generated image.

27. The method of claim 24, wherein training the second neural network to learn to identify the position of the object of interest includes:

obtaining, by the one or more processors, a training sub-image including the position of the object of interest identified by the first neural network; and

applying the training sub-image to the second neural network to determine a position of the object of interest.