Method of volume-panorama imaging processing

• US 7,565,019 B2
• Filed: 12/22/2005
• Issued: 07/21/2009
• Est. Priority Date: 03/29/2005
• Status: Active Grant

• Current Assignee: Shenzhen Mindray Bio-Medical Electronics Co., Ltd. (Excelsior Union Ltd.)
• Original Assignee: Shenzhen Mindray Bio-Medical Electronics Co., Ltd. (Excelsior Union Ltd.)
• Inventors: Yao, Bin, Ni, Dong, Dong, Jian
• Primary Examiner(s): Bali; Vikkram
• Assistant Examiner(s): Bitar; Nancy
• Application Number:
  US11/316,440
• Publication Number:
  US 20060239571A1
• Time in Patent Office: 1,307 Days
• Field of Search: 348/473, 348/474, 348/699, 348/700, 600/443, 600/447, 600454-456, 600/458, 382/236, 382/239, 382/173, 382/128, 382/260, 382/190, 382/243, 382/255, 382/268, 375/240.21
• US Class Current: 382/236
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image

Abstract

The present invention discloses a method of volume-panorama imaging processing, which generates a volume-panorama image by subsequently splicing respective image frames from an image sequence obtained in a real-time way or stored in a medium based upon the fact that the immediately adjacent image frames have the largest correlation. The method comprises the steps of: reading the image sequence, and firstly initializing an aligned image and a spliced image; dividing the i-th image frame F_i into a plurality of sub-regions; calculating a motion vector of the i-t image frame with respect to the aligned image; fitting the motion vector to calculate a transform coefficient; splicing the F_i to the current spliced image based upon the transform coefficient, and updating the aligned image; entering into a self-adaptive selection of a next image frame until the end of the splicing; and outputting the current spliced image as a resultant image. Additionally, when the image F_i is spliced, double-filtering architecture of selecting characteristic points through a filtering and selecting the valid motion vector of the selected characteristic points through the other filtering to reduce an alignment error may be adopted. According to the present invention, the volume-panorama imaging can be done quickly and accurately so that the reliability of images particularly meets a very high requirement of the ultrasonic iatric diagnose.

Claims

1. A method of panoramic image processing, for generating an object image from an image sequence obtained in a real-time way or stored in a medium, in which a serial number i of an image is 1, 2, 3, . . . , and Δ

is an interval for picking up the image sequence, comprising the steps of;

a. obtaining the image sequence from an ultrasonic imaging apparatus comprising a probe for transmitting an ultrasonic wave to a detected organism and receiving an ultrasonic echo reflected from the detected organism, and initializing an aligned image and a spliced image as a first image frame;

b. dividing an i-th image frame F;

into a plurality of sub-regions, and selecting valid sub-regions from the sub-regions of the i-th image frame F;

through a first filtering;

c. calculating motion vectors for the valid sub-regions of the i-th image frame with respect to a current aligned image;

d. adaptively adjusting the interval Δ

within a value range of Δ

based on a translation amount of the i-th image frame Fi being within a controllable range

e. calculating a transform coefficient based upon a fitting of the motion vectors;

f. splicing the i-th image frame F to a current spliced image based upon the transform coefficient, and configuring or splicing the current aligned image with the i-th image frame Fi; and

g. displaying the current spliced image as a resultant image on a display device.

2. The method according to claim 1 wherein the controllable range of the translation amount is set in a range from 1 to 100 pixels.

3. The method according to claim 1, wherein the step of c involves performing the steps of:

c1. selecting a plurality of characteristic points on the i-th image frame F_i;

c2. searching the current aligned image for a corresponding point of each characteristic point; and

c3. calculating the motion vector of each point pair constituted by the characteristic point and the corresponding point.

4. The method according to claim 3, wherein the step of b involves performing the steps of:

1) determining a template on the i-th image frame F_i for a selected one of the characteristic points which is a region that includes a set of adjacent points centering on the selected characteristic point;

2) determining a searched region on the current aligned image; and

3) determining a corresponding point of the selected characteristic point within the searched region through a similarity calculation, which is a central point of an adjacent region with a same size as the template and with the largest similarity.

5. The method according to claim 4, wherein the searched region in the step of 2) is defined through a prediction:

$\left[x+\mathrm{pre\_offx}+\mathrm{XLft},x+\mathrm{pre\_offx}+\mathrm{XRgt}\right]*\left[y+\mathrm{pre\_offy}-\mathrm{Dy},y+\mathrm{pre\_offy}+\mathrm{Dy}\right];<br><br>$ $\mathrm{XLft}=\{\begin{array}{cc}-\mathrm{\mu <br><br>}& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br><br><br>\mathrm{true}\\ -v& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{false}\end{array}<br><br><br><br>\mathrm{XRgt}=\{\begin{array}{cc}v& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br><br><br>\mathrm{is}<br><br><br><br><br><br>\mathrm{true}\\ \mathrm{\mu <br><br>}& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{false}\end{array}$

where, (x, y) are coordinates of the selected characteristic point, (pre_offx, pre_offy) is an offset vector of the previous image frame, ScanDirection represents a general direction for splicing the images, true represents that the splicing is performed toward the positive direction of the X coordinate axis, and false represents that the splicing is performed toward the negative direction of the X coordinate axis;

Dy =γ

, γ

is a constant, and μ

<

v, each of which is a predictive value.

6. The method according to claim 4, wherein the i-th image frame F_i, which is floating with respect to the current aligned image, is rotated, before the step of 1), by an angle which is defined through a prediction:

$\mathrm{\sigma <br><br>}=\{\begin{array}{cc}\mathrm{\alpha <br><br>}& \mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}<<br><br>\mathrm{\varepsilon <br><br>}\\ \mathrm{\alpha <br><br>}+\left(\mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}\right)& \mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}\ge <br><br>\mathrm{\varepsilon <br><br>}\end{array}$

where, α

is the rotation angle of the previous floating image, β

is the rotation angle of the further previous floating image, and ε

is a factor of error.

7. The method according to claim 3, further comprising a step of:

c4. filtering the motion vectors to select valid motion vectors.

8. The method according to claim 1, wherein:

if an offset of the calculated transform coefficient is abnormal, or a number of valid motion vectors selected through a second filtering is less than a preset number, a current image frame is not subject to a subsequent step of the splicing;

the processing of the current image frame is returned to an initial status and the interval Δ

for picking up the image sequence is adjusted based on the processing of another frame; and

when there occurs a case continuously several times in which the current image frame is not subject to the subsequent splicing, the current spliced image is outputted as the resultant image and an information of interruption is displayed on the display device.

9. The method according to claim 7, wherein:

if an offset of the calculated transform coefficient is abnormal, or a number of the valid motion vectors selected through the filtering is less than a preset number, a current image frame is not subject to a subsequent step of the splicing;

the processing of the current image frame is returned to an initial status and the interval Δ

for picking up the image sequence is adjusted based on the processing of another frame; and

when there occurs a case continuously several times in which the current image frame is not subject to the subsequent splicing, the current spliced image is outputted as the resultant image and an information of interruption is displayed on the display device.

10. The method according to claim 1, wherein in the step of e:

the single i-th image frame F_i is spliced to the current spliced image with a method of a weighted average of the partially overlapped region; and

for the splicing of the current aligned image, the single i-th image frame F_i is directly embedded in the current aligned image based upon the transform coefficient.

11. The method according to claim 10, wherein the method of the weighted average of the partially overlapped region comprises the steps of:

i. dividing the i-th image frame F_i with a width of wFlt into three parts respectively with a width of wFlt·

ω

₁, wFlt·

ω

₂ and wFlt·

ω

₃ along the direction of the splicing, where, ω

₁+ω

₂+ω

₃=1; and

ii. for the first part, directly using the value of each pixel to set a value for a corresponding point on a new spliced image;

for the third part, reserving the values of respective pixels on the new spliced image that are corresponding to the part; and

for the second part, a value Gray of the corresponding point on the new spliced image being defined as;

Gray=GFlt*weight+GScp*(1−

weight)

where, GFlt is the value of each pixel on the i-th image frame F_i, GScp is the original value of the corresponding point on the current spliced image, and weight is a value of weighting.

12. The method according to claim 11, wherein:

the value of weighting weight ranges from 0.1 to 0.9 and is defined as;

weight=0.8*(x−

XBegin)/(XEnd−

XBegin)+0.1

where xε

[XBegin, XEnd], and [XBegin, XEnd] is an abscissa range of the second part of the F_i.

14. The method according to claim 1, wherein, for the first image frame, i=2 and Δ

=1.

13. A method of panoramic image processing, for generating an object image from an image sequence obtained in a real-time way or stored in a medium, in which the serial number i of an image is 1, 2, 3, . . . , comprising the steps of:

a. obtaining the image sequence from an ultrasonic imaging apparatus comprising a probe for transmitting an ultrasonic wave to a detected organism and receiving an ultrasonic echo reflected from the detected organism, and initializing a spliced image as a first image frame;

b. searching an i-th image frame F_i for characteristic points by performing divisions for a middle part of the i-th image frame F_i, selecting valid sub-regions from sub-regions of the i-th image frame F_i through a first filtering; and

selecting a characteristic point in each valid sub-region;

c. calculating motion vectors for the valid sub-regions of the i-th image frame F_i, selecting valid motion vectors from the motion vectors through a second filtering, and jumping to step of f if the number of the valid motion vectors is smaller than a preset number;

d. calculating a transform coefficient based upon a fitting of the valid motion vectors;

e. splicing the i-th image frame F_i to a current spliced image based upon the transform coefficient; and

f. displaying the current spliced image as a resultant image on a display device.

15. The method according to claim 13, wherein, for the first image frame, i=2.

16. A panoramic imaging system comprising:

an ultrasonic imaging apparatus comprising;

a probe to transmit ultrasonic waves to a detected organism and receive ultrasonic echoes reflected from the detected organism;

a beam combiner coupled to the probe to receive the ultrasonic echoes;

a detector coupled to the beam combiner to provide focus delay, weighting, and channel summing to formed ultrasonic beams; and

a digital signal transformer to receive a signal from the detector, the digital signal transformer to perform a coordinate transform on the signal to produce an ultrasonic image sequence, in which a serial number i of an image is 1, 2, 3, . . . , and A is an interval for picking up the image sequence;

a processor to combine the image sequence into a panoramic image, the processor configured to;

initialize an aligned image and a spliced image as a first image frame;

divide an i-th image frame F;

into a plurality of sub-regions, and select valid sub-regions from sub-regions of the i-th image Fi;

through a first filtering

calculate motion vectors for the valid sub-regions of the i-th image frame Fi;

with respect to a current aligned image;

adaptively adjust the interval Δ

within a value range of Δ

based on a translation amount of the i-th image frame Fi being within a controllable range

calculate a transform coefficient based upon a fitting of the motion vectors;

splice the i-th image frame F;

to a current spliced image based upon the transform coefficient, and configure or splice the current aligned image with the i-th image frame Fi; and

output the current spliced image as a resultant image; and

a display device to display the resultant image.

17. The system according to claim 16, wherein the processor is further configured to:

adaptively adjusting the interval Δ

within a value range of Δ

based on a translation amount being within a controllable range.

18. The system according to claim 17, wherein the controllable range of the translation amount is set in a range from 1 to 100 pixels.

19. The system according to claim 16 wherein the processor is further configured to:

adaptively adjusting the interval Δ

which a value range of Δ

based on a translation amount being within a controllable range.

20. The system according to claim 19, wherein the processor is further configured to:

determine a template on the i-th image frame F_i for a selected one of the characteristic points, which is a region that includes a set of adjacent points centering on the selected characteristic point;

determine a searched region on the current aligned image; and

determine a corresponding point of the selected characteristic point within the searched region through a similarity calculation, which is a central point of an adjacent region with a same size as the template and with the largest similarity.

21. The system according to claim 20, wherein the processor is further configured to define the searched region through a prediction:

$\left[x<br><br>+<br><br>\mathrm{pre\_offx}<br><br>+<br><br>\mathrm{XLft},<br><br>x<br><br>+<br><br>\mathrm{pre\_offx}<br><br>+<br><br>\mathrm{XRgt}\right]*\hspace{1em}<br><br><br><br>[\hspace{1em}<br><br>y<br><br>+<br><br>\mathrm{pre\_offy}<br><br>-<br><br>\mathrm{Dy},<br><br>y<br><br>+<br><br>\mathrm{pre\_offy}<br><br>+<br><br>\mathrm{Dy}];<br><br><br><br>\mathrm{XLft}=\{\begin{array}{cc}-\mathrm{\mu <br><br>}& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{ture}\\ -v& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{false}\end{array}<br><br><br><br>\mathrm{XRgt}=\{\begin{array}{cc}v& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{true}\\ \mathrm{\mu <br><br>}& \mathrm{if}<br><br><br><br>\mathrm{ScanDirection}<br><br><br><br>\mathrm{is}<br><br><br><br>\mathrm{false}\end{array}$

where, (x, y) are coordinates of the selected characteristic point, (pre_offx, pre_offy) is an offset vector of the previous image frame, ScanDirection represents a general direction for splicing the images, true represents that the splicing is performed toward the positive direction of the X coordinate axis, and false represents that the splicing is performed toward the negative direction of the X coordinate axis;

Dy=γ

, γ

is a constant, and μ

<

v, each of which is a predictive value.

22. The system according to claim 20, wherein the processor is further configured to rotate the i-th image frame F_i, which is floating with respect to the current aligned image by an angle which is defined through a prediction:

$\mathrm{\sigma <br><br>}=\{\begin{array}{cc}<br><br>\mathrm{\alpha <br><br>}& \mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}<<br><br>\mathrm{\varepsilon <br><br>}\\ <br><br>\mathrm{\alpha <br><br>}+\left(\mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}\right)& \mathrm{\alpha <br><br>}-\mathrm{\beta <br><br>}\ge <br><br>\mathrm{\varepsilon <br><br>}\end{array}$

where, α

is the rotation angle of the previous floating image, β

is the rotation angle of the further previous floating image, and ε

is a factor of error.

23. The system according to claim 19, wherein the processor is further configured to:

filter the motion vectors to select valid motion vectors.

24. The system according to claim 16, wherein the processor is further configured to:

if an offset of the calculated transform coefficient is abnormal, or a number of valid motion vectors selected through a second filtering is less than a preset number, not subject a current image frame to a subsequent splicing;

return the processing of the current image frame to an initial status and adjust the interval Δ

for picking up the image sequence based on the processing of another frame; and

when there occurs a case continuously several times in which the current image frame is not subject to the subsequent splicing, output the current spliced image as the resultant image.

25. The system according to claim 23, wherein the processor is further configured to:

if an offset of the calculated transform coefficient is abnormal, or a number of the valid motion vectors selected through the filtering is less than a preset number, not subject a current image frame to a subsequent splicing;

return the processing of the current image frame to an initial status and adjust the interval Δ

for picking up the image sequence based on the processing of another frame; and

when there occurs a case continuously several times in which the current image frame is not subject to the subsequent splicing, output the current spliced image as the resultant image.

26. The system according to claim 16, wherein when the processor performs splicing, the processor is configured to:

splice the single i-th image frame F_i to the current spliced image with a method of a weighted average of the partially overlapped region; and

splice the current aligned image by directly embedding the single i-th image frame F_i in the current aligned image based upon the transform coefficient.

27. The system according to claim 26, wherein the processor performs the method of the weighted average of the partially overlapped region by:

dividing the i-th image frame F_i with a width of wFlt into three parts respectively with a width of wFlt·

ω

₁, wFlt·

ω

₂ and wFlt·

ω

₃ along the direction of the splicing, where, ω

₁+ω

₂+ω

₃=1; and

for the first part, directly using the value of each pixel to set a value for a corresponding point on a new spliced image;

for the third part, reserving the values of respective pixels on the new spliced image that are corresponding to the part; and

for the second part, a value Gray of the corresponding point on the new spliced image being defined as;

Gray=GFlt*weight+GScp*(1−

weight)

where, GFlt is the value of each pixel on the i-th image frame F_i, GScp is the original value of the corresponding point on the current spliced image, and weight is a value of weighting.

28. The system according to claim 27, wherein:

the value of weighting weight ranges from 0.1 to 0.9 and is defined as;

weight=0.8*(x−

XBegin)/(XEnd−

XBegin)+0.1

where xε

[XBegin, XEnd], and [XBegin, XEnd] is an abscissa range of the second part of the F_i.