Method and apparatus for processing echocardiogram video images

• US 6,514,207 B2
• Filed: 04/16/2001
• Issued: 02/04/2003
• Est. Priority Date: 04/13/2000
• Status: Expired due to Fees

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Wu, Henry, Ebadollahi, Shahram, Chang, Shih-Fu
• Primary Examiner(s): Manuel, George
• Assistant Examiner(s): Imam, Ali M.
• Application Number:
  US09/835,166
• Publication Number:
  US 20020007117A1
• Time in Patent Office: 659 Days
• Field of Search: 600/437, 600/407, 600/440-447, 600/449-459, 736/25, 736/26, 736/41, 367/7, 367/11, 367/138, 128/916, 424/9.52, 382/294, 348/441, 348/443, 348/384.1
• US Class Current: 600/450
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: H04N 5/45 Picture in picture , e.g. displaying simultaneously another television channel in a region of the screen

Abstract

Methods and a system are disclosed for processing an echocardiogram video of a patient'"'"'s heart. The echocardiogram comprises at least a first sequence of consecutive video frames corresponding to a first view of the patient'"'"'s heart concatenated with a second sequence of consecutive video frames corresponding to a second view of the patient'"'"'s heart. The end-diastole phase of the patient'"'"'s heart is monitored in each frame by detecting the electrocardiograph wave, and a key frame is selected upon the occurrence of the R-wave peak in the electrocardiograph wave in each of the first sequence of consecutive video frames and in the second sequence of consecutive video frames. The shape and color content of the echocardiogram image window is monitored in certain video frames, and a transition is detected when there is a change in the first feature between adjacent frames. A summary is generated which comprises by the video frames corresponding to the end-diastole phase.

Claims

1. A method for processing an echocardiogram video of a patient'"'"'s heart, the echocardiogram video comprising a first sequence of consecutive video frames corresponding to a first view of the patient'"'"'s heart concatenated with a second sequence of consecutive video frames corresponding to a second view of the patient'"'"'s heart, the method comprising:

receiving the first sequence of consecutive video frames and the second sequence of consecutive video frames;

monitoring a first feature in each one of the video frames of the first and second sequences of consecutive video frames;

selecting a video frame in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames corresponding to an occurrence of the first feature in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames;

monitoring a second feature in the first sequence of consecutive video frames and the second sequence of consecutive video frames;

detecting a transition between the first sequence of consecutive video frames and the second sequence of consecutive video frames by determining a change in the second feature between adjacent video frames; and

generating a summary comprising video frames selected in the selecting step.

2. The method as recited in claim 1, wherein the first feature is the occurrence of the end-diastole phase of the patient'"'"'s heart cycle and wherein monitoring the first feature in the first sequence of consecutive video frames and the second sequence of consecutive video frames comprises:

detecting the occurrence of the end-diastole phase of the patient'"'"'s heart cycle.

3. The method as recited in claim 1, wherein each video frame includes an associated electrocardiograph wave and wherein the first feature is the occurrence of an R-wave peak in the electrocardiograph wave of a video frame and wherein monitoring the first feature in the first sequence of consecutive video frames and the second sequence of consecutive video frames comprises:

monitoring the electrocardiograph wave associated with each video frame.

4. The method as recited in claim 3, wherein selecting the video frame in each of the first plurality of video frames and the second plurality of video frames corresponding to an occurrence of the first feature in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames comprises:

detecting an occurrence of the R-wave peak in the electrocardiograph wave corresponding to each one of the first and second sequences of consecutive video frames.

5. The method as recited in claim 4, wherein the electrocardiograph wave comprises a time marker on the electrocardiograph wave indicating the position on the electrocardiograph wave corresponding to current electrocardiograph information and wherein detecting the occurrence of the R-wave peak in the electrocardiograph wave for each of the first plurality of video frames and the second plurality of video frames comprises:

detecting the location of the time marker in the electrocardiograph wave in each video frame;

monitoring the location of R-wave peaks in the electrocardiograph wave in each video frame; and

detecting when one of the R-wave peaks substantially coincides with the time marker.

6. The method as recited in claim 1, wherein each of the first sequence of consecutive video frames and the second sequence of video frames comprises an image window having one of a plurality of predefined shapes, wherein the second feature is the shape of the image window and wherein monitoring the second feature comprises:

detecting the shape of the image window in each of the video frames of the first and second sequences of consecutive video frames.

7. The method as recited in claim 6, wherein detecting the transition between the first sequence of consecutive video frames and the second sequence of consecutive video frames comprises:

detecting the occurrence of a change in the shape of the image window between adjacent video frames.

8. The method as recited in claim 6, wherein detecting the shape of the image window in each one of the video frames of the first and second sequences of consecutive video frames comprises:

segmenting the video frame into a plurality of gray levels;

clustering the plurality of gray levels into two main levels comprising the image window as one of the main levels and a background as the other of the main levels;

detecting a location and an orientation of a plurality of edges between the image window and the background; and

determining the shape of the image window based on the location and the orientation of the plurality of edges.

9. The method as recited in claim 1, wherein the second feature is the color content of each one of the video frames and wherein monitoring the second feature comprises:

monitoring the color content of each one of the video frames.

10. The method as recited in claim 9, wherein detecting the transition between the first sequence of consecutive video frames and the second sequence of consecutive video frames comprises:

detecting the occurrence of a change in the color content between adjacent video frames.

11. The method as recited in claim 1, wherein generating the summary comprises:

generating a sequence of individual video frames corresponding to the occurrence of the first feature in each of the first sequence of consecutive video frames and the second plurality of consecutive video frames.

12. The method as recited in claim 11, further comprising:

providing an association between each individual video frame of the summary and the corresponding sequence of consecutive video frames that includes the individual video frame.

13. The method as recited in claim 11, wherein generating the summary further comprises:

generating a first subset of the first sequence of consecutive video frames bounded by a first occurrence of the first feature and a second occurrence of the first feature, wherein the first subset includes the individual video frame corresponding to the occurrence of the first feature in the first sequence of consecutive video frames;

generating a second subset of the second sequence of consecutive video frames bounded by a first occurrence of the first feature and a second occurrence of the first feature, wherein the second subset includes the individual video frame corresponding to the occurrence of the first feature in the second sequence of consecutive video frames; and

generating a sequence of the subset of the first sequence of consecutive video frames and the subset of the second sequence of consecutive video frames.

14. The method as recited in claim 13, further comprising:

providing an association between each individual video frame of the summary, and the corresponding subset that includes the respective individual video frame.

15. A method for processing an echocardiogram video of a patient'"'"'s heart for diagnosis by a physician, the echocardiogram video comprising a first sequence of consecutive video frames corresponding to a first view of the patient'"'"'s heart concatenated with a second sequence of consecutive video frames corresponding to a second view of the patient'"'"'s heart, the method comprising:

receiving the first sequence of consecutive video frames and the second sequence of consecutive video frames;

monitoring a first feature in each one of the video frames of the first sequence of consecutive video frames and the second sequence of consecutive video frames;

selecting a video frame in each of the first plurality of video frames and the second plurality of video frames corresponding to an occurrence of the first feature in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames;

monitoring a second feature in the first sequence of consecutive video frames and the second sequence of consecutive video frames;

detecting a transition between the first sequence of consecutive video frames and the second sequence of consecutive video frames by determining a change in the second feature between adjacent video frames;

generating a summary comprising video frames selected in the selecting step;

providing the summary to the physician for viewing; and

providing the physician with the option, after viewing the summary, of supplying diagnosis information related to a plurality of predetermined structures of the patient'"'"'s heart that are visible in the first and second sequences of consecutive video frames.

16. The method as recited in claim 15, further comprising:

providing an association between each of the plurality of predetermined structures of the patient'"'"'s heart and the corresponding one of the first and second sequence of consecutive frames in which each of the plurality of predetermined structures of the patient'"'"'s heart is visible.

17. The method as recited in claim 16, further comprising:

after providing the physician with the option of supplying diagnosis information, providing an association between of the diagnosis information related to a plurality of predetermined structures of the patient'"'"'s heart supplied in the supplying step with the corresponding one of the first and second sequence of consecutive frames in which each of the plurality of predetermined structures of the patient'"'"'s heart is visible.

18. The method as recited in claim 15, wherein generating the summary occurs at one location and providing the summary to the physician comprises:

transmitting the summary to the physician at a second location remote from the first location.

19. A system for processing an echocardiogram video of a patient'"'"'s heart, the echocardiogram video comprising a first sequence of consecutive video frames corresponding to a first view of the patient'"'"'s heart concatenated with a second sequence of consecutive video frames corresponding to a second view of the patient'"'"'s heart, the system comprising:

video data storage for receiving the first sequence of consecutive video frames and the second sequence of consecutive video frames;

a programmed processor comprising;

a key frame selector configured to monitor a first feature in each one of the video frames of the first sequence of consecutive video frames and the second sequence of consecutive video frames, and select a video frame in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames corresponding to an occurrence of the first feature in each of the first sequence of consecutive video frames and the second sequence of consecutive video frames;

a view boundary detector configured to monitor a second feature in the first sequence of consecutive video frames and the second sequence of consecutive video frames from said video data storage, detect a transition between the first sequence of consecutive video frames and the second sequence of consecutive video frames by determining a change in the second feature between adjacent video frames; and

summary generator configured to generate a summary comprising video frames selected by the key frame selector.

20. The system as recited in claim 19, wherein the first feature is the occurrence of an end-diastole phase of the patient'"'"'s heart cycle and wherein the key frame selector is configured to monitor the occurrence of the end-diastole phase of the patient'"'"'s heart cycle.

21. The system as recited in claim 19, wherein each video frame includes an associated electrocardiograph wave and wherein the first feature is the occurrence of an R-wave peak in the electrocardiograph wave of a video frame and wherein the key frame selector is configured to monitor the electrocardiograph wave associated with each one of the video frames of the first and second sequences of consecutive video frames.

22. The system as recited in claim 21, wherein the key frame selector is configured to detect an occurrence of the R-wave peak in the electrocardiograph wave corresponding to each of one of the first sequence of consecutive video frames and the second sequence of consecutive video frames.

23. The system as recited in claim 22, wherein the electrocardiograph wave comprises a time marker on the electrocardiograph wave indicating the position on the electrocardiograph wave corresponding to current electrocardiograph information and wherein the key frame selector is configured to detect the location of the time marker in the electrocardiograph wave in each video frame;

to detect the location of R-wave peaks in the electrocardiograph wave in each video frame; and

to detect when one of the R-wave peaks substantially coincides with the time marker.

24. The system as recited in claim 19, wherein each of the first sequence of consecutive video frames and the second sequence of video frames comprises an image window having one of a plurality of predefined shapes, wherein the second feature is the shape of the image window, and wherein the view-boundary detector is configured to detect the shape of the image window in each video frame.

25. The system as recited in claim 24, wherein the view boundary detector is configured to detect the occurrence of a change in the shape of the image window between adjacent video frames.

26. The system as recited in claim 22, wherein the view boundary detector is configured to segment each one of the video frames into a plurality of gray levels, cluster the plurality of gray levels into two main levels comprising the image window as one of the main levels and a background as the other of the main levels;

detect a location and an orientation of a plurality of edges between the image window and the background; and

determine the boundaries of the image win dow based on the location and the orientation of the plurality of edges.

27. The system as recited in claim 19, wherein the second feature is the color content of each one of the video frames and wherein the view boundary detector is configured to detect the color content of each video frame.

28. The system as recited in claim 27, wherein the view boundary detector is configured to detect the occurrence of a change in the color content between adjacent video frames.

29. The system as recited in claim 19, wherein the summary generator is configured to generate a sequence of individual video frames selected by the key frame selector.

30. The system as recited in claim 29, wherein the summary generator is configured to provide an association between each individual video frame of the summary and the corresponding sequence of consecutive video frames that includes the individual video frame.

31. The system as recited in claim 19, wherein the summary generator is configured to generate a subset of the first sequence of consecutive video frames bounded by a first occurrence of the first feature selected by the key frame selector and a second occurrence of the first feature selected by the key frame selector;

generate a subset of the second sequence of consecutive video frames bounded by a first occurrence of the first feature and a second occurrence of the first feature selected by the key frame selector; and

generate a sequence of the subset of the first sequence of consecutive video frames and the subset of the second sequence of consecutive video frames.

32. The system as recited in claim 31, wherein the summary generator is configured to provide an association between the first and second subset and the corresponding sequences of consecutive video frames that include the first and second subsets, respectively.