Frame rate converting apparatus, pan/tilt determining apparatus, and video apparatus

• US 20060215037A1
• Filed: 03/23/2006
• Published: 09/28/2006
• Est. Priority Date: 03/25/2005
• Status: Active Grant

• Current Assignee: Xacti Corporation (Office Support KK)
• Original Assignee: Sanyo Electric Company Limited (Panasonic Holdings Corporation)
• Inventors: Maenaka, Akihiro, Tsunekawa, Norikazu, Mori, Yukio
• Granted Patent: US 7,929,611 B2
• Time in Patent Office: Days
• US Class Current: 348/211.990
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/6815 by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A frame rate converting apparatus includes a pan/tilt determining unit which determines whether an image is in a pan/tilt state on the basis of information related to motion of the image detected by a motion detecting unit for each frame of a moving image, a moving distance setting unit which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining unit, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting unit with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image, and a prediction image generating unit which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting unit.

Claims

1. A frame rate converting apparatus which converts a frame rate of a moving image, comprising:

motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted;

pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted;

moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; and

prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting means.

3. The frame rate converting apparatus according to claim 1, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and

the pan/tilt determining means includes

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.

5. The frame rate converting apparatus according to claim 1, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and

the pan/tilt determining means includes:

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity;

means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value;

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.

2. A frame rate converting apparatus which converts a frame rate of a moving image, comprising:

motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted;

pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted;

moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image;

time-direction smoothing means which smoothes the moving distance of the image set for each frame by the moving distance setting means in a direction of time; and

prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image obtained by the time-direction smoothing means.

4. The frame rate converting apparatus according to claim 2, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and

the pan/tilt determining means includes

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.

6. The frame rate converting apparatus according to claim 2, wherein the motion detecting means detects a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values as information related to the motion of the image for each of a plurality of motion vector detecting regions set in a video area of each frame by a typical point matching method, and

the pan/tilt determining means includes:

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity;

means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value;

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.

7. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein

the frame rate converting apparatus includes:

motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted;

pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted;

moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image; and

prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image set for each frame by the moving distance setting means.

8. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein

the frame rate converting apparatus includes:

motion detecting means which detects information related to motion of an image for each frame of a moving image to be converted;

pan/tilt determining means which determines whether the image is in a pan/tilt state on the basis of the information related to the motion of the image detected by the motion detecting means for each frame of the moving image to be converted;

moving distance setting means which sets 0 as a moving distance of an image with respect to a frame the image of which is not determined as an image in a pan/tilt state by the pan/tilt determining means, which calculates a moving distance of an image from the information related to the motion of the image detected by the motion detecting means with respect to a frame the image of which is determined as an image in a pan/tilt state, and which sets the obtained moving distance as a moving distance of the image;

time-direction smoothing means which smooths the moving distance of the image set for each frame by the moving distance setting means in a direction of time; and

prediction image generating means which generates a prediction image necessary for frame rate conversion on the basis of the moving distance of the image obtained by the time-direction smoothing means.

9. A pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, comprising:

means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method;

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.

10. A pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, comprising:

means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method;

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity;

means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value;

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.

11. A video apparatus having a pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, wherein

the pan/tilt determining apparatus includes:

means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method;

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, determines that the frame is panned/tilted.

12. A video apparatus having a pan/tilt determining apparatus which determines whether an image is in a pan/tilt state for each frame of a moving image, wherein

the pan/tilt determining apparatus includes:

means which calculates a motion vector, a minimum value of correlative accumulated values, and an average value of the correlative accumulated values for each of a plurality of motion vector detecting regions set in a video area of each frame of a moving image by a typical point matching method;

means which specifies a region in which the detected motion vector has high reliability of the motion vector detecting regions on the basis of the average value of the correlative accumulated values detected for each of the motion vector detecting regions and which calculates the number of detection regions in which the detected motion vectors have high reliability;

means which determines whether the number of detection regions in which the detected motion vectors have high reliability is not less than a first threshold value;

means which, when the number of detection regions in which the detected moving vectors have high reliability is less than the first threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the detected motion vectors have high reliability is not less than the first threshold value, specifies a detection region in which motions of a plurality of types are present and a detection region in which motions of a plurality of types are not present in the detection regions in which the motion vectors have high reliability on the basis of an average value of correlative accumulated values detected for the detection regions in which the motion vectors have high reliability and a minimum value of the correlative accumulated values, and which calculates the number of detection regions in which the motions of the plurality of types are present;

means which determines whether the number of detection regions in which the motions of the plurality of types are present is less than a second threshold value;

means which, when the number of detection regions in which the motions of the plurality of types are present is not less than the second threshold value, determines that the frame is not panned/tilted;

means which, when the number of detection regions in which the motions of the plurality of types are present is less than the second threshold value, specifies detection regions in which motion vectors have similarity in the detection regions in which the motion vectors have high reliability and the motions of the plurality of types are not present on the basis of a motion vector detected for the detection region in which the motion vector has high reliability and the motions of the plurality of types are not present, and which calculates the number of detection regions in which motion vectors have similarity;

means which determines whether the number of detection regions in which motion vectors have similarity is not less than a third threshold value;

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is less than the third threshold value, determines that the frame is not panned/tilted; and

means which, when the number of detection regions which are determined as detection regions in which the motion vectors have similarity is not less than the third threshold value, determines that the frame is panned/tilted.

13. A frame rate converting apparatus which converts a frame rate of a moving image, comprising:

means which defines one of two continuous frames in a moving image as a current frame, defines the other as a previous frame, and defines a frame generated between the current frame and the previous frame as a prediction frame, which calculates a shift distance of the prediction image with respect to a previous frame image and a shift distance of the prediction image with respect to a current frame image, on the basis of a moving distance of an image between a previous frame image and a current frame image and a frame rate;

means which shifts the previous frame image by using the shift distance of the prediction image with respect to the previous frame image to generate a first virtual prediction image constituted by a first blank portion in which the previous frame image is not present and a first original image portion in which the previous frame image is present;

means which shifts the current frame image by using the shift distance of the prediction image with respect to the current frame image to generate a second virtual prediction image constituted by a second blank portion in which the current frame image is not present and a second original image portion in which the current frame image is present; and

prediction image generating means which generates the prediction image on the basis of the previous frame image, the current frame image, the first virtual prediction image, and the second virtual prediction image,

wherein the prediction image generating means, if the first virtual prediction image and the second virtual prediction image are overlapped, generates a prediction image from the second virtual prediction image in a portion where the first blank portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image, generates a prediction image from the first virtual prediction image in a portion where the second blank portion in the second virtual prediction image is overlapped on the first original image portion in the first virtual prediction image, generates a prediction image by an image obtained by weighted-summing the current frame image and the previous frame image in a portion where the first blank portion and the second blank portion are overlapped, and generates a prediction image from one of the first virtual prediction image and the second virtual prediction image in a portion where the first original image portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image.

14. A video apparatus having a frame rate converting apparatus which converts a frame rate of a moving image, wherein

the frame rate converting apparatus includes:

means which defines one of two continuous frames in a moving image as a current frame, defines the other as a previous frame, and defines a frame generated between the current frame and the previous frames as a prediction frame, which calculates a shift distance of the prediction image with respect to a previous frame image and a shift distance of the prediction image with respect to a current frame image, on the basis of a moving distance of an image between a previous frame image and a current frame image and a frame rate;

means which shifts the previous frame image by using the shift distance of the prediction image with respect to the previous frame image to generate a first virtual prediction image constituted by a first blank portion in which the previous frame image is not present and a first original image portion in which the previous frame image is present;

means which shifts the current frame image by using the shift distance of the prediction image with respect to the current frame image to generate a second virtual prediction image constituted by a second blank portion in which the current frame image is not present and a second original image portion in which the current frame image is present; and

prediction image generating means which generates the prediction image on the basis of the previous frame image, the current frame image, the first virtual prediction image, and the second virtual prediction image,

wherein the prediction image generating means, if the first virtual prediction image and the second virtual prediction image are overlapped, generates a prediction image from the second virtual prediction image in a portion where the first blank portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image, generates a prediction image from the first virtual prediction image in a portion where the second blank portion in the second virtual prediction image is overlapped on the first original image portion in the first virtual prediction image, generates a prediction image by an image obtained by weighted-summing the current frame image and the previous frame image in a portion where the first blank portion and the second blank portion are overlapped, and generates a prediction image from one of the first virtual prediction image and the second virtual prediction image in a portion where the first original image portion in the first virtual prediction image is overlapped on the second original image portion in the second virtual prediction image.