Method for detecting camera-motion induced scene changes

• US 6,211,912 B1
• Filed: 02/04/1994
• Issued: 04/03/2001
• Est. Priority Date: 02/04/1994
• Status: Expired due to Term

• Current Assignee: WSOU Investments, LLC (WSOU Holdings, LLC)
• Original Assignee: Lucent Technologies, Inc. (Nokia Corporation)
• Inventors: Shahraray, Behzad
• Primary Examiner(s): Garber, Wendy R.
• Assistant Examiner(s): YOUNG, JOHN L
• Application Number:
  US08/191,234
• Time in Patent Office: 2,615 Days
• Field of Search: 348/228, 348/154, 348/155, 348/382, 348/699, 348/700
• US Class Current: 348/228.1
• CPC Class Codes: G11B 2220/65 wherein solid state memory is used for storing indexing information or metadata
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/147 Scene change detection

Abstract

A method for determining camera-induced scene changes in a sequence of visual information-bearing frames which constitute a single shot first determines camera-induced motion between each of a plurality of pairs frames within a single camera shot. The camera-induced motion for each of the pairs of frames is decomposed into at least a first component. The values of the first component for each of the pairs of frames are summed to form a first cumulative signal. A scene change is indicated when the first cumulative signal meets a certain decision criteria.

Claims

1. A method of determining whether a scene change in a sequence of visual information-bearing frames comprising content is a shot boundary scene change at a shot boundary between two successive shots or a camera-induced motion scene change at a point within a shot where the content changes considerably as a result of camera-induced motion, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal to indicate a scene change between the two successive shots;

(b) responsive to camera-induced motion within a particular shot of the two successive shots, generating a first signal representing camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames within the particular shot;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into at least a first component signal;

(d) summing the values of the first component signal for each of the pairs of frames to form a first cumulative signal; and

(e) generating a camera-induced motion scene change indicator signal to indicate a scene change within the particular shot of the successive shots when the first cumulative signal meets a predetermined decision criterion,

wherein the particular shot comprises a plurality of consecutive frames generated during a continuous time interval by a single camera.

4. The method of claim 1 further comprising the steps of:

(f) repeating step (c) for second and third components of the camera-induced motion to form second and third component signals such that the first, second, and third components represent image pan, image tilt, and image zoom, respectively; and

(g) repeating step (d) for the second and third components to respectively form second and third cumulative signals; and

wherein step (e) comprises the step of generating a camera-induced motion scene change indicator signal when at least one of the first, second or third cumulative signals meet a predetermined decision criterion.

5. The method of claim 1 further comprising the steps of:

(f) generating at least a first active signal corresponding to the first component of camera-induced motion, said first active signal having first and second values that represent, respectively, the absence of camera-induced motion and the presence of camera-induced motion for the first component of camera-induced motion; and

wherein the step of generating a camera-induced motion scene change indicator signal comprises the step of generating a camera-induced motion scene change indicator signal when the first active signal and the first cumulative signal each meet a certain decision criterion.

6. The method of claim 5 further comprising the steps of:

(g) repeating step (c) for second and third components of the camera-induced motion to form second and third component signals, such that the first, second, and third components represent image pan, image tilt, and image zoom, respectively;

(h) repeating step (d) for the second and third components to respectively form second and third cumulative signals;

(i) repeating step (f) to determine second and third active signals that correspond respectively to the second and third components of the camera motion; and

wherein the step of generating a camera-induced motion scene change indicator signal comprises the steps of generating a camera-induced motion scene change indicator signal when at least one of the first, second or third active signals and the cumulative signal representing the same component of camera-induced motion as the at least one of the first, second, or third active signals each meet a predetermined decision criterion.

7. The method of claim 4, wherein the predetermined decision criteria comprise predetermined thresholds for each of the cumulative signals, said decision criteria being met when at least one of the first, second, or third cumulative signals has a magnitude at least as large as its corresponding predetermined threshold.

8. The method of claim 6, wherein the step of generating a camera-induced motion scene change indicator signal further comprises the step of:

generating a camera-induced motion scene change indicator signal after the end of an interval over which at least one of the active signals assumes its second value and the magnitude of the corresponding cumulative signal representing the same component of camera-induced motion as the at least one active signal at the end of an active interval is at least as large as a predetermined threshold.

14. The method of claim 1, wherein the step of generating a camera-induced motion scene change indicator signal further comprises the step of retaining a frame from a scene defined by the indicated scene change.

19. The method of claim 5, further comprising the steps of:

(g) repeating step (c) for second and third components of the camera-induced motion to form second and third component signals, such that the first, second, and third components represent image pan, image tilt, and image zoom, respectively;

(h) repeating step (d) for a combination of the first, second, and third component signals to form a combined cumulative signal;

(i) repeating step (f) to generate a combined active signal that corresponds to the combined cumulative signal; and

wherein the step of generating a camera-induced motion scene change indicator signal comprises the step of generating a camera-induced motion scene change indicator signal when the combined active signal and the combined cumulative signal each meet a predetermined decision criterion.

2. A method of determining whether a scene change in a sequence of visual information-bearing frames comprising content is a shot boundary scene change at a shot boundary between two successive shots or a camera-induced scene change at a point within a shot where the content changes considerably as a result of camera-induced motion, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal;

(b) responsive to the occurrence of any camera-induced motion within a particular shot of the successive shots defined by the shot boundary, generating a signal representing the camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames within the particular shot;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into components representing values of image pan, image tilt, and image zoom to form an image pan signal, an image tilt signal, and an image zoom signal, respectively;

(d) summing each of the image pan, image tilt, and image zoom signals to form a cumulative pan signal, a cumulative tilt signal, and a cumulative zoom signal, respectively; and

(e) generating a camera-induced motion scene change indicator to indicate a scene change within one of the successive shots when at least one of the cumulative signals meet a predetermined decision criterion,

wherein the particular shot comprises a plurality of consecutive frames generated during a continuous time interval by a single camera.

9. The method of claim 2 further comprising the step of:

(f) generating an active signal for each of the image pan, image tilt, and image zoom components, each of said active signals having first and second values, the first value representing a component having a magnitude of zero and the second value representing a component having a magnitude that is non-zero; and

wherein the step of generating a camera-induced motion scene change indicator signal comprises the step of generating a camera-induced motion scene change indicator signal when at least one of the active signals and the cumulative signal representing the same component of camera-induced motion as the at least one active signal each meet predetermined decision criteria.

10. The method of claim 2, wherein the predetermined decision criterion are met when the magnitude of at least one of the cumulative signals surpasses a predetermined threshold.

11. The method of claim 2, wherein predetermined decision criterion comprise predetermined thresholds for each of the cumulative signals, said decision criteria being met when at least one of the cumulative signals surpasses its corresponding predetermined threshold.

12. The method of claim 9, wherein the step of generating a camera-induced motion scene change indicator signal further comprises the step of:

generating a camera-induced motion scene change indicator signal after the end of an interval, over which at least one of the active signals assumes its second value and the magnitude of the corresponding cumulative signal representing the same component of camera-induced motion as the at least one active signal at the end of the active interval is at least as large as a predetermined threshold.

13. The method of claim 2, wherein the step of generating a first signal comprises the steps of:

(a) performing motion estimation between each of the pairs of frames; and

(b) discriminating between camera-induced motion and object motion for each of the pairs of frames.

3. A method of determining camera-induced scene changes in a sequence of visual information-bearing frames comprising content, said method comprising the steps of:

(a) determining at least two camera shots within said visual information-bearing frames, each respective shot comprising a plurality of consecutive frames generated during a continuous time interval by a single camera; and

for at least one of said camera shots,

(b) generating a signal representing a considerable change in the content as a result of camera-induced motion between each of a plurality of pairs of frames within said at least one camera shot;

(c) summing a plurality of the signals generated for a respective plurality of pairs of frames to form a first cumulative signal; and

(d) generating an indicator signal that indicates a camera-induced scene change when the first cumulative signal meets a predetermined decision criterion.

15. A method of determining whether a scene change is a shot boundary scene change or a camera-induced motion scene change in a sequence of visual information-bearing frames, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal to indicate a scene change between the two successive shots;

(b) responsive to camera-induced motion, generating a first signal representing camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into at least a first component signal;

(d) summing the values of the first component signal for each of the pairs of frames to form a first cumulative signal;

(e) generating a camera-induced motion scene change indicator signal to indicate a scene change within one of the successive shots when the first cumulative signal meets a predetermined decision criterion;

(f) repeating steps (b) through (e) until all camera-induced motion scene changes within the sequence of visual information-bearing frames have been indicated, wherein the step of summing the values of the first component signal to form the first cumulative signal comprises the step of summing only those values of the first component signal that have been determined since the immediately preceding camera-induced motion scene change; and

(g) repeating steps (a)-(f) until all shot boundary scene changes and camera-induced motion scene changes within the sequence of visual information-bearing frames have been indicated.

16. The method of claim 15 further comprising the step of retaining a frame from each scene defined by the indicated scene changes.

17. The method of claim 15 further comprising the step of forming an index containing the location of a frame from each scene defined by the indicated scene changes.

18. The method of claim 17, wherein the retained frames form a set of representative frames that comprise a visual component of multimedia data.

20. A method of determining whether a scene change is a shot boundary scene change or a camera-induced motion scene change in a sequence of visual information-bearing frames, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal to indicate a scene change between the two successive shots;

(b) responsive to camera-induced motion, generating a first signal representing camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into at least a first component signal;

(d) summing the values of the first component signal for each of the pairs of frames to form a first cumulative signal;

(e) generating a camera-induced motion scene change indicator signal to indicate a scene change within one of the successive shots when the first cumulative signal meets a predetermined decision criterion;

(f) repeating step (a) until all shot boundary scene changes within the sequence of visual information-bearing frames have been indicated before proceeding to step (b).

21. A method of determining whether a scene change is a shot boundary scene change or a camera-induced motion scene change in a sequence of visual information-bearing frames, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal to indicate a scene change between the two successive shots;

(b) responsive to camera-induced motion, generating a first signal representing camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into at least a first component signal;

(d) summing the values of the first component signal for each of the pairs of frames to form a first cumulative signal;

(e) generating a camera-induced motion scene change indicator signal to indicate a scene change within one of the successive shots when the first cumulative signal meets a predetermined decision criterion;

(f) repeating steps (b) through (e) until a plurality of camera-induced motion scene changes within the sequence of visual information-bearing frames have been indicated, wherein the step of summing the values of the first component signal to form the first cumulative signal comprises the step of summing only those values of the first component signal that have been determined since the immediately preceding camera-induced motion scene change; and

(g) repeating steps (a)-(f) until a plurality of shot boundary scene changes and camera-induced motion scene changes within the sequence of visual information-bearing frames have been indicated.

22. A method of determining whether a scene change is a shot boundary scene change or a camera-induced motion scene change in a sequence of visual information-bearing frames, said method comprising the steps of:

(a) responsive to a shot boundary between two successive shots, generating a shot boundary scene change indicator signal to indicate a scene change between the two successive shots;

(b) responsive to camera-induced motion, generating a first signal representing camera-induced motion between each of a plurality of pairs of frames within said sequence of visual information-bearing frames;

(c) decomposing the camera-induced motion represented by the first signal for the plurality of pairs of frames into at least a first component signal;

(d) summing the values of the first component signal for each of the pairs of frames to form a first cumulative signal;

(e) generating a camera-induced motion scene change indicator signal to indicate a scene change within one of the successive shots when the first cumulative signal meets a predetermined decision criterion;

(f) repeating step (a) until a plurality of shot boundary scene changes within the sequence of visual information-bearing frames have been indicated before proceeding to step (b).