Intelligent video system

• US 7,362,949 B2
• Filed: 09/28/2001
• Issued: 04/22/2008
• Est. Priority Date: 09/30/2000
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Lee, Sang Yong, Yoon, Kyoung Ro, Jun, Sung Bae, Lee, Ji Eun
• Primary Examiner(s): Miller; John
• Assistant Examiner(s): ATALA, JAMIE JO
• Application Number:
  US09/964,533
• Publication Number:
  US 20020039481A1
• Time in Patent Office: 2,398 Days
• Field of Search: 386/68, 386125-126, 386/45, 386/96, 386/46, 386/95, 386/70, 386/75, 386/104, 386/81, 360/33.1, 360 101- 103
• US Class Current: 386/343
• CPC Class Codes: G11B 27/005 Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 5/783 Adaptations for reproducing at a rate different from the recording rate

Abstract

Described is an intelligent fast-forward video system that automatically controls a relative play speed of the video based on a complexity of the content reflecting the real story of a video, thereby enabling to carry out a fast-forward viewing for summarizing an entire story of a digital video or moving fast to a major concerning part. The present invention obtains the complexity of the content using information of motion vector, shot, face, text, and audio for an entire video and adaptively controls the play speed for each of the intervals on a fast-forward viewing of the corresponding video on the basis of the obtained complexity of the content. And, in more detail, the present invention plays a complicated story interval relatively slow and a simple and tedious part relatively fast, thereby enabling to provide a user with a summarized story of the corresponding video without viewing the entire video as well as a fast search for a part wanted by the user.

Claims

1. A method of controlling an intelligent fast-forward video system comprising:

calculating complexity of a video story development based on motion information by sequentially indexing an entire video;

determining a play speed using the calculated motion-based complexity; and

variably controlling the play speed of the video based on the determined play speed,

wherein the complexity is defined based on additional information on a length of shot segment, wherein the complexity is defined as simple in a case that a length of a shot segment is long and the complexity is defined as more complicated in a case that shots having short shot segments consecutively appear.

2. The method of claim 1, wherein the motion information is based on an amount of a numerically expressed motion in one frame of the video.

3. The method of claim 1, wherein the numerically expressed motion in one frame is defined by motion vectors and information on intra-coded macro-blocks and not-coded macro-blocks.

4. The method of claim 3, wherein the macro-blocks are considered as motion vectors having a predetermined size and the predetermined size of the motion vector is determined as one of a maximum motion vector size, an arbitrary value in accordance with usage, and ‘

0’

.

5. The method of claim 4, wherein the amount of the numerically expressed motion in one frame is defined by considering the intra-coded and not-coded macro-blocks as motion vectors having designated sizes and then by taking an average value of all vector sizes of all the remaining macro-blocks.

6. The method of claim 1, wherein the motion information is defined by an average value of motion information in the respective frames of an interval in specific consecutive intervals of the video.

7. The method of claim 6, wherein the motion information is camera motion information and wherein an adjustment of the motion information of a corresponding interval is carried out to be inverse-proportional to a size of the camera motion in accordance with importance of the camera motion.

8. The method of claim 7, wherein the motion information of an interval having an important specific camera motion is adjusted to be more complicated or the motion information of an interval having an unimportant specific camera motion is adjusted to be simpler.

9. The method of claim 1, wherein the complexity based on the motion information is adjusted by using additional information for improving performance such that face detection information of characters, audio information and image text information in case the complexity of the video story development is reflected on the motion information.

10. The method of claim 9, wherein (a) the motion information of a magnified face part is adjusted as simpler in case the complexity is adjusted using the face detection information, (b) the motion information of a part in which a size of the audio information increases is adjusted as more complicated in case the complexity is adjusted using the audio information, and (c) the motion information of a part in which a caption appears or is changed is adjusted as more complicated in case the content complexity is adjusted using the image text information.

11. The method of claim 1, wherein the play speed based on the content complexity is adjusted to be inverse-proportional to complexity of video information, to be an absolute value, or to be designated as a value relative to an average and a standard deviation of entire video complexity.

12. The method of claim 1, wherein frames to be played are selected based on the content complexity, and wherein (a) the frames are selected such that total play time T of the frames classified to realize n times play speed is chosen to be equal to T/n in case a play time of normal speed of a video is T, (b) the frames are selected to be equally distributed in display time by considering decoding time of the respective picture types, (c) picture types having short decoding, time are selected preferentially to reduce the black-out caused by long decoding time of the pictures, or (d) at least one frame of the shot is designated and played to prevent a short shot from being omitted entirely.

13. The method of claim 1, wherein the index information is defined by a specific interval of the video and image complexity of the interval, and wherein the play is carried out by (a) determining a play speed of a corresponding interval based on the image complexity information and then (b) selecting a frame to be played based on the determined play speed in the corresponding interval.

14. The method of claim 1, wherein the index information is defined by a specific interval of the video and a play speed of the specific interval, wherein the specific interval is played at a designated speed thereto.

15. The method of claim 1, wherein the index information is defined by a sequence of frames to be played and wherein the frame sequence to be played is sequentially played.

16. The method of claim 1, the method further comprising changing the play speed so as to support a user to view more slowly or faster than the play speed provided by the system by the user'"'"'s adjustment of a content complexity value attained by the indexing information, wherein all values through entire intervals of the corresponding video are increased/decreased collectively or at a constant ratio by the user'"'"'s adjustment or wherein specific image complexity values are increased/decreased selectively by the user'"'"'s adjustment.

17. The method of claim 16, wherein the adjusted content complexity value is stored with information on the corresponding interval so as to be used as a new image complexity value when the corresponding interval starts to be played.

18. The method of claim 1, the method further comprising changing the play speed so as to support a user to view more slowly or faster than the play speed provided by the system by the user'"'"'s adjustment of a play speed value attained by the indexing information, wherein all values through entire intervals of the corresponding video are increased/decreased collectively or at a constant ratio by the user'"'"'s adjustment or wherein specific image complexity values are selectively increased/decreased by the user'"'"'s adjustment.

19. The method of claim 18, wherein the adjusted play speed value is stored with information on the corresponding interval so as to be used as a new image complexity value when the corresponding interval starts to be played.

20. The method of claim 1, wherein the index information is defined by a sequence of frames to be played so as to provide a viewing speed faster or slower than the play speed provided by the system in accordance with the index information, and wherein the play speed is decreased or increased by increasing or decreasing the defined frame sequence in the respective intervals of the corresponding video with a constant ratio, or by increasing or decreasing the defined frame sequence in specific intervals selectively.

21. The method of claim 1, the method further comprising:

setting a viewing time limit of fast-forward play of the video;

computing a required time for fast-forward play of the entire video on the basis of the index information; and

adjusting the fast-forward play speed to reach the time limit using the computed time and viewing time limit.

22. In variably controlling a play speed of a video based on a content complexity of video story development as motion information obtained by indexing the entire video, a method of constructing index information for controlling an intelligent fast-forward viewing includes constituting the content complexity as index information defined as a predetermined form enabling to designate each play speed of the corresponding content complexity,

wherein the content complexity is defined based on additional information on a length of shot segment, wherein the content complexity is defined as simple in a case that a length of a shot segment is long and the content complexity is defined as more complicated in a case that shots having short shot segments consecutively appear.

23. The method of claim 22, wherein the index information is defined by (a) a specific interval of the video and an image complexity corresponding to the specific interval, (b) an interval and a play speed corresponding to the interval, or (c) a sequence of frames to be played.

24. An apparatus for controlling intelligent fast-forward viewing of a video comprising:

a means for calculating a complexity of video content in accordance with story development of a corresponding video from a digital video stream including at least motion information;

a means for storing the calculated content complexity; and

a means for playing and displaying the corresponding video in different play speeds for each specific interval on the basis of the stored content complexity,

wherein the content complexity is defined based on additional information on a length of shot segment, wherein the content complexity is defined as simple in a case that a length of a shot segment is long and the content complexity is defined as more complicated in a case that shots having short shot segments consecutively appear.

25. The apparatus of claim 24, wherein shot information, face information of characters, image text information and audio information are used for producing the content complexity.

26. The apparatus of claim 24, wherein the play speed is controlled by the content complexity computed by automatically reflecting additional information according to a genre or by reflecting, the additional information by a user'"'"'s designation and selection.

27. The apparatus of claim 24, wherein the play speed is readjusted by a user'"'"'s designation at a playing stage.

28. The apparatus of claim 24, the apparatus further comprising a producing means for generating information which designates a play speed for a specific interval of the corresponding video on the basis of the computed content complexity, wherein the corresponding video is played and displayed by adjusting the play speed for each of the specific intervals on the basis of the speed designation information.

29. The apparatus of claim 28, wherein the speed designation information is at least one of the specific interval of the video and image complexity in the specific interval, a speed value in the video interval and corresponding interval, or a frame sequence corresponding to the play speed.

30. The apparatus of claim 24, the apparatus further comprising:

a means for generating information which designates a play speed in a specific interval of the corresponding video on the basis of the computed content complexity; and

a means for calculating number of frame to be played on the basis of the generated play speed,

wherein the corresponding video is played and displayed on the basis of the frame number to be played.