Method for creating and utilizing electronic image book and recording medium having recorded therein a program for implementing the method

• US 6,571,054 B1
• Filed: 11/09/1998
• Issued: 05/27/2003
• Est. Priority Date: 11/10/1997
• Status: Expired due to Fees

• Current Assignee: Nippon Telegraph and Telephone Corporation
• Original Assignee: Nippon Telegraph and Telephone Corporation
• Inventors: Taniguchi, Yukinobu, Sato, Takashi, Ozawa, Hideaki, Tonomura, Yoshinobu
• Primary Examiner(s): NGUYEN, HUY THANH
• Application Number:
  US09/188,196
• Time in Patent Office: 1,660 Days
• Field of Search: 386/46, 386/95, 386/120, 386/121, 386/125, 386/126, 358/909.1, 345/328, 345/350
• US Class Current: 386/241
• CPC Class Codes: G06F 15/0283 for data storage and retrieval
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

In the creation of an electronic image book provided with a book-type interface; a video is analyzed; images are extracted from the video under predetermined event type conditions; the extracted images are stored as index information in an index management table; images are selected from the index management table under desired conditions and laid out as a sequence of representative images in a page screen; item numbers of the laid out representative images, information about their positions on the page and information about a video file linked to the representative images are recorded in a page management table in correspondence with pages; and at the same time, representative image data corresponding to the item number is recorded in an image data table.

Claims

1. A method for creating from a video an electronic image book with a book-type interface, comprising the steps of:

(a) detecting predetermined types of events by analyzing the video, and creating a video index management table in which the detected events, start and end time of the events and feature information of images computed during the analysis of the video are managed as video index information in correspondence with partial video images of the events in said video;

(b) making a user specify conditions for creating representative images of the electronic book;

(c) referring to the video index information in said video index management table to meet the user'"'"'s specified conditions, selecting from said video a sequence of images that satisfy said creating conditions, and generating a sequence of representative images;

(d) laying out said sequence of representative images in their temporal order in a page of a predetermined size defined in a memory, deciding whether the next representative image to be laid out relative to the position of the current representative image already laid out extends off the current page region and, if so, performing a page break and laying out the subsequent representative image on the new page, and computing the variation level indicating the degree of a change in the video contents, and performing a page break when said variation level exceeds a predetermined value; and

(e) generating a page management table which manages information about the layout of said representative images and link information from said representative images to the corresponding partial video images in said video.

2. The method of claim 1, wherein said step (a) includes a step of detecting a scene-change as an event, and said step (c) includes a step of selecting, as a representative image, an image immediately following the scene-change event or an image at a certain elapsed time.

3. The method of claim 1, wherein said step (c) includes a step of synthesizing at least one of said representative images from a plurality of images extracted from said video.

4. The method of claim 3, wherein said step (a) includes a step of detecting an event concerning a camera operation, and said step (c) includes a step of generating, for a scene involving no camera operation event, an image immediately following the scene-change event or an image at a certain elapsed time as a representative image, and for a scene involving the camera operation event, generating as a representative image a panoramic image synthesized from a plurality of images extracted from that scene.

5. The method of claim 1, wherein said step (d) includes a step of placing an icon, representing the existence of particular sound information, on or adjacent to said representative image linked to said video segment containing a particular sound information.

6. The method of claim 1, wherein said page management table generating step (e) includes a step of generating a page contents table which contains, in correspondence with item numbers of representative images as items to be displayed in each page, positional information specifying the positions of those representative image in the page layout and said link information to said partial video corresponding to the representative images.

7. The method of claim 6, wherein said page management table generating step (e) includes a step of generating an image data table with representative image data written therein in correspondence with said item numbers in said page contents table.

8. The method of claim 6 or 7, wherein said page contents table generating step includes a step of writing link information for partial videos in said video corresponding to the item numbers.

9. The method of claim 6 or 7, wherein said electronic image book comprises a front cover page, a table-of-contents page, a main-body page, and an index page, and which further comprises a step of generating page contents tables corresponding to said pages and a page index table that holds pointers for said page contents tables corresponding to said pages.

10. The method of claim 9, wherein said page index table generating step includes:

a step of enumerating those of said representative images in said page contents table whose additional information in said page contents table meet the condition specified by the user, at the time of or after laying out said representative images;

a step of laying out the enumerated representative images as the front cover, or the table of contents, or indexes in the front cover page, or table-of-contents page, or index page, and additionally generating a page contents table of the front cover page, or table-of-contents table, or index page for holding information about the layout in correspondence with the representative images; and

a step of recording information associated with said front cover page, table-of-contents page, or index page in said page index table.

11. The method of claim 10, wherein event types and additional image information are given predetermined weighting coefficients in advance and the sum of weighting coefficients for each representative image is defined as the level of image contents variation of said each representative image, and wherein said page index table generating step includes a step of calculating the level of image contents variation for each representative image, deciding whether the calculated value exceeds a predetermined value and, if so, adding said representative image to said table-of-contents page.

12. The method claim 10, which further comprises a step of deciding whether each representative image carries a telop centrally thereof and, if so, placing the representative image as a display image on the front cover page.

13. The method of claim 10, which further comprises a step of repeating, for each representative image, a process of deciding whether said each representative image contains a telop displayed centrally thereof and, if so, adding the representative image as an index item to the index page, and generating the index page by arranging said index items by a predetermined rule.

14. The method of claim 6, wherein said page contents table generating step includes a step of adding to said page contents table information for highlighting the representative image on the layout when additional information about said representative image meets predetermined particular condition.

15. The method of claim 1, wherein event types and additional image information, which cause page breaks, are given predetermined weighting coefficients in advance and the sum of the weighting coefficients for each representative image is defined as the level of its image contents variation, and wherein said step (d) includes a step of calculating, in the layout of said sequence of representative image, the level of contents variation of each representative images, comparing the calculated value with a predetermined threshold value and, if the former is larger than the latter, performing a page break and laying out the subsequent representative images on the new page.

16. The method of claim 1, wherein said step (d) includes a step of placing the representative images at positions where to minimize the area of a wasted regions under the conditions that satisfies layout constraints resulting from the temporal order of said representative images and that their layout fits in the page concerned.

17. A method for utilizing an electronic image book having a book-type interface, which has:

a page contents table having written therein, for each page, item numbers and display positions of representative images to be displayed in said each page, and information about video segments linked to the representative images;

an image data table for holding image data corresponding to the item number of each representative image;

a video file storing video data of linked video segments;

a page index table having held therein each page number and a pointer for pointing at the page contents table in correspondence with the page number, and a tag table for a user, said method comprising the steps of;

(a) deciding whether a page control button was pressed and, if so, referring to said page contents table corresponding to the page specified by the control button and reading out the item number and positional information, then reading out of said image data table the image data corresponding to the read-out item number, and displaying the read-out data on a page screen at the position specified by said positional information to thereby display a specified page screen;

(b) deciding whether a representative image button was pressed and, if so, link information of the corresponding representative image is read out of said page contents table corresponding to the pressed representative image button, then reading out from said video file the partial video specified by the read-out link information and playing it back on the screen;

(c) deciding whether a tag pasting button on the page screen was pressed and, if so, writing the item number of the representative image specified by the user on the current page in said tag table in correspondence with the current page number, and reading out the corresponding image from said image data table and pasting it as a tag image at a specified position and in a specified size;

(d) deciding whether the tag image was pressed in an arbitrary page and, if so, reading out of said tag table the page number specified by the pressed tag image, then referring to said page contents table corresponding to the page specified by the read-out page number, then reading out therefrom the item number and position information of the image, then reading out from said image data table the image data corresponding to the read-out item number, and displaying the read-out image data on the page screen at the position specified by said positional information to thereby a display screen with said tag pasted thereon.

18. The method of claim 17, which further comprises the steps of:

(c) when a representative image as a particular item of the table-of-contents page is specified, accessing the page contents table containing the item number of the specified image;

(d) obtaining the item number and positional information shown in the page contents table and reading out of said image data table the image data corresponding to the item number; and

(e) displaying the read-out image data on the page screen at the position specified by said positional information to thereby display the specified page screen.

19. The method of claim 17, which further comprises the steps of:

(c) when a representative image as a particular item of the index page is designated, accessing the page contents table containing the item number of the specified image;

(d) obtaining the item number and positional information shown in the page contents table and reading out of said image data table the image data corresponding to the item number; and

(e) displaying the read-out image data on the page screen at the position specified by said positional information to thereby display the specified page screen.

20. The method of claim 17, which further comprises the step of deciding whether a user specified a sound visualization image displayed in the page and, if so, reading out the associated information from said page management table and displaying it, and reading out the associated partial video from said video file and playing it back.

21. The method of claim 17, which further comprises the steps of:

(c) deciding whether a button specifying an edit mode was pressed and, if so, entering the edit mode;

(d) upon each specification of the representative image representing the partial video in said edit mode, storing the linked partial videos of the specified representative images in the order in which they were specified; and

(e) deciding whether a playback button was pressed and, if so, accessing the linked video segments via said stored link information in the order of their specification, reading the corresponding partial videos from said video file and playing them back.

22. A recording medium having stored therein a program for creating from a video an electronic image book with a book-type interface, comprising the steps of:

(a) detecting predetermined types of events by analyzing the video, and creating a video index management table in which the detected events, start and end time of the events and feature information of images computed during the analysis of the video are managed as video index information in correspondence with a partial video images of the events in said video;

(b) making a user to specify conditions for creating representative images of the electronic book;

(c) referring to the video index information in said video index management table to meet the user'"'"'s specified conditions, selecting from said video a sequence of images that satisfy said creating conditions, and generating a sequence of representative images;

(d) laying out said sequence of representative images in their temporal order in a page of a predetermined size defined in a memory, deciding whether the representative image to be laid out next relative to the position of the current representative image already laid out extends off the current page region and, if so, performing a page break and laying out the subsequent representative image on the new page, and computing the variation level indicating the degree of a change in the video contents, and performing a page break when said variation level exceeds a predetermined value; and

(e) generating a page management table which manages information about the layout of said representative images and link information from said representative images to the corresponding partial video images in said video.

23. The recording medium of claim 22, wherein said process (d) includes a process (d) of placing an icon, representing the existence of particular sound information, on or adjacent to said representative image linked to said video segment containing a particular sound information.

24. The recording medium of claim 22, wherein said page management table generating process (e) includes a process of generating a page contents table which contains, in correspondence with item numbers of representative images as items to be displayed in each page, positional information specifying the positions of those representative image in the page layout and said link information to said partial video corresponding to the representative images.

25. The recording medium of claim 22, wherein said page management table generating process (e) includes a process of generating an image data table with representative image data written therein in correspondence with said item numbers in said page contents table.

26. The recording medium of claim 24 or 25, wherein said page contents table generating process includes a process of writing link information for partial videos in said video corresponding to the item numbers.

27. The recording medium of claim 24 or 25, wherein said electronic image book comprises a front cover page, a table-of-contents page, a main-body page, and an index page, and which further comprises a process of generating page contents tables corresponding to said pages and a page index table that holds pointers for said page contents tables corresponding to said pages.

28. The recording medium of claim 27, wherein said page index table generating process includes:

a process of enumerating those of said representative images in said page contents table whose additional information in said page contents table meet the condition specified by the user, at the time of or after laying out said representative images;

a process of laying out the enumerated representative images in the front cover page, or table-of-contents page, or index page, and additionally generating a page contents table of the front cover page, or table-of-contents table, or index page for holding information about the layout in correspondence with the representative images; and

a process of recording information associated with said front cover page, table-of-contents page, or index page in said page index table.

29. The recording medium of claim 28, wherein event types and additional image information are given predetermined weighting coefficients in advance and the sum of weighting coefficients for each representative image is defined as the level of image contents variation of said each representative image, and wherein said page index table generating process includes a process of calculating the level of image contents variation for each representative image, deciding whether the calculated value exceeds a predetermined value and, if so, adding said representative image to said table-of-contents page.

30. The recording medium of claim 22, wherein event types and additional image information, which cause page breaks, are given predetermined weighting coefficients in advance and the sum of the weighting coefficients for each representative image is defined as the level of its image contents variation, and wherein said process (d) includes a process of calculating, in the layout of said sequence of representative image, the level of contents variation of each representative images, comparing the calculated value with a predetermined threshold value and, if the former is larger than the latter, performing a page break and laying out the subsequent representative images on the new page.

31. The recording medium of claim 22, wherein said process (d) includes a process of placing the representative images at positions where to minimize the area of wasted regions under the conditions that satisfies layout constraints resulting from the temporal order of said representative images and that their layout fits in the page concerned.

36. The method of claim 22, wherein said step (a) includes a step of detecting a scene-change a an event, and said step (c) includes a step of selecting, as a representative image, an image immediately following the scene-change event or an image at a certain elapsed time.

37. The method of claim 22, wherein said step (c) includes a step of synthesizing at least one of said representative images from a plurality of images extracted from said video.

38. The method of claim 37, wherein said step (a) includes a step of detecting an event concerning a camera operation, and said step (c) includes a step of generating, for a scene involving no camera operation event, an image immediately following the scene-change event or an image at a certain elapsed time as a representative image, and for a scene involving the camera operation event, generating as a representative image a panoramic image synthesized from a plurality of images extracted from that scene.

39. The method claim 28, which further comprises a step of deciding whether each representative image carries a telop centrally thereof and, if so, placing the representative image as a display image on the front cover page.

40. The method of claim 29, which further comprises a step of repeating, for each representative image, a process of deciding whether said each representative image contains a telop displayed centrally thereof and, if so, adding the representative image as an index item to the index page, and generating the index page by arranging said index items by a predetermined rule.

41. The method of claim 24, wherein said page contents table generating step includes a step of adding to said page contents table information for highlighting the representative image on the layout when additional information about said representative image meets predetermined particular condition.

32. A recording medium having stored therein a program to provide a method for utilizing an electronic image book having a book-type interface, which has:

a page contents table having written therein, for each page, item numbers and display positions of representative images to be displayed in said each page, and information about video segments linked to the representative images;

an image data table for holding image data corresponding to the item number of each representative image;

a video, file storing video data of linked video segments;

a page index table having held therein each page number and a pointer for pointing at the page contents table in correspondence with the page number, and a tag table for a user, said method comprising the steps of;

(a) deciding whether a page control button was pressed and, if so, referring to said page contents table corresponding to the page specified by the control button and reading out the item number and positional information, then reading out of said image data table the image data corresponding to the read-out item number, and displaying the read-out data on a page screen at the position specified by said positional information to thereby display a specified page screen;

(b) deciding whether a representative image button was pressed and, if so, link information of the corresponding representative image is read out of said page contents table corresponding to the pressed representative image button, then reading out from said video file the partial video specified by the read-out link information and playing it back on the screen;

(c) deciding whether a tag pasting button on the page screen was pressed and, if so, writing the item number of the representative image specified by the user on the current page in said tag table in correspondence with the current page number, and reading out the corresponding image from said image data table and pasting it as a tag image at a specified position and in a specified size;

(d) deciding whether the tag image was pressed in an arbitrary page and, if so, reading out of said tag table the page number specified by the pressed tag image, then referring to said page contents table corresponding to the page specified by the read-out page number, then reading out therefrom the item number and position information of the image, then reading out from said image data table the image data corresponding to the read-out item number, and displaying the read-out image data on the page screen at the position specified by said positional information to thereby a display screen with said tag pasted thereon.

33. The recording medium of claim 32, which further comprises the processes of:

(c) when a representative image as a particular item of the table-of-contents page is specified, accessing the page contents table containing the item number of the specified image;

(d) obtaining the item number and positional information shown in the page contents table and reading out of said image data table the image data corresponding to the item number; and

(e) displaying the read-out image data on the page screen at the position specified by said positional information to thereby display the specified page screen.

34. The recording medium of claim 32, which further comprises the process of:

(c) when a representative image as a particular item of the index page is designated, accessing the page contents table containing the item number of the specified image;

(d) obtaining the item number and positional information shown in the page contents table and reading out of said image data table the image data corresponding to the item number; and

(e) displaying the read-out image data on the page screen at the position specified by said positional information to thereby display the specified page screen.

35. The recording medium of claim 32, which further comprises the process of deciding whether a user specified a sound visualization image displayed in the page and, if so, reading out the associated information from said page management table and displaying it, and reading out the associated partial video from said video file and playing it back.

42. The method of claim 32, which further comprises the steps of:

(c) deciding whether a button specifying an edit mode was pressed and, if so, entering the edit mode;

(d) upon each specification of the representative image representing the partial video in said edit mode, storing the linked partial videos of the specified representative images in the order in which they were specified; and

(e) deciding whether a playback button was pressed and, if so, accessing the linked video segments via said stored link information in the order of their specification, reading the corresponding partial videos from said video file and playing them back.