Interactive video icon with designated viewing position

• US 5,963,203 A
• Filed: 07/03/1997
• Issued: 10/05/1999
• Est. Priority Date: 07/03/1997
• Status: Expired

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Obvious Technology, Inc.
• Inventors: Madrane, Nabil, Goldberg, Cihl Mozest
• Primary Examiner(s): Breene, John E.
• Application Number:
  US08/887,992
• Time in Patent Office: 824 Days
• Field of Search: 345/326-358, 345/976, 345/977, 345/473-475, 345/127, 345/131, 345/302, 345/439, 348/699, 348/700
• US Class Current: 715/723
• CPC Class Codes: G06F 16/745 the internal structure of a single video sequence
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: G11B 2220/213 Read-only discs
• CPC Class Codes: G11B 2220/2545 CDs
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/105 of operating discs
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: Y10S 715/976 3-D Icon

Abstract

Interactive interfaces to video information provide a displayed view of a quasi-object called a root image. The root image consists of a plurality of basic frames selected from the video information, arranged such that their respective x and y directions are aligned with the x and y directions in the root image and the z direction in the root image corresponds to time, such that base frames are spaced apart in the z direction of the root image in accordance with their time separation. The displayed view of the root image changes in accordance with a designated viewing position, as if the root image were a three-dimensional object. The user can manipulate the displayed image by designating different viewing positions, selecting portions of the video information for playback and by special effects, such as cutting open the quasi-object for a better view. A toolkit permits interface designers to design such interfaces, notably so as to control the types of interaction which will be possible between the interface and an end user. Implementations of the interfaces may include editors and viewers.

Claims

1. An interface to an associated video sequence, the interface comprising:

a) information defining a three-dimensional root image, the root image consisting of a plurality of basic frames selected from said video sequence, and/or a plurality of portions of video frames corresponding to selected objects represented in the video sequence, x and y directions in the root image corresponding, to x and y directions in the video frames and the z direction in the root image corresponding to the time axis whereby the basic frames are spaced apart from one another in the z direction of the root image by distances corresponding to the time separation between the respective video frames;

b) means for displaying views of the root image;

c) means for designating a viewing position relative to said root image; and

d) means for calculating image data representing said three-dimensional root image viewed from the designated viewing, position, and for outputting said calculated image data to the displaying means.

2. An interactive interface according to claim 1, wherein the designating means is user-operable means for designating a viewing position relative to a displayed representation of the root image.

3. An interactive interface according to claim 1 wherein the means for calculating image data for display is adapted to include in the calculated output image data, dependent upon the designated viewing position, image data corresponding to portions of basic frames which are not visible in the reference view of the root image.

4. An interface according to claim 1 wherein the means for calculating image data for display is adapted to include in the calculated image data, dependent upon the distance between the designated viewing position and the root image, image data from frames of the video sequence additional to the basic frames.

5. An interactive interface according to claim 4 wherein the means for calculating image data for display is adapted to select for use in calculating the image for display additional frames chosen based upon criteria specified in additional information stored in association with the root image definition.

6. An interface according to claim 1, wherein the means for calculating image data for display is adapted to calculate output image data corresponding to a different number of frames and/or a displayed image of enlarged or reduced size, dependent upon the distance between the user-designated viewing position and the root image.

7. An interface according to any previous claim, wherein the video sequence includes image data representing one or more selected objects, the means for calculating image data for display being adapted, for each displayed frame containing a respective selected object, selectively to output image data causing display of only that image data which corresponds to the selected object(s), causing, the remainder of the respective displayed frame to appear transparent.

8. An interface according to claim 7, wherein there is provided means for the user to select objects represented in the displayed image, and wherein the means for calculating image data for display is adapted to output image data causing portions of frames to appear transparent in response to the selection of objects by the user.

9. An interface according to any one of claims 1-5 or 8 for a video sequence comprising a main sequence of video frames and at least one additional subsequence of video frames constituting an alternative path to or from a particular video frame in the main sequence, wherein the user can access image information relating to an alternative sub-sequence by designating a viewing position close to a point in the root image corresponding to said particular video frame, the means for calculating image data for display being adapted to graft on to the displayed view of the root image, at the branching point, a secondary root image representing said alternative sub-sequence.

10. An interactive interface according to claim 9, wherein by operation of the viewing position designating means the user can navigate through root images and secondary root images corresponding to the different possible scenarios contained in the video sequence.

11. Apparatus for creation of an interface to a video sequence, the apparatus comprising:

a) means for accessing image information in digital form representing a video sequence;

b) means for creating a root image representing the video sequence, the root image creation means comprising;

i) means for selecting a sub-set of frames from the video sequence, or portions of said sub-set which correspond to objects represented in the video sequence, to serve as basic frames of the root image; and

ii) means for setting the relative placement positions of the basic frames in the root image, and

c) means for associating with the root image routines for changing the displayed view of the root image depending upon a designated viewing position relative to the root image.

12. Apparatus according to claim 11 and further comprising means for identification of objects represented in the image information of the video sequence and for designating objects as selectable by an end user.

13. Apparatus according to claim 11 wherein the means for setting the relative placement positions of the basic frames in the root image comprises means for accessing stored information representing a plurality of templates and means for inputting selection information designating one of the stored templates.

14. Apparatus according to claim 11 wherein the means for setting the relative placement positions of the basic frames in the root image comprises means for detecting motion in the video sequence and means for placing the basic frames within the root image in relative positions which provide a visual representation of said motion.

15. Apparatus according to claim 14 wherein the means for placing the basic frames within the root image is adapted to effect a progressive change in the dimensions of the basic frames in the root image in order to visually represent a zoom-in or zoom-out operation.

16. Apparatus according to claim 11, wherein the means for selecting a sub-set of frames from the video sequence to serve as basic frames of the root image is adapted to select frames as a function of the rate of change of back-round information in the image.

17. Apparatus according to claim 11, and comprising means for inputting parameters constraining the ways in which the displayed view of the root image can be changed depending upon a user-designated viewing position, the constraint parameters being assimilated into the routines associated with the root image by the associating means.

18. Apparatus according to claim 17 wherein the constraint parameter inputting means is adapted to input data identifying the rate at which additional frames should be included in a displayed view of the root image when a user-designated viewing position approaches the root image.

19. Apparatus according to claim 11, and comprising means for creating secondary root images corresponding to additional sub-sequences of video frames constituting alternative paths to or from a particular video frame in the main video sequence.

20. A process for creating an interface corresponding to a predetermined video sequence, comprising the steps of:

a) retrieving the video sequence from a data store;

b) analyzing data corresponding to at least some frames of the video sequence based upon at least one predetermined algorithm;

c) selecting at least some frames from the video sequence based at least in part on frame ranking measure parameters stored in a frame ranking template store;

d) arranging the selected frames to form a succession of frames defining at least in part the interface; and

e) transferring data corresponding to said selected and arranged frames to an interface store.

21. A process according to claim 20 in which said step of selection is conducted automatically.

22. A process according to claim 20 in which said step of selection is conducted at least in part manually.

23. A process according to claim 20 in which said step of selection is based at least in part on the degree of motion of objects in the frames.

24. A process according to claim 20 in which said step of selection is based at least in part on estimating camera motion by separating foreground and background portions of the images.

25. A process according to claim 20 in which said step of selection is based at least in part on vector data in the digital representation of images in the frames.

26. A process according to claim 20 further comprising the steps of:

a) analyzing data corresponding to at least some frames of the video sequence in order to evaluate objects within said frames;

b) selecting at least one object from a plurality of the frames;

c) selecting at least some frames from the video sequence based at least in part on said at least one object;

d) tracking said at least one object through the selected frames; and

e) arranging the selected frames based at least in part on said at least one object.

27. A process according to claim 26 in which said step of selecting said at least one object is conducted automatically.

28. A process according to claim 26 in which said step of selecting said at least one object is conducted at least in part manually.

29. A process according to claim 20 further comprising the steps of:

a) arranging the selected frames to form a succession of frames defining at least in part the interface;

b) selecting at least one additional frame to add to the succession of frames corresponding to a new viewing position based at least in part on certain predetermined factors; and

c) selecting at least one frame to remove from the succession of frames corresponding to a new viewing position based at least in part on certain predetermined factors.

30. A process according to claim 29 in which said step of selecting at least one additional frame to add is conducted automatically.

31. A process according to claim 29 in which said step of selecting at least one additional frame to add is conducted at least in part manually.

32. A process according to claim 29 in which said step of selecting at least one additional frame to remove is conducted automatically.

33. A process according to claim 29 in which said step of selecting at least one additional frame to remove is conducted at least in part manually.

34. A process according to claim 20 further comprising the steps of:

a) arranging the selected frames based at least in part on user specified criteria; and

b) calculating the arrangement of the selected frames based at least in part on predetermined algorithms.

35. A process according to claim 20 further comprising the steps of:

a) creating an interface data file which contains data corresponding at least in part to said interface; and

b) storing said interface data file in a data store.

36. A process according to claim 20 comprising the further steps of:

a) creating a effects detail file which contains data corresponding to the said selection and arrangement of the selected frames; and

b) storing said effects detail file in a data store.

37. A process according to claim 20 comprising the further steps of:

a) creating a video sequence file which contains data corresponding to the selected frames; and

b) storing said video sequence file in a data store.

38. A process according to claim 20 comprising the further steps of:

a) extracting a predetermined set of information from the said interface;

b) creating a script file consisting at least in part of said predetermined set of information; and

c) storing the said script file in a data store.

39. A method for processing an interface corresponding to a predetermined video sequence, comprising the steps of:

a) retrieving the video sequence from a data store;

b) analyzing data corresponding to at least some frames of the video sequence based upon at least one predetermined algorithm;

c) selecting at least some frames from the video sequence based at least in part on frame ranking measure parameters stored in a frame ranking template store;

d) arranging the selected frames to form a succession of frames defining at least in part the interface; and

e) transferring data capable of generating at least one image corresponding to said succession of frames to a viewer;

f) generating an image from a desired perspective using said data capable of generating at least one image; and

g) displaying said image on a display device.

40. A method according to claim 39 further comprising the steps of:

a) generating an image from a second desired perspective using said data capable of generating at least one image; and

b) displaying said image on a display device.

41. A method according to claim 39 in which the step of generating an image from a desired perspective using said data capable of generating at least one image comprises the steps of:

a) determining frames in the interface to dropped or added;

b) calculating the position of all frames relative to each other; and

c) generating an image that renders said frames positioned appropriately relative to each other and that takes into account the predetermined perspective.