Interactive system

• US 7,158,676 B1
• Filed: 01/28/2000
• Issued: 01/02/2007
• Est. Priority Date: 02/01/1999
• Status: Expired due to Fees

• Current Assignee: Emuse Media Limited
• Original Assignee: Emuse Media Limited
• Inventors: Rainsford, Patrick J.
• Primary Examiner(s): Wu; Jingge
• Assistant Examiner(s): CARTER, AARON W
• Application Number:
  US09/889,966
• Time in Patent Office: 2,531 Days
• Field of Search: 382/305, 345/951, 348/700, 348/701, 386/46, 386/52, 715716-726, 725 37- 61
• US Class Current: 382/190
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/2542 for selling goods, e.g. TV shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/47815 Electronic shopping payment schemes, payment architectures or payment protocols for electronic shopping systems G06Q20/12
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02
• CPC Class Codes: H04N 21/8545 for generating interactive applications
• CPC Class Codes: H04N 21/858 Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 21/8583 by creating hot-spots
• CPC Class Codes: H04N 21/8586 by using a URL processing chained hypermedia data for information retrieval G06F16/94; information retrieval from the Internet by using URLs G06F16/955; URL in broadcast information H04H20/93; Web-based protocols H04L67/02
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 7/52 Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99948 Application of database or data structure, e.g. distributed, multimedia, or image

Abstract

An interactive system provides a video program signal and generates interactive content data to be associated with at least one object within a frame of the video program. The interactive content data is embedded with the object and the object is tracked through a sequence of frames and the interactive content data is embedded into each one of the frames. The program frames with the embedded data are multiplexed with video and audio signals and may be broadcast. A receiver identifies an object of interest and the embedded data associated with the object is retrieved. The embedded data may be used for e-commerce.

Claims

1. An apparatus for associating interactive data with an object in a digital video programme including means for providing a digital video programme having plural individual frames at least some of which incorporate said object, means for parsing the video programme by identifying separate shots in the video programme to produce an edit list, means for identifying shots containing related content to form a sequence of shots containing related content, means for selecting at least one key-frame within each sequence of shots, means for selecting said object within the key-frame with which the interactive data is to be associated, means for extracting attributes of the object from the key-frame, means for associating the interactive data with the object in the key-frame, means for utilising the attributes of the object for tracking the object through the sequence of shots, whereby said interactive data is associated with the object in the sequence of shots and said interactive data is embedded with data representative of said object in a data sequence.

2. An apparatus as claimed in claim 1, wherein the means for identifying shots containing related content to form a sequence of shots containing related content includes means for producing a hierarchy of groups of shots.

3. An apparatus as claimed in claim 1, wherein said means for parsing include means for inputting criteria to be used to recognise a change of shot.

4. An apparatus as claimed in claim 1, wherein the means for extracting attributes of the object includes means for isolating the object within a boundary formed on the frame, means for performing edge detection within the boundary to identify and locate edges of said object, and storing means for storing a geometric model of said object.

5. An apparatus as claimed in claim 1, wherein said means for extracting attributes of said object also includes means for recording at least one of the attributes of shape, size, position, colour, texture, intensity gradient of said object, and time series statistics based on said attributes.

6. An apparatus as claimed in claim 1, wherein said means for extracting attributes of said object includes means for comparing said attributes of said object with attributes of objects previously stored to determine whether the object is distinguishable therefrom, and when said object is determined not to be distinguishable, providing means for redefining the object.

7. An apparatus as claimed in claim 1, wherein said means for extracting said attributes includes means for comparing the location in the frame of said object with the location of objects already stored for that frame to determine whether that object is distinguishable therefrom, and where the location of said object is not distinguishable from the location of another object providing means for assigning rank to the objects to determine which object will be associated with that location.

8. An apparatus, as claimed in claim 1, wherein the means for utilising the attributes of the object for tracking the object includes means for updating the stored attributes of the object change from frame to frame.

9. An apparatus as claimed in claim 1, wherein said tracking means utilising the attributes of the object for tracking the object includes plural algorithm means for calculation of independent tracks of objects for use depending on the visual complexity of a sequence to automatically track objects in different types of visual environment.

10. An apparatus as claimed in claim 1, wherein said tracking means for utilising the attributes of the object for tracking includes means for converting all the frames to be tracked to a low-level representation, means for determining the position of each object in the frames by minimising a distance measure to locate each object in each frame, means for processing the positions of said object to smooth over occlusions and the entrances and exits of objects into and out of said frames, and means for reviewing the object within a tracked sequence and for correcting the location attributes of any misplaced objects.

11. An apparatus, as claimed in claim 1, wherein the means for associating interactive data with the object in the key-frame includes means for providing a database of different types of data including one or more of URLs, HTML pages, video clips, audio clips, text files and multimedia catalogues, and means for selecting said interactive content data from the database to associate with said object.

12. An apparatus, as claimed in claim 1, wherein the means for associating interactive data with the object in the key-frame produces said data sequence using means for determining whether the embedded interactive content data is frame synchronous data associated with object positions, shapes, ranks and pointers in a frame, or group-synchronous data associated with all the objects in a group, or is data to be streamed just in time, wherein means are provided for associating frame synchronous data with the corresponding frame, means are provided for associating group synchronous data with the frame at which a group changes, and means are provided for streaming just in time data to a user before it is required to be associated with the corresponding objects.

13. An apparatus as claimed in claim 1, wherein means are provided to associate different interactive content data with respectively different objects.

14. A method for associating interactive data with an object in a digital video programme including the steps of:

a) providing a digital video programme having a plurality of individual frames at least some of which incorporate said object with which interactive data is to be associated,

b) parsing the video programme by identifying separate shots in the video programme to produce an edit list,

c) identifying, from the edit list, shots containing related content to form a sequence of shots containing related content,

d) selecting at least one key-frame containing the object from within the sequence of shots,

e) locating said object within the at least one key-frame,

f) extracting attributes of the object from the at least one key-frame,

g) associating the interactive data with the object in the at least one key-frame,

h) tracking the object through the sequence of shots utilising the attributes of the object,

i) associating said interactive data with the object in frames in the sequence of shots, and

j) embedding said interactive data with data representative of said object in a data sequence representative of the digital video programme.

15. A method as claimed in claim 14, wherein step b) includes the step of inputting criteria (420) to be used to recognise a change of shot.

16. A method as claimed in claim 14, wherein step c) includes the step of producing a hierarchy of groups of sequences of shots.

17. A method as claimed in claim 14, wherein step e) includes the steps of:

isolating the object (600) within a boundary (510) formed on the frame, performing edge detection within the boundary to identify and locate edges of said object, and step f) includes storing a geometric model of said object.

18. A method as claimed in claim 14, wherein step f) includes the step of recording at least one of the attributes of shape, size, position, colour, texture, intensity gradient of said object, and time series statistics based on said attributes.

19. A method as claimed in claim 14, wherein step f) includes the step of comparing said attributes of said object with attributes of objects previously stored to determine whether the object is distinguishable therefrom, and when said object is determined not to be distinguishable, the step of re-defining the object.

20. A method as claimed in claim 14, wherein step f) includes the step of comparing the location in the frame of said object with the location of objects already stored for that frame to determine whether that object is distinguishable therefrom, and where the location of said object is not distinguishable from the location of another object, the step of assigning rank to the objects to determine which object will be associated with that location.

21. A method as claimed in claim 14, wherein step h) includes the step of updating the stored attributes of the object as the attributes of the object change from frame to frame.

22. A method as claimed in claim 14, wherein step h) includes the step of using plural algorithm means for calculation of independent tracks of objects for use depending on the visual complexity of a sequence automatically to track said objects in different types of visual environment.

23. A method as claimed in claim 14, wherein step h) includes the steps of converting all the frames to be tracked to a low-level representation, determining the position of each object in the frames by minimising a distance measure to locate each object in each frame, processing the positions of said object to smooth over occlusions and the entrances and exits of objects into and out of said frames, reviewing the object within a tracked sequence and correcting the location attributes of any misplaced objects.

24. A method as claimed in claim 14, wherein step g) includes the steps of providing a database of different types of data including one or more of URLs, HTML pages, video clips, audio clips, text files and multimedia catalogues, and selecting said interactive content data from the database to associate with said object.

25. A method as claimed in claim 14, wherein step j) includes determining whether the embedded interactive content data is frame synchronous data associated with object positions, shapes, ranks and pointers in a frame, or group—

synchronous data associated with all the objects in a group, or is data to be streamed just in time, and associating frame synchronous data with the corresponding frame, associating group synchronous data with the frame at which a group changes, and streaming just in time data to a user before it is required to be associated with the corresponding objects, respectively.

26. A method as claimed in claim 14, wherein in steps d) to j) different interactive content data are associated with respectively different objects.

27. A computer readable medium embodying a computer program comprising code means for performing all the steps of the method of claim 14 when the program is run on one or more computers.

28. An apparatus for associating interactive data with an object in a digital video programme including means for providing a digital video programme having plural individual frames at least some of which incorporate said object, means for parsing the video programme by identifying separate shots in the video programme to produce an edit list, means for identifying shots containing related content to form a sequence of shots containing related content, means for selecting at least one key-frame within each sequence of shots, means for selecting said object within the key-frame with which the interactive data is to be associated, means for extracting attributes of the object from the key-frame, means for associating the interactive data with the object in the key-frame, means for utilising the attributes of the object for tracking the object through the sequence of shots, whereby said interactive data is associated with the object in the sequence of shots and said interactive data is embedded with data representative of said object in a data sequence, wherein said means for extracting attributes of said object includes means for comparing said attributes of said object with attributes of objects previously stored to determine whether the object is distinguishable therefrom, and when said object is determined not to be distinguishable, providing means for redefining the object.

29. An apparatus for associating interactive data with an object in a digital video programme including means for providing a digital video programme having plural individual frames at least some of which incorporate said object, means for parsing the video programme by identifying separate shots in the video programme to produce an edit list, means for identifying shots containing related content to form a sequence of shots containing related content, means for selecting at least one key-frame within each sequence of shots, means for selecting said object within the key-frame with which the interactive data is to be associated, means for extracting attributes of the object from the key-frame, means for associating the interactive data with the object in the key-frame, means for utilising the attributes of the object for tracking the object through the sequence of shots, whereby said interactive data is associated with the object in the sequence of shots and said interactive data is embedded with data representative of said object in a data sequence, wherein said means for extracting said attributes includes means for comparing the location in the frame of said object with the location of objects already stored for that frame to determine whether that object is distinguishable therefrom, and where the location of said object is not distinguishable from the location of another object providing means for assigning rank to the objects to determine which object will be associated with that location.

30. An apparatus for associating interactive data with an object in a digital video programme including means for providing a digital video programme having plural individual frames at least some of which incorporate said object, means for parsing the video programme by identifying separate shots in the video programme to produce an edit list, means for identifying shots containing related content to form a sequence of shots containing related content, means for selecting at least one key-frame within each sequence of shots, means for selecting said object within the key-frame with which the interactive data is to be associated, means for extracting attributes of the object from the key-frame, means for associating the interactive data with the object in the key-frame, means for utilising the attributes of the object for tracking the object through the sequence of shots, whereby said interactive data is associated with the object in the sequence of shots and said interactive data is embedded with data representative of said object in a data sequence, wherein said tracking means utilising the attributes of the object for tracking the object includes plural algorithm means for calculation of independent tracks of objects for use depending on the visual complexity of a sequence to automatically track objects in different types of visual environment.

31. An apparatus for associating interactive data with an object in a digital video programme including means for providing a digital video programme having plural individual frames at least some of which incorporate said object, means for parsing the video programme by identifying separate shots in the video programme to produce an edit list, means for identifying shots containing related content to form a sequence of shots containing related content, means for selecting at least one key-frame within each sequence of shots, means for selecting said object within the key-frame with which the interactive data is to be associated, means for extracting attributes of the object from the key-frame, means for associating the interactive data with the object in the key-frame, means for utilising the attributes of the object for tracking the object through the sequence of shots, whereby said interactive data is associated with the object in the sequence of shots and said interactive data is embedded with data representative of said object in a data sequence, wherein said tracking means for utilising the attributes of the object for tracking includes means for converting all the frames to be tracked to a low-level representation, means for determining the position of each object in the frames by minimising a distance measure to locate each object in each frame, means for processing the positions of said object to smooth over occlusions and the entrances and exits of objects into and out of said frames, and means for reviewing the object within a tracked sequence and for correcting the location attributes of any misplaced objects.

32. A method for associating interactive data representative of an object with a digital video programme including the steps of:

a) providing a digital video programme having a plurality of individual frames at least some of which incorporate said object with which data is to be associated,

b) parsing the video programme by identifying separate shots in the video programme to produce an edit list,

c) identifying, from the edit list, shots containing related content to form a sequence of shots containing related content,

d) selecting at least one key-frame containing the object from within the sequence of shots,

e) locating said object within the at least one key-frame,

f) extracting attributes of the object from the at least one key-frame including comparing said attributes of said object with attributes of objects previously stored to determine whether the object is distinguishable therefrom, and when said object is determined not to be distinguishable, re-defining the object,

g) associating interactive data with the object in the at least one key-frame,

h) tracking the object through the sequence of shots utilising the attributes of the object,

i) associating said interactive data with the object in frames in the sequence of shots, and

j) embedding said interactive data with data representative of said object in a data sequence representative of the digital video programme.

33. A method for associating interactive data representative of an object with a digital video programme including the steps of:

a) providing a digital video programme having a plurality of individual frames at least some of which incorporate said object with which data is to be associated,

b) parsing the video programme by identifying separate shots in the video programme to produce an edit list,

c) identifying, from the edit list, shots containing related content to form a sequence of shots containing related content,

d) selecting at least one key-frame containing the object from within the sequence of shots,

e) locating said object within the at least one key-frame,

f) extracting attributes of the object from the at least one key-frame, including comparing the location in the frame of said object with the location of objects already stored for that frame to determine whether that object is distinguishable therefrom, and where the location of said object is not distinguishable from the location of another object, the step of assigning rank to the objects to determine which object will be associated with that location,

g) associating interactive data with the object in the at least one key-frame,

h) tracking the object through the sequence of shots utilising the attributes of the object,

i) associating said interactive data with the object in frames in the sequence of shots, and

j) embedding said interactive data with data representative of said object in a data sequence representative of the digital video programme.

34. A method for associating interactive data representative of an object with a digital video programme including the steps of:

a) providing a digital video programme having a plurality of individual frames at least some of which incorporate said object with which data is to be associated,

b) parsing the video programme by identifying separate shots in the video programme to produce an edit list,

c) identifying, from the edit list, shots containing related content to form a sequence of shots containing related content,

d) selecting at least one key-frame containing the object from within the sequence of shots,

e) locating said object within the at least one key-frame,

f) extracting attributes of the object from the at least one key-frame,

g) associating interactive data with the object in the at least one key-frame,

h) tracking the object through the sequence of shots utilising the attributes of the object, including using plural algorithm means for calculation of independent tracks of objects for use depending on the visual complexity of a sequence automatically to track said objects in different types of visual environment,

i) associating said interactive data with the object in frames in the sequence of shots, and

j) embedding said interactive data with data representative of said object in a data sequence representative of the digital video programme.

35. A method for associating interactive data representative of an object with a digital video programme including the steps of:

a) providing a digital video programme having a plurality of individual frames at least some of which incorporate said object with which data is to be associated,

b) parsing the video programme by identifying separate shots in the video programme to produce an edit list,

c) identifying, from the edit list, shots containing related content to form a sequence of shots containing related content,

d) selecting at least one key-frame containing the object from within the sequence of shots,

e) locating said object within the at least one key-frame,

f) extracting attributes of the object from the at least one key-frame,

g) associating interactive data with the object in the at least one key-frame,

h) tracking the object through the sequence of shots utilising the attributes of the object, including converting all the frames to be tracked to a low-level representation, determining the position of each object in the frames by minimising a distance measure to locate each object in each frame, processing the positions of said object to smooth over occlusions and the entrances and exits of objects into and out of said frames, reviewing the object within a tracked sequence and correcting the location attributes of any misplaced objects,

i) associating said interactive data with the object in frames in the sequence of shots, and

j) embedding said interactive data with data representative of said object in a data sequence representative of the digital video programme.