Method and apparatus for audio/data/visual information

• US 8,528,019 B1
• Filed: 11/18/1999
• Issued: 09/03/2013
• Est. Priority Date: 11/18/1999
• Status: Expired due to Fees

• Current Assignee: Sisvel S.p.A. (Fineur International S.A.)
• Original Assignee: Koninklijke Philips N.V.
• Inventors: Dimitrova, Nevenka, McGee, Thomas, Elenbaas, Jan Hermanus, Agnihotri, Lalitha, Jasinschi, Radu, Dagtas, Serhan, Mendelsohn, Aaron S
• Primary Examiner(s): SALCE, JASON P
• Application Number:
  US09/442,960
• Time in Patent Office: 5,038 Days
• Field of Search: 382/118, 725/14, 725/19, 725/28, 725/27, 725 45- 47
• US Class Current: 725/46
• CPC Class Codes: G06F 16/735 Filtering based on additional data, e.g. user or group profiles
• CPC Class Codes: G06F 16/7834 using audio features
• CPC Class Codes: G06F 16/7844 using original textual content or text extracted from visual content or transcript of audio data
• CPC Class Codes: G06F 16/786 using motion, e.g. object motion or camera motion
• CPC Class Codes: H04H 60/27 Arrangements for recording or accumulating broadcast information or broadcast-related information
• CPC Class Codes: H04H 60/46 for recognising users' preferences
• CPC Class Codes: H04H 60/58 of audio determination or detection of speech characteristics in general G10L25/00; speech recognition in general G10L15/00
• CPC Class Codes: H04H 60/59 of video recognising characters or patterns in general G06F18/00, G06V20/00
• CPC Class Codes: H04H 60/65 for using the result on users' side
• CPC Class Codes: H04H 60/72 using electronic programme guides [EPG] focusing on identifying broadcast space time H04H60/39 ; menu type display of EPG in television receivers H04N21/47
• CPC Class Codes: H04N 21/434 Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream demultiplexing of data packets for data networks, e.g. RTP/UDP H04L65/00
• CPC Class Codes: H04N 21/4351 involving reassembling additional data, e.g. rebuilding an executable program from recovered modules
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44224 Monitoring of user activity on external systems, e.g. Internet browsing
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4755 for defining user preferences, e.g. favourite actors or genre retrieval personalisation and generation of user profiles for the retrieval of video data G06F16/739; user profiles in network data switching protocols H04L67/306
• CPC Class Codes: H04N 7/163 by receiver means only

Abstract

A method of selecting, storing and delivering desired audio/data/visual information includes the steps of determining viewing preferences of a viewer and receiving a first group of audio/data/visual signals, for example, broadcast and cable television signals or internet-based signals. Based on the first group of audio/data/visual signals, a second group of audio/data/visual signals, which is a subset of the first group of audio/data/visual signals, is identified. The second group of audio/data/visual signals is selected based on the association of EPG data for each signal with the viewing preferences of the viewer. Content data is then extracted from the second group of audio/data/visual signals and compared with the viewing preferences. The content data may include, for example, closed-captioned text, EPG data, audio information, visual information and transcript information. Based on the comparison of the content data extracted from the second group of audio/data/visual signals with the viewing preferences, audio/data/visual information contained in the second group of audio/data/visual signals which is of interest to the viewer is identified and stored for review at the viewers convenience.

Claims

1. A method of selecting desired audio/data/visual information from a broadcast of a first plurality of programs, the method comprising acts of:

determining viewing preferences of a viewer, the viewing preferences are supplied by the viewer, are automatically updated, and indicate a focus selected from at least one of an identity of a participant in, a type of an event in, a location of the event in, a specific action performed in one or more of the plurality of programs;

receiving on a user apparatus at a viewer'"'"'s location, an EPG describing characteristics of a first plurality of audio/data/visual programs;

selecting a second plurality of unmodified audio/data/visual programs from the first plurality of audio/data/visual programs excluding programs described in the EPG as not having characteristics associated with the viewing preferences;

extracting and indexing closed-captioned or transcribed text and monitoring frequency of occurrence of words in the text from each of the second plurality of unmodified audio/data/visual programs to indicate their focus;

comparing the focus of said viewing preferences with the focus extracted from each of the second plurality of unmodified audio/data/visual programs to identify desired audio/data/visual information;

ranking the desired audio/data/visual information based on conformance of their focus with the focus of the viewing preferences; and

providing access to the ranked desired audio/data/visual information, wherein access to higher ranked desired audio/data/visual information is given priority.

2. The method of selecting desired audio/data/visual information according to claim 1, further comprising acts of:

storing the ranked desired audio/data/visual information; and

retrieving the desired audio/data/visual information when requested by the viewer, higher ranked desired audio/data/visual information is retrieved prior to lower ranked desired audio/data/visual information.

3. The method of selecting desired audio/data/visual information according to claim 1, wherein the comparing act comprises acts of:

extracting content data corresponding to each of said identified second plurality of unmodified audio/data/visual programs;

indexing said extracted content data; and

comparing said indexed content data with said viewing preferences to identify the desired audio/data/visual information.

4. The method of selecting desired audio/data/visual information according to claim 3, wherein the extracted content data comprises at least one of audio content and visual content.

5. The method of selecting desired audio/data/visual information according to claim 4, wherein the extracted content data comprises visual content, wherein the visual content comprises at least one of color, motion, shape, texture of 2-D/3-D objects, cut detection, face detection, motion estimation/segmentation/detection and camera motion.

6. The method of selecting desired audio/data/visual information according to claim 4, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of speaker identification, music classification and dialog detection based on speaker identification.

7. The method of selecting desired audio/data/visual information according to claim 4, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of frequency, pitch, timbre, sound, and melody.

8. The method of selecting desired audio/data/visual information according to claim 1, further comprising an act of integrating at least two of closed-captioning text, EPG data, extracted audio content, extracted visual content, and extracted transcript, wherein said viewing preferences are based on said integrating.

9. The method of selecting desired audio/data/visual information according to claim 8, wherein the integrating act provides for integrating at least three of the closed-captioning text, the EPG data, the extracted audio content, the extracted visual content, and the extracted transcript.

10. The method of selecting desired audio/data/visual information according to claim 9, wherein the integrating act comprises detection of at least one of human faces and scenery.

11. The method of selecting desired audio/data/visual information according to claim 1, wherein said desired audio/data/visual information comprises at least one of broadcast television programs, cable television programs, internet-based programs and data.

12. The method of selecting desired audio/data/visual information according to claim 1, wherein the monitoring act comprises an act of comparing the EPG data for each of the first plurality of audio/data/visual programs with the viewing preferences to identify the second plurality of unmodified audio/data/visual programs.

13. The method of selecting desired audio/data/visual information according to claim 1, wherein the determining act comprises an act of inputting, via at least one of a keypad, keyboard, on-screen display, remote control, touchscreen, verbal commands and touchpad, characteristics of the desired audio/data/visual information that the viewer desires to watch.

14. The method of selecting desired audio/data/visual information according to claim 1, further comprising an act of automatically updating the viewing preferences each time that the viewer accesses at least one of broadcast television programs, cable television programs and internet-based programs from the first plurality of audio/data/visual programs.

15. The method of selecting desired audio/data/visual information according to claim 1, further comprising an act of identifying commercial and noncommercial portions of the second plurality of unmodified audio/data/visual programs.

16. The method of selecting desired audio/data/visual information according to claim 1, wherein the providing act comprises an act of storing at least a portion of said desired audio/data/visual information in memory.

17. The method of selecting desired audio/data/visual information according to claim 1, further comprising an act of notifying the viewer that desired audio/data/visual information has been identified.

18. The method of selecting desired audio/data/visual information according to claim 17, wherein the viewer is notified while the viewer is interacting with audio/data/visual programs.

19. The method of selecting desired audio/data/visual information according to claim 1, wherein the method is concurrently performed for a plurality of viewers.

20. A method of selecting desired audio/data/visual information from a broadcast of a first plurality of programs, the method comprising acts of:

determining by a user apparatus viewing preferences of a user, the viewing preferences are supplied by the user, are automatically updated, and indicate a focus selected from at least one of an identity of a participant in, a type of an event in, a location of the event in, a specific action performed in one or more of the plurality of programs;

selecting a subset from a plurality of audio/data/visual programs on a basis of a characteristics described in an EPG received by the user apparatus at a users location, wherein the programs in the subset excludes programs described in the EPG as not having characteristics associated with the viewing preferences;

extracting and indexing closed-captioned or transcribed text and monitoring frequency of occurrence of words in the text from each of the second plurality of unmodified audio/data/visual programs to indicate their focus;

comparing the focus of said viewing preferences with the focus extracted from each program of the subset of the plurality of unmodified audio/data/visual programs to identify desired audio/data/visual information;

ranking the desired audio/data/visual information based on conformance of their focus with the focus of the viewing preferences; and

providing access to the ranked desired audio/data/visual information, wherein access to higher ranked desired audio/data/visual information is given priority.

21. The method of selecting desired audio/data/visual information according to claim 20, further comprising an act of selecting at least one of the subset of the plurality of unmodified audio/data/visual programs for comparison with said viewing preferences.

22. The method of selecting desired audio/data/visual information according to claim 21, wherein the selection of at least one of the plurality of unmodified audio/data/visual programs is performed at least one of randomly, sequentially, and periodically.

23. The method of selecting desired audio/data/visual information according to claim 20, further comprising acts of:

storing the ranked desired plurality of audio/data/visual information; and

retrieving the desired plurality of audio/data/visual information, wherein the higher ranked desired audio/data/visual information is retrieved prior to retrieving the lower ranked desired audio/data/visual information.

24. The method of selecting desired audio/data/visual information according to claim 20, wherein the comparing act comprises acts of:

extracting content data corresponding to each program of said subset of the plurality of unmodified audio/data/visual programs;

indexing said extracted content data; and

comparing said indexed content data with said viewing preferences to identify the desired audio/data/visual information.

25. The method of selecting desired audio/data/visual information according to claim 24, wherein the extracted content data comprises at least one of audio content and visual content.

26. The method of selecting desired audio/data/visual information according to claim 25, wherein the extracted content data comprises visual content, wherein the visual content comprises at least one of color, motion, shape, texture of 2-D/3-D objects, cut detection, face detection, motion estimation/segmentation/detection and camera motion.

27. The method of selecting desired audio/data/visual information according to claim 25, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of speaker identification, music classification and dialog detection based on speaker identification.

28. The method of selecting desired audio/data/visual information according to claim 25, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of frequency, pitch, timbre, sound, and melody.

29. The method of selecting desired audio/data/visual information according to claim 20, further comprising an act of integrating at least two of closed-captioning text, EPG data, extracted audio content, extracted visual content, and extracted transcript, wherein said viewing preferences are updated based on said integrating.

30. The method of selecting desired audio/data/visual information according to claim 29, wherein the integrating act provides for integrating at least three of the closed-captioning text, the EPG data, the extracted audio program, the extracted visual content, and the extracted transcript.

31. The method of selecting desired audio/data/visual information according to claim 30, wherein the integrating act comprises detection of at least one of human faces and scenery.

32. The method of selecting desired audio/data/visual information according to claim 20, wherein said desired audio/data/visual information comprises at least one of broadcast television programs, cable television programs, internet-based programs, and data.

33. The method of selecting desired audio/data/visual information according to claim 20, wherein the comparing act comprises an act of comparing electronic programming guide (EPG) data for each program of the subset of the plurality of unmodified audio/data/visual programs with the viewing preferences to identify a second plurality of unmodified audio/data/visual programs which are associated with the viewing preferences.

34. The method of selecting desired audio/data/visual information according to claim 20, wherein the determining act comprises an act of inputting, via at least one of a keypad, keyboard, on-screen display, remote control, touchscreen, verbal commands and touchpad, characteristics of audio/data/visual information that the user desires.

35. The method of selecting desired audio/data/visual information according to claim 20, wherein the determining act comprises an act of monitoring the user'"'"'s interaction with the plurality of audio/data/visual signals to formulate characteristics of audio/data/visual information that the user desires.

36. The method of selecting desired audio/data/visual information according to claim 20, further comprising an act of automatically updating the viewing preferences each time the user accesses at least one of broadcast television programs, cable television programs, internet-based programs and data.

37. The method of selecting desired audio/data/visual information according to claim 20, further comprising an act of identifying commercial and noncommercial portions of the plurality of unmodified audio/data/visual programs.

38. The method of selecting desired audio/data/visual information according to claim 20, wherein the providing act comprises an act of storing at least a portion of said ranked desired audio/data/visual information in a memory, a higher ranked desired audio/data/visual information is stored prior to storing a lower ranked desired audio/data/visual information.

39. The method of selecting desired audio/data/visual information according to claim 20, further comprising an act of notifying the user that desired audio/data/visual information has been identified.

40. The method of selecting desired audio/data/visual information according to claim 38, wherein the user is notified while the user is interacting with audio/data/visual programs.

41. The method of selecting desired audio/data/visual information according to claim 20, wherein the method is concurrently performed for a plurality of users.

42. An audio/data/visual program selection system comprising:

an input device for providing viewing preferences of a viewer, the viewing preferences are supplied by the viewer, are automatically updated, and indicate a focus selected from at least one of an identity of a participant in, a type of an event in, a location of the event in, a specific action performed in one or more of the plurality of programs; and

an information selector for;

receiving at a viewer'"'"'s location, an EPG describing characteristics of a first plurality of audio/data/visual programs;

selecting a second plurality of unmodified audio/data/visual programs from the first plurality of audio/data/visual programs excluding programs described in the EPG as not having characteristics associated with the viewing preferences;

extracting and indexing closed-captioned or transcribed text and monitoring frequency of occurrence of words in the text from each of the second plurality of unmodified audio/data/visual programs to indicate their focus;

comparing the focus of said viewing preferences with the focus extracted from each of the second plurality of unmodified audio/data/visual programs to identify desired audio/data/visual information;

ranking the desired audio/data/visual information based on conformance of their focus with the focus of the viewing preferences; and

providing access to the ranked identified audio/data/visual information, access to higher ranked desired audio/data/visual information is given priority.

43. The audio/data/visual program selection system according to claim 42, wherein the information selector also stores the ranked desired audio/data/visual information, and retrieves the desired audio/data/visual information upon request by the viewer, the higher ranked desired audio/data/visual information is retrieved prior to retrieving the lower ranked desired audio/data/visual information.

44. The audio/data/visual program selection system according to claim 42, wherein said desired plurality of audio/data/visual information comprises at least one of broadcast television programs, cable television programs, internet-based programs and data.

45. The audio/data/visual program selection system according to claim 42, wherein the information selector compares the EPG data for each of the first plurality of audio/data/visual programs with the viewing preferences to identify the second plurality of unmodified audio/data/visual programs which are associated with the viewing preferences.

46. The audio/data/visual program selection system according to claim 42, wherein the input device comprises at least one of a keypad, keyboard, on-screen display, remote control, touchscreen, audio sensor and touchpad.

47. The audio/data/visual program selection system according to claim 42, wherein the information selector monitors the viewing habits of the viewer with the first plurality of audio/data/visual programs to formulate characteristics of audio/data/visual information that the viewer desires to watch.

48. The audio/data/visual program selection system according to claim 42, wherein the information selector:

extracts content data from each of the second plurality of unmodified audio/data/visual programs;

indexes the extracted content data; and

compares the indexed content data with the viewing preferences to identify the desired audio/data/visual information.

49. The audio/data/visual program selection system according to claim 48, wherein the extracted content data comprises at least one of audio content and video content.

50. The audio/data/visual program selection system according to claim 49, wherein the extracted content data comprises video content, wherein the video content comprises at least one of color, motion, shape, texture of 2-D/3-D objects, cut detection, face detection, motion estimation/segmentation/detection and camera motion.

51. The audio/data/visual program selection system according to claim 49, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of speaker identification, music classification and dialog detection based on speaker identification.

52. The audio/data/visual program selection system according to claim 49, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of frequency, pitch, timbre, sound, and melody.

53. The audio/data/visual program selection system according to claim 42, wherein said viewing preferences is based on integrating at least two of closed-captioning text, EPG data, audio content, video content, and transcript.

54. The audio/data/visual program selection system according to claim 53, wherein the integrating provides for integrating at least three of the closed-captioning text, the EPG data, the extracted audio content, the extracted video content, and the extracted transcript.

55. The audio/data/visual program selection system according to claim 54, wherein the integrating comprises detection of at least one of human faces and scenery.

56. The audio/data/visual program selection system according to claim 42, wherein the information selector determines commercial and noncommercial portions of the second plurality of audio/data/visual programs.

57. The audio/data/visual program selection system according to claim 42, further comprising a memory, operatively coupled to the information selector, for storing at least a portion of said desired audio/data/visual information.

58. The audio/data/visual program selection system according to claim 42, wherein the information selector automatically updates the viewer preferences each time that the viewer accesses at least one of broadcast television programs, cable television programs, internet-based programs and data.

59. The audio/data/visual program selection system according to claim 42, further comprising:

an audio/data/visual program receiver for receiving the first plurality of audio/data/visual programs; and

a demultiplexer operatively coupled between the audio/data/visual program receiver and the audio/data/visual information selection, storage and delivery device for demultiplexing the first plurality of audio/data/visual programs and providing a demultiplexed first plurality of audio/data/visual programs as an output signal.

60. The audio/data/visual program selection system according to claim 42, wherein the information selector notifies the viewer that the desired audio/data/visual information has been identified.

61. The audio/data/visual program selection system according to claim 60, wherein the viewer is notified while the viewer is interacting with audio/data/visual programs.

62. The audio/data/visual program selection system according to claim 42, wherein the system is centrally located and operatively coupled for use by a plurality of viewers.

63. An audio/data/visual program selection system comprising:

an input device for providing viewing preferences of a user, the viewing preferences are supplied by the user, are automatically updated, and indicate a focus selected from at least one of an identity of a participant in, a type of an event in, a location of the event in, a specific action performed in one or more of the plurality of programs; and

an information selector for;

selecting a subset from a plurality of audio/data/visual programs on a basis of a characteristics described in an EPG received at a users location, wherein the programs in the subset excludes programs described in the EPG as not having characteristics associated with the viewing preferences;

extracting and indexing closed-captioned or transcribed text and monitoring frequency of occurrence of words in the text from each of the second plurality of unmodified audio/data/visual programs to indicate their focus;

comparing the focus of said viewing preferences and the focus extracted from each of the subset of audio/data/visual programs to identify desired audio/data/visual information;

ranking the desired audio/data/visual information based on conformance of their focus with the focus of the viewing preferences; and

providing access to the desired audio/data/visual information, wherein access to higher ranked desired audio/data/visual information is given priority.

64. The audio/data/visual program selection system according to claim 63, further comprising a memory, operatively coupled to the information selector, for storing at least a portion of said ranked desired audio/data/visual information, wherein a higher ranked desired audio/data/visual information is stored prior to storing a lower ranked desired audio/data/visual information.

65. The audio/data/visual program selection system according to claim 63, further comprising:

an audio/data/visual program receiver for receiving the first set of the plurality of audio/data/visual programs; and

a demultiplexer operatively coupled between the audio/data/visual programs receiver and the information selector for demultiplexing the first set of the plurality of audio/data/visual programs and providing a demultiplexed first plurality of audio/data/visual programs as an output signal.

66. The audio/data/visual program selection system according to claim 63, wherein the information selector also stores the desired ranked audio/data/visual information, and retrieves the desired audio/data/visual information upon request by the user, wherein the higher ranked programs are retrieved prior to retrieving the lower ranked programs.

67. The audio/data/visual program selection system according to claim 63, wherein said desired plurality of audio/data/visual information comprises at least one of broadcast television programs, cable television programs, internet-based programs and data.

68. The audio/data/visual program selection system according to claim 63, wherein the input device comprises at least one of a keypad, keyboard, on-screen display, remote control, touchscreen, audio sensor and touchpad.

69. The audio/data/visual program selection system according to claim 63, wherein the information selector monitors the user'"'"'s interaction with the first set of the plurality of audio/data/visual programs to formulate characteristics of audio/data/visual information that the users desire.

70. The audio/data/visual program selection system according to claim 63, wherein the information selector:

extracts content data from each of the first set of the plurality of audio/data/visual programs;

indexes the extracted content data; and

compares the indexed content data and the identified subset of audio/data/visual programs with the viewing preferences to identify the desired plurality of audio/data/visual information.

71. The audio/data/visual program selection system according to claim 70, wherein the extracted content data comprises at least one of audio content and video content.

72. The audio/data/visual program selection system according to claim 71, wherein the extracted content data comprises video content, wherein the video content comprises at least one of color, motion, shape, texture of 2-D/3-D objects, cut detection, face detection, motion estimation/segmentation/detection and camera motion.

73. The audio/data/visual program selection system according to claim 71, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one speaker identification, music classification and dialog detection based on speaker identification.

74. The audio/data/visual program selection system according to claim 71, wherein the extracted content data comprises audio content, wherein the audio content comprises at least one of frequency, pitch, timbre, sound, and melody.

75. The audio/data/visual program selection system according to claim 71, wherein said viewing preferences are updated based on integrating at least two of the closed-captioning text, the EPG data, the audio content, the video content, and the transcript.

76. The audio/data/visual program selection system according to claim 75, wherein the integration provides for integrating at least three of the closed-captioning text, the EPG data, the extracted audio content, the extracted video content, and the extracted transcript.

77. The audio/data/visual program selection system according to claim 76, wherein the integrating comprises detection of at least one of human faces and scenery.

78. The audio/data/visual program selection system according to claim 63, wherein the information selector determines commercial and noncommercial portions of the plurality of audio/data/visual programs.

79. The audio/data/visual program selection system according to claim 63, wherein the information selector automatically updates the viewing preferences each time the user accesses at least one of broadcast television programs, cable television programs, internet-based programs and data from the first set of the plurality of audio/data/visual programs.

80. The audio/data/visual program selection system according to claim 63, wherein the information selector notifies the user that the desired audio/data/visual information has been identified.

81. The audio/data/visual program selection system according to claim 80, wherein the user is notified while the user is interacting with audio/data/visual information.

82. The audio/data/visual program selection system according to claim 63, wherein the system is centrally located and operatively coupled for use by plurality of users.