Method and apparatus for generation, distribution and display of interactive video content

• US 8,170,392 B2
• Filed: 11/21/2007
• Issued: 05/01/2012
• Est. Priority Date: 11/21/2007
• Status: Active Grant

• Current Assignee: Vdoqwest, Inc.
• Original Assignee: Novafora, Inc.
• Inventors: Rakib, Shlomo Selim, Bronstein, Alexander, Bronstein, Michael, Devictor, Gilles Bruno Marie
• Primary Examiner(s): Zand, Kambiz
• Assistant Examiner(s): Sanders, Stephen
• Application Number:
  US11/944,290
• Publication Number:
  US 20090129755A1
• Time in Patent Office: 1,623 Days
• Field of Search: 386/124
• US Class Current: 386/124
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Proposed is a model for generation and use of metadata for interactive video navigation and video content identification.

Claims

1. A method of operating upon a video stream, the method comprising the steps of:

at a user processing system;

receiving a segment of the video stream, the video stream including video information for representing sequential frames, the video information including video data used substantially only for rendering sequential frame images;

receiving a request for an action corresponding to a spatial-temporal location of displayed images located within the segment of the video stream;

automatically detecting features in a vicinity of the spatial-temporal region of the segment of the video stream, the step of automatically detecting the features within the segment of the video stream operating upon the video data that is used for rendering the sequential frame images;

obtaining a representation map of various regions of the video stream based upon the detected features;

computing an electronic signature based upon the request and the representation map by aggregating a plurality of said features wherein said features and said aggregation are chosen to produce an electronic signature that is insensitive to spatial transformations;

and using the electronic signature to initiate performance of the action.

2. The method according to claim 1 wherein the request for action is an information request, and wherein the step of using the electronic signature includes the steps of:

transmitting the electronic signature to an external server;

receiving information in response to the transmitted electronic signature; and

displaying the information.

3. The method according to claim 1 wherein the request for action is an information request, and wherein the step of using the electronic signature includes the steps of:

performing a local request using the electronic signature to obtain information that is stored entirely within the user processing system;

receiving the information in response to performing the local request;

and displaying the information.

4. The method according to claim 1 wherein the request for action is a content action request, and wherein the step of using the electronic signature includes the step of performing initiates a process related to content within the segment of the video stream using the electronic signature.

5. The method according to claim 4 wherein the content action request is a request to skip, swap, or overlay content corresponding to the electronic signature.

6. The method according to claim 4 wherein the content action request is a request for an interactive session corresponding to the electronic signature, and wherein the step of computing the electronic signature requires an additional request.

7. The method according to claim 1 wherein the step of receiving the request includes at least one of receiving an input from a user controlled controller and receiving a text message from a user controlled device.

8. The method according to claim 1 wherein the video data operated upon in the step of detecting is uncompressed video data.

9. The method according to claim 8 wherein the uncompressed video data is pixel data.

10. The method according to claim 8 wherein the step of obtaining the representation map obtains implicit objects as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the implicit objects from the representation map that is associated with the request.

11. The method according to claim 8 wherein the step of obtaining the representation map obtains video elements as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the video elements from the representation map that is associated with the request.

12. The method according to claim 8 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

13. The method according to claim 1 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

14. The method according to claim 1 wherein the video information further includes audio data and wherein the step of detecting also operates upon the audio data.

15. The method according to claim 1 wherein the step of obtaining the representation map obtains implicit objects as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the implicit objects from the representation map that is associated with the request.

16. The method according to claim 1 wherein the step of obtaining the representation map obtains video elements as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the video elements from the representation map that is associated with the request.

17. The method according to claim 1 wherein the video data operated upon in the step of detecting is compressed video data.

18. The method according to claim 17 wherein the compressed frame data is motion vector data.

19. The method according to claim 17 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

20. The method according to claim 17 wherein the step of obtaining the representation map obtains implicit objects as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the implicit objects from the representation map that is associated with the request.

21. The method according to claim 17 wherein the step of obtaining the representation map obtains video elements as the regions based upon the detected features, and wherein the step of computing the electronic signature computes the electronic signature based upon one of the video elements from the representation map that is associated with the request.

22. The method according to claim 17 wherein the video information further includes audio data and wherein the step of detecting also operates upon the audio data.

23. The method according to claim 1 wherein the step of receiving receives the segment of the video stream from one of a DVD disposed within the user processing system and broadcast content that is received from an external source.

24. The method according to claim 1 wherein the user processing system includes one of a microprocessor, a microcontroller, a DSP and an application specific integrated circuit processor.

47. The method according to claim 1, wherein said features comprise one or more features selected from the group consisting of Harris corner detectors, Scale Invariant Feature Transform (SIFT), motion vectors, spatio-temporal edges, color distribution, texture description, and coeffecients of pixel decomposition in a defined dictionary of wavelets or curvelets, said features are represented as vectors, and said vectors undergo vector quantization or clustering to determine spatio-temporal regions with similar feature vectors.

25. A method of operating upon a video stream that includes sequential frames, the method comprising the steps of:

at a first computing location;

receiving the video stream, the video stream including video information for representing sequential frames, the video information including video data used substantially only for rendering sequential frame images;

obtaining a representation map that represents regions within the video stream, including;

automatically detecting features within the video stream, the step of automatically detecting the features within the video stream operating upon the video data that is used for rendering the sequential frame images;

and computing electronic signatures for different ones of the various regions, respectively, based upon the detected features by aggregating a plurality of said features wherein said features and said aggregation are chosen to produce an electronic signature that is insensitive to spatial transformations;

and identifying an action to associate with each electronic signature.

26. The method according to claim 25 further comprising the steps of:

at a second computing location that is the same or different from the first computing location;

receiving a client electronic signature;

matching the client electronic signature with a particular one of the electronic signatures from the representation map;

identifying the action associated with the particular one of the electronic signatures;

and initiating the action.

27. The method according to claim 26 wherein the step of identifying the action includes the step of associating a user profile with the particular signature so that one action is identified based upon the user profile and the particular signature.

28. The method according to claim 26 wherein the step of identifying the action includes the step of associating a monetary amount with the particular signature so that one action is identified based upon the monetary amount and the particular signature.

29. The method according to claim 26 wherein the particular signature from the step of matching is a closest electronic signature to the client electronic signature that exists within the representation map.

30. The method according to claim 29 wherein the closest signature that exists within the representation map is an identical signature.

31. The method according to claim 25 wherein the video data operated upon in the step of detecting is uncompressed video data.

32. The method according to claim 31 wherein the uncompressed video data is pixel data.

33. The method according to claim 31 wherein the step of obtaining the representation map obtains implicit objects as the regions based upon the detected features, and wherein the step of computing the electronic signatures computes the electronic signatures based upon the respective implicit objects.

34. The method according to claim 31 wherein the step of obtaining the representation map obtains video elements as the regions based upon the detected features, and wherein the step of computing the electronic signatures computes the electronic signatures based upon the respective video elements.

35. The method according to claim 31 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

36. The method according to claim 35 wherein the video information further includes audio data and wherein the step of detecting also operates upon the audio data.

37. The method according to claim 25 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

38. The method according to claim 25 wherein the video information further includes audio data and wherein the step of detecting also operates upon the audio data.

39. The method according to claim 25 wherein the step of obtaining the representation map obtains implicit objects as the regions based upon the detected features, and wherein the step of computing the electronic signatures computes the electronic signatures based upon the respective implicit objects.

40. The method according to claim 25 wherein the step of obtaining the representation map obtains video elements as the regions based upon the detected features, and wherein the step of computing the electronic signatures computes the electronic signatures based upon the respective video elements.

41. The method according to claim 25 wherein the video data operated upon in the step of detecting is compressed video data.

42. The method according to claim 41 wherein the compressed frame data is motion vector data.

43. The method according to claim 41 wherein the video information further includes supplemental video parameter information that is separate from the video data and wherein the step of detecting also operates upon the supplemental video parameter information.

44. The method according to claim 25 wherein the step of receiving receives the segment of the video stream from one of a DVD disposed within the user processing system and broadcast content that is received from an external source.

48. The method according to claim 25, wherein said features comprise one or more features selected from the group consisting of Harris corner detectors, Scale Invariant Feature Transform (SIFT), motion vectors, spatio-temporal edges, color distribution, texture description, and coeffecients of pixel decomposition in a defined dictionary of wavelets or curvelets, said features are represented as vectors, and said vectors undergo vector quantization or clustering to determine spatio-temporal regions with similar feature vectors.

45. A method of operating upon a video stream, the method comprising the steps of:

at each of a user processing system and a server processing system that are physically separated;

receiving the video stream, the video stream including video information for representing sequential frames, the video information including video data used substantially only for rendering sequential frame images;

obtaining a representation map that represents regions within the video stream, including;

automatically detecting features within the video stream, the step of automatically detecting the features within the video stream operating upon the video data that is used for rendering the sequential frame images;

and computing electronic signatures for different ones of the various regions, respectively, based upon the detected features by aggregating a plurality of said features wherein said features and said aggregation are chosen to produce an electronic signature that is insensitive to spatial transformations;

and, wherein;

one of the electronic signatures computed at the user processing system is transmitted to the server processing system;

the one electronic signature is compared to the electronic signatures computed at the server processing system to determine which particular electronic signature computed at the server processing system is a match to the one electronic signature; and

based upon the match, the server processing system initiates an action.

46. The method according to claim 45 wherein the electronic signatures obtained by each of the user processing system and the server processing system are substantially identical.

49. The method according to claim 45, wherein said features comprise one or more features selected from the group consisting of Harris corner detectors, Scale Invariant Feature Transform (SIFT), motion vectors, spatio-temporal edges, color distribution, texture description, and coeffecients of pixel decomposition in a defined dictionary of wavelets or curvelets, said features are represented as vectors, and said vectors undergo vector quantization or clustering to determine spatio-temporal regions with similar feature vectors.