Method and system for browsing, searching and sharing of personal video by a non-parametric approach

• US 9,189,137 B2
• Filed: 03/07/2011
• Issued: 11/17/2015
• Est. Priority Date: 03/08/2010
• Status: Active Grant

• Current Assignee: Vimeo, Inc. (Match Group, Inc.)
• Original Assignee: Magisto Ltd. (Match Group, Inc.)
• Inventors: Boiman, Oren, Rav-Acha, Alex
• Primary Examiner(s): HUANG, MIRANDA M
• Application Number:
  US13/041,457
• Publication Number:
  US 20110218997A1
• Time in Patent Office: 1,716 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/732 Query formulation
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/783 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/7837 using objects detected or recognised in the video content
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/329 on a disc [VTOC]
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

A method for determining a predictability of a media entity portion, the method includes: receiving or generating (a) reference media descriptors, and (b) probability estimations of descriptor space representatives given the reference media descriptors; wherein the descriptor space representatives are representative of a set of media entities; and calculating a predictability score of the media entity portion based on at least (a) the probability estimations of the descriptor space representatives given the reference media descriptors, and (b) relationships between the media entity portion descriptors and the descriptor space representatives. A method for processing media streams, the method may include: applying probabilistic non-parametric process on the media stream to locate media portions of interest; and generating metadata indicative of the media portions of interest.

Claims

1. A method comprising:

obtaining a query being a media portion;

obtaining at least one media reference set being a set of media portions;

obtaining a descriptors reference set being a set of local descriptors extracted from the at least one media reference set, wherein the descriptors are defined over a descriptors space;

extracting descriptor space representatives from the descriptors reference set;

calculating a non-parametric probability density estimation of the descriptor space representatives, given at least one of the media reference sets;

extracting a set of query descriptors from the query;

calculating for at least some of the query descriptors, at least one of their nearest descriptor space representatives;

calculating a non-parametric probability density estimation for at least some of the query descriptors, given at least one of the reference sets, using a weighted average of the non-parametric probability density estimations of the nearest descriptor space representatives, wherein the weights are determined based on the respective distances in the descriptor space, between the query descriptors and the corresponding descriptor space representatives;

calculating a non-parametric probability density estimation of the query, given at least one of the reference sets, by applying an operator to at least some of the calculated non-parametric probability density estimations of the query descriptors;

determining a saliency level of the query, based on the calculated non-parametric probability density estimation of the query given the at least one reference set, so that a lower level of the calculated non-parametric probability density estimation is indicative of a higher level of saliency of the query;

using the saliency level of the query as a criteria for selecting a sequence of media portions of interest; and

generating an audiovisual media stream based on at east some of the selected media portions of interest,

wherein any of the obtaining, the calculating, the determining, the extracting, and the generating are executed by a computer processor.

2. The method according to claim 1, wherein the media portions are video portions and the selecting is video selecting, respectively.

3. The method according to claim 1, wherein said operator is a multiplication operator.

4. The method according to claim 1, wherein said extracting descriptor space representatives is carried out by at least one of:

sampling or vector quantization.

5. The method according to claim 1, wherein at least one of the representative descriptors that is used to compute the non-parametric probability density estimation of the query given the at least one reference set is not derived from this reference set.

6. The method according to claim 1, wherein at least one of the representative descriptors is used to compute a plurality of non-parametric probability density estimations of the query given a plurality of reference sets.

7. The method according to claim 6, wherein the query and the reference sets are both portions of the same video.

8. The method according to claim 1, wherein the at least one media reference set comprises a plurality of different media reference sets and wherein the method further comprises calculating a conditioned likelihood of the query to belong to one of the plurality of the different media reference sets, by comparing the non-parametric probability density estimations of the query given the plurality of the different media reference sets.

9. The method according to 8, further comprising computing a posteriori probability of the query to belong to at least one of the reference media sets.

10. The method according to claim 1, wherein the descriptors are local image or video descriptors.

11. system comprising:

an input module configured to obtain;

a query being a media portion;

at least one media reference set being a set of media portions; and

descriptors reference set being a set of descriptors extracted from the at least one media reference set, wherein the descriptors are defined over a descriptors space; and

a computer processor configured to;

extract descriptor space representatives from the descriptors reference set;

calculate a non-parametric probability density estimation of the descriptor space representatives, given at least one of the media reference sets;

extract a set of query descriptors from the query;

calculate for at least some of the query descriptors, at least one of their nearest descriptor space representatives;

calculate a non-parametric probability density estimation for at least some of the query descriptors, given at least one of the reference sets, using a weighted average of the non-parametric probability density estimations of the nearest descriptor space representatives, wherein the weights are determined based on the respective distances in the descriptor space, between the query descriptors and the corresponding descriptor space representatives;

calculate a non-parametric probability density estimation of the query, given at least one of the reference sets, by applying an operator to at least some of the calculated non-parametric probability density estimations of the query descriptors;

determine a saliency level of the query, based on the calculated non-parametric probability density estimation of the query given the at least one reference set, so that a lower level of the calculated non-parametric probability den estimation is indicative of a higher level of saliency of the query;

use the saliency level of the query as a criteria for selecting a sequence of media portions of interest; and

generate an audiovisual media stream based on at least some of the selected media portions s of interest.

12. A tangible computer program product comprising:

a non-transitory computer readable storage medium having computer readable program embodied therewith, the computer readable program comprising;

computer readable program configured to obtain a query being a media portion;

computer readable program configured to obtain at least one media reference set being a set of media portions;

computer readable program configured to obtain a descriptors reference set being a set of descriptors extracted from the at least one media reference set, wherein the descriptors are defined over a descriptors space;

computer readable program configured to extract descriptor space representatives from the descriptors reference set;

computer readable program configured to calculate a non-parametric probability density estimation of the descriptor space representatives, given at least one of the media reference sets;

computer readable program configured to extract a set of query descriptors from the query;

computer readable program configured to calculate for at least some of the query descriptors, at least one of their nearest descriptor space representatives;

computer readable program configured to calculate a non-parametric probability density estimation for at least some of the query descriptors, given at least one of the reference sets, using a weighted average of the non-parametric probability density estimations of the nearest descriptor space representatives, wherein the weights are determined based on the respective distances in the descriptor space, between the query descriptors and the corresponding descriptor space representatives;

computer readable program configured to calculate a non-parametric probability density estimation of the query, given at least one of the reference sets, by applying an operator to at least some of the calculated non-parametric probability density estimations of the query descriptors;

computer readable program configured to determine a saliency level of the query, based on the calculated non-parametric probability density estimation of the query given the at least one reference set, so that a lower level of the calculated non-parametric probability density estimation is indicative of a higher level of saliency of the query;

computer readable program configured to use the saliency level of he query as a criteria for selecting a sequence of media portions of interest; and

computer readable program configured to generate an audiovisual media stream based on at least some of the selected media portions of interest.