Method of retrieving moving pictures using joint points based on pose information

• US 6,459,824 B1
• Filed: 07/09/1999
• Issued: 10/01/2002
• Est. Priority Date: 07/22/1998
• Status: Expired due to Fees

• Current Assignee: Electronics and Telecommunications Research Institute
• Original Assignee: Electronics and Telecommunications Research Institute
• Inventors: Kim, Jin Woong, Lee, Han Kyu
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Mariam, Daniel G.
• Application Number:
  US09/350,390
• Time in Patent Office: 1,180 Days
• Field of Search: 382/103, 382/118, 382/173, 382/181, 382/190, 382/201, 382/203, 382/204, 382/209, 382/217, 382/219, 382/224, 382/229, 382/239, 382/243, 382/276, 382/291, 382/305, 345/420, 345/474, 345/706, 345/949, 345/951, 345/953, 345/968, 348/97, 348/102, 348/220, 348/135, 348/152, 348/155, 348/169, 386/46, 386/69, 707/1-6, 707/100-102, 707/103.R, 707/104.1, 375/240, 375/240.1, 358/403
• US Class Current: 382/305
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: Y10S 345/949 Animation processing method
• CPC Class Codes: Y10S 345/953 Geometric processing
• CPC Class Codes: Y10S 707/99931 Database or file accessing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 715/968 interface for database querying and retrieval

Abstract

There is disclosed a method of retrieving moving pictures using joint point based moving information for allowing an user to retrieve pictures including a desired person object or the pose of the person object for still picture/moving picture data of multimedia data. The method of retrieving moving pictures using joint points based on pose information according to the present invention comprises the steps of extracting and expressing joint points for the joint point model against pictures where a major person exists from moving pictures or still pictures which are the object of retrival by means of joint points model based indexing apparatus, and retriving the person object according to the joint points extracted by said joint point model based indexing apparatus.

Claims

1. A method of retrieving moving pictures using joint points based on pose information, comprising the steps of:

extracting and expressing joint points of a joint point model existing in moving pictures or still pictures which are the object of retrieval by using an indexing apparatus and a retrieving apparatus based on a joint point model; and

retrieving a person which is an object of retrieval according to the joint points extracted from said indexing apparatus by using a retrieving apparatus based on a joint point model,

wherein said retrieving apparatus comprises;

joint point model based user query input and retrieval result interface for providing joint points information and their related information which allow a user to find a desired person which is an object of retrieval or providing the retrieval result screen to the user;

joint point model based query point representing means for representing the joint points input through said joint point model based user query and retrieval result interface;

moving picture/still picture data and indexing information storage for storing the representative pictures or still pictures which are higher in a total similarity degree in consideration of weighted values per respective similarity degree which is defined by the user;

similarity degree measuring means between query/person object for performing similarity degree measurement on various characteristics between the person object input by the query on the basis of the joint points represented from said joint point model based query joint point representing means and the indexing information stored at said moving picture/still picture data and indexing information storage; and

similarity degree analysis and picture extracting means for performing a similarity degree measurement from the value of similarity degree per respective characteristics which are measured from said similarity degree measuring means between query/person object and the indexing information which is stored at said moving picture/still picture data and indexing information storage, and sending the results to the user through said joint point model based user query input and retrieval result interface.

2. The method of retrieving moving pictures using joint points based on pose information as claimed in claim 1, wherein said indexing apparatus comprises:

moving pictures/still pictures data and indexing information storage for storing moving pictures or still pictures data which are the object of retrieval;

scene change and key-frame detecting means for receiving the moving picture data output from said moving pictures/still pictures data and indexing information storage as inputs, dividing said moving pictures into respective time axes with shots by scene change and then determining representative pictures corresponding to respective shots;

joint point extraction user interface for extracting joint points of a person which is an object of retrieval in a manual way or a semi-manual way;

joint point model based joint point extraction and description means for receiving the moving picture data output from said moving pictures/still pictures data and indexing information storage and the joint point information extracted through said joint point extraction user interface as inputs, and then representing the joint point information corresponding to the representative pictures; and

joint point model based analysis and contents based video indexing means for receiving the representative pictures transmitted from said scene change and key-frame detecting means and the joint point information transmitted from said joint point model based joint point extraction and description means, performing the contents based indexing process which analyzes the contents within the shots or the relationship between the shots, and transmitting said indexed information to said moving pictures/still pictures data and indexing information storage.

3. A method of retrieving moving pictures using joint points based on pose information as claimed in claim 1 , wherein said joint points which are represented by said joint point model based query joint point representing means are expressed into the following equation 8

4. The method of retrieving moving pictures using joint points based on pose information as claimed in claim 1, wherein the value of pose similarity degree which is measured by said similarity degree measuring means between the query/person object is expressed into the following equation 9;

$\begin{array}{cc}{\mathrm{Position}}_{\mathrm{—<br><br>}}<br><br>\mathrm{measure}=\frac{{\max }_{\mathrm{—<br><br>}}<br><br>\mathrm{distance}-\mathrm{\Delta <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{JP}}_2}{{\max }_{\mathrm{—<br><br>}}<br><br>\mathrm{distance}},<br><br>\mathrm{where}<br><br>\text{\hspace{1em}<br><br>}<br><br>\max <br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{distance}=\sqrt{{\left(x_{\mathrm{q2}}-x_{\mathrm{d2}}\right)}^2+{\left(y_{\mathrm{q2}}-y_{\mathrm{d2}}\right)}^2}& \left[\mathrm{Equation}<br><br>\text{\hspace{1em}<br><br>}<br><br>9\right]\end{array}$

where, width is the number of pixels in a horizontal direction within a frame, height is the number of pixels in a vertical direction within a frame, (xq2,yq2) is the 2nd number among the query input joint points, i.e, JP₂, and (xd2,yd2) is a position vector corresponding the 2nd number among the joint points of a major person within the retrieval object frame.

5. The method of retrieving moving pictures using joint points based on pose information as claimed in claim 1, wherein the value of pose similarity degree which is measured by said similarity degree measuring means between the query/person object is expressed into the following equation 10;

$\begin{array}{cc}{\mathrm{Pose}}_{\mathrm{—<br><br>}}<br><br>\mathrm{measure}=\frac{1}{M}<br><br>\underset{i=\left[1,3,\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>}<br><br>15\right]}{\sum <br><br>}<br><br>\frac{\left({\max }_{\mathrm{—<br><br>}}<br><br>\mathrm{angle}-{\mathrm{\Delta <br><br>\theta <br><br>}}_i\right)}{{\max }_{\mathrm{—<br><br>}}<br><br>\mathrm{angle}},<br><br>\mathrm{where}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\max }_{\mathrm{—<br><br>}}<br><br>\mathrm{angle}=180<br><br>\text{<br><br>}<br><br>\begin{array}{c}M<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{is}<br><br>\text{\hspace{1em}<br><br>}<br><br>a<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{number}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{of}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{joint}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{points},{\mathrm{Jp}}_{\mathrm{qi}}<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{or}<br><br>\text{\hspace{1em}<br><br>}<br><br>{\mathrm{Jp}}_{\mathrm{di}}=\left(-1,-1\right)\end{array}& \left[\mathrm{Equation}<br><br>\text{\hspace{1em}<br><br>}<br><br>10\right]\end{array}$

where, M is the total number of the joint points where the query joint point or the joint point being the object of retrieval is commonly existing among the 14 joint points except for JP₂.

6. The method of retrieving moving pictures using joint points based on pose information as claimed in claim 1, wherein the similarity degree per respective characteristics which is measured by said similarity degree measuring means between the query/person object is a scale of similarity degree, and a value of the scale similarity degree expressed into the following equation 11 $\begin{array}{c}{\mathrm{JP}}_{1.2}=\sqrt{{\left(x_2-x_1\right)}^2+{\left(y_2-y_1\right)}^2}\end{array}$



Scale—



measure=[JPq1–



2/JPd1–



2,if

 



JPd1–



2≥

JPq1–



2JPd1–



2/JPq1–



2,if

 



JPd1–



2<

JPq1–



2][Equation

 



11]

where JP_q1-2 is the query joint points, and JP_d1-2 is the object joint points.