Semantic parsing of objects in video

• US 8,774,522 B2
• Filed: 07/23/2013
• Issued: 07/08/2014
• Est. Priority Date: 07/28/2010
• Status: Active Grant

• Current Assignee: Kyndryl Incorporated
• Original Assignee: International Business Machines Corporation
• Inventors: Brown, Lisa Marie, Feris, Rogerio Schmidt, Hampapur, Arun, Vaquero, Daniel Andre
• Primary Examiner(s): Bitar, Nancy
• Application Number:
  US13/948,325
• Publication Number:
  US 20130308868A1
• Time in Patent Office: 350 Days
• Field of Search: None
• US Class Current: 382/195
• CPC Class Codes: G06F 18/213 Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 30/2504 Coarse or fine approaches, e.g. resolution of ambiguities or multiscale approaches
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Methods, systems, and computer program products for parsing objects in a video are provided herein. A method includes producing a plurality of versions of an image of an object, wherein each version has a different resolution of said image of said object, and computing an appearance score at each of a plurality of regions on the lowest resolution version for at least one semantic attribute for said object. Such a method also includes analyzing one or more other versions to compute a resolution context score for each of the plurality of regions in the lowest resolution version, wherein said resolution context score denotes an extent to which finer spatial structure exists in the one or more others versions than in the lowest resolution version, and determining a configuration of the at least one semantic attribute in the lowest resolution version based on the appearance score and the resolution context score.

Claims

1. A method comprising:

Producing a plurality of versions of an image of an object derived from a video input, wherein each version has a different resolution of said image of said object;

computing an appearance score at each of a plurality of regions on the lowest resolution version of said plurality of versions of said image for at least one semantic attribute for said object, wherein said appearance score denotes a probability of the at least one semantic attribute appearing in the region;

analyzing one or more other versions of the multiple versions to compute a resolution context score for each of the plurality of regions in the lowest resolution version, wherein said resolution context score denotes an extent to which finer spatial structure exists in the one or more others versions than in the lowest resolution version for each of the plurality of regions;

determining a configuration of the at least one semantic attribute in the lowest resolution version based on the appearance score and the resolution context score in each of the plurality of regions in the lowest resolution version; and

displaying said configuration.

2. The method of claim 1, comprising:

storing said configuration.

3. The method of claim 1, comprising:

computing a geometric score for each region of said plurality of regions on the lowest resolution version, said geometric score computing a probability of a region matching stored reference data for a reference object corresponding to the detected object with respect to angles and distances among the plurality of regions.

4. The method of claim 3, wherein the resolution context score for the lower resolution version of said image is computed as a weighted average score computed from a plurality of scores for a next higher resolution version of said higher resolution versions of said image.

5. The method of claim 4, wherein said plurality of scores for said next higher resolution version of said image comprise appearance scores and geometric scores.

6. The method of claim 4, wherein said plurality of scores for said next higher resolution version of said image comprise appearance scores, geometric scores and resolution context scores.

7. The method of claim 6, wherein said weighted average score for the next higher resolution version of the image is computed using the following formula divided by I:

8. The method claim of 7, comprising:

storing and/or displaying output of at least one portion of said image in at least one version of said higher level versions of said image with spatial information on semantic attributes.

9. A computer program product comprising a computer readable storage hardware device having computer readable program code embodied in the computer readable storage hardware device, said computer readable program code containing instructions that perform a method for estimating parts and attributes of an object in video, said method comprising:

producing a plurality of versions of an image of an object derived from a video input, wherein each version has a different resolution of said image of said object;

computing an appearance score at each of a plurality of regions on the lowest resolution version of said plurality of versions of said image for at least one semantic attribute for said object, wherein said appearance score denotes a probability of the at least one semantic attribute appearing in the region;

analyzing one or more other versions of the multiple versions to compute a resolution context score for each of the plurality of regions in the lowest resolution version, wherein said resolution context score denotes an extent to which finer spatial structure exists in the one or more others versions than in the lowest resolution version for each of the plurality of regions; and

determining a configuration of the at least one semantic attribute in the lowest resolution version based on the appearance score and the resolution context score in each of the plurality of regions in the lowest resolution version; and

displaying said configuration.

10. The computer program product of claim 9, said method comprising:

storing said configuration.

11. The computer program product of claim 9, said method comprising:

computing a geometric score for each region of said plurality of regions on the lowest resolution version, said geometric score computing a probability of a region matching stored reference data for a reference object corresponding to the detected object with respect to angles and distances among the plurality of regions.

12. The computer program product of claim 11, wherein the resolution context score for the lower resolution version of said image is computed as a weighted average score computed from a plurality of scores for a next higher resolution version of said higher resolution versions of said image.

13. The computer program product of claim 12, wherein said plurality of scores for said next higher resolution version of said image comprise appearance scores and geometric scores.

14. The computer program product of claim 12, wherein said plurality of scores for said next higher resolution version of said image comprise appearance scores, geometric scores and resolution context scores.

15. The computer program product of claim 14, wherein said weighted average score for the next higher resolution version of the image is computed using the following formula divided by I:

16. The computer program product of claim 15, said method comprising:

storing and/or displaying output of at least one portion of said image in at least one version of said higher level versions of said image with spatial information on semantic attributes and associated parts.

17. A computer system comprising a processor and a computer readable memory unit coupled to the processor, said computer readable memory unit containing instructions that when run by the processor implement a method for estimating parts and attributes of an object in video, said method comprising:

producing a plurality of versions of an image of an object derived from a video input, wherein each version has a different resolution of said image of said object;

computing an appearance score at each of a plurality of regions on the lowest resolution version of said plurality of versions of said image for at least one semantic attribute for said object, wherein said appearance score denotes a probability of the at least one semantic attribute appearing in the region;

analyzing one or more other versions of the multiple versions to compute a resolution context score for each of the plurality of regions in the lowest resolution version, wherein said resolution context score denotes an extent to which finer spatial structure exists in the one or more others versions than in the lowest resolution version for each of the plurality of regions; and

determining a configuration of the at least one semantic attribute in the lowest resolution version based on the appearance score and the resolution context score in each of the plurality of regions in the lowest resolution version; and

displaying said configuration.

18. The system of claim 17, said method comprising:

storing said configuration.