METHODS AND SYSTEMS FOR PROCESSING OF VIDEO DATA

• US 20100239123A1
• Filed: 10/13/2008
• Published: 09/23/2010
• Est. Priority Date: 10/12/2007
• Status: Active Grant

• Current Assignee: Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation), Cambridge Enterprise Limited (University of Cambridge)
• Original Assignee: Toyota Motor Europe SA (Toyota Motor Corporation), Cambridge Enterprise Limited (University of Cambridge)
• Inventors: Yanagihara, Hiromichi, Brostow, Gabriel, Fauqueur, Julien, Funayama, Ryuji, Cipolla, Roberto
• Granted Patent: US 9,002,055 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/251 involving models

Abstract

A method (100) and system (300) is described for processing video data comprising a plurality of images. The method (100) comprising obtaining (104, 106), for each of the plurality of images, a segmentation in a plurality of regions and a set of keypoints, and tracking (108) at least one region between a first image and a subsequent image resulting in a matched region in the subsequent image taking into account a matching between keypoints in the first image and the subsequent image. The latter results in accurate tracking of regions. Furthermore the method may optionally also perform label propagation taking into account keypoint tracking.

Claims

1-21. -21. (canceled)

22. A method for processing video data comprising a plurality of images, the method comprising obtaining, for each of said plurality of images, a segmentation in a plurality of regions and a set of keypoints,

tracking at least one region between a first image and a subsequent image based on an appearance of the at least one region and a region in said subsequent image and finding a matched region in the subsequent image taking into account a matching between keypoints in the first image and the subsequent image, wherein the tracking of two regions, one from the first image and the other from the subsequent image, are matched if they have a number of matching keypoints in common which is non-zero and if their overall similarity or what they have in common is larger than a predetermined value.

23. The method for processing video data according to claim 22, wherein tracking at least one region between a first image and a subsequent image comprises associating at least one first keypoint to the at least one region in said first image matching a subsequent keypoint in said subsequent image with said first keypoint, and determining a matching of the at least one region with a subsequent region in said subsequent image based on information regarding said matched keypoints.

24. The method for processing video data according to claim 23, wherein matching a subsequent keypoint with the first keypoint comprises bidirectional matching.

25. The method for processing video data according to claim 22, wherein the tracking furthermore takes into account a measure expressing how much the first region and the subsequent region are similar or have in common.

27. The method for processing video data according to claim 22, wherein the tracking furthermore takes into account groups of connected regions in the first and/or the subsequent image.

28. The method for processing video data according to claim 22, the method further comprising

obtaining a label for at least one object in the first image, the object comprising at least one of a region or a keypoint, and

propagating the label to the object in the subsequent image using the correspondence between the at least one region or keypoint in the first image and the at least one region or keypoint in the subsequent image.

29. The method for processing video data according to claim 28, wherein propagating the label comprises

obtaining a keypoint-label for a first keypoint in the object in the first image based on the label of said object,

associating a corresponding keypoint-label to the matching subsequent keypoint based on the keypoint-label for the first keypoint,

deriving a label for the object in the subsequent image based on the keypoint-label of the subsequent keypoint in the subsequent image.

30. The method for processing video data according to claim 28, wherein propagating the label comprises

obtaining a region-label for a first region in the object in the first image based on the label of said object,

associating a corresponding region-label to the matching subsequent region based on the region-label for the first region,

deriving a label for the object in the subsequent image based on the region-label of the subsequent region in the subsequent image.

31. The method for processing video data according to claim 22, wherein a keypoint in the image comprises a pixel location and an area surrounding the pixel location having an average radius.

32. The method for processing video data according to claim 22, wherein said segmentation is an oversegmentation of the image such that at least one object corresponds to a plurality of segments.

38. A method for controlling an application system, the method comprising processing video data according to the method of claim 22 and deriving from the processed video data control instructions for controlling the application system.

39. The method for controlling according to claim 38, wherein the application system is an automotive vehicle and wherein deriving control instructions comprises deriving driving instructions.

40. A computer program product for performing, when executed on a processing means, a method according to claim 22.

41. A non-volatile machine-readable data storage device storing the computer program product of claim 40.

26. (canceled)

33. A processing system for processing video data comprising a plurality of images, the system comprising

means for obtaining a segmented image, for each of the plurality of images in the video data, the segmented image being segmented in a plurality of regions,

means for obtaining a set of keypoints for each of the plurality of images in the video data,

region tracking means for tracking at least one region between a first image and a subsequent image resulting in a matched region in the subsequent image, the region tracking means taking into account an appearance of the at least one region and the region in said subsequent image and taking into account a matching between keypoints in the first image and the subsequent image.

34. A processing system according to claim 33, wherein the system furthermore comprises

a label obtaining means for receiving at least one label for a first image, and

a label propagating means for propagating the at least one label in the subsequent image using the correspondence between the at least one region or keypoint in the first image and the at least one region or keypoint in the subsequent image.

35. An application system for performing an action, the application system comprising a processing system according to claim 33 and a controller for controlling the action taking into account tracked region information for the video data obtainable using the processing system.

36. The application system according to claim 34 wherein the processing system is a processing system according to claim 33, and wherein the controller is adapted for controlling the action taking into account propagated label information for the video data obtainable using the processing system.

37. The application system according to claim 35, wherein the application system is an automotive vehicle.

42. (canceled)