Method and system for image processing to classify an object in an image

• US 10,096,118 B2
• Filed: 01/07/2016
• Issued: 10/09/2018
• Est. Priority Date: 01/13/2009
• Status: Active Grant

• Current Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Futurewei Technologies Incorporated (Huawei Investment & Holding Co., Ltd.)
• Inventors: Huang, Yu, Li, Hongbing, Tian, Jun, Yu, Hong Heather
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US14/990,441
• Publication Number:
  US 20160117835A1
• Time in Patent Office: 1,006 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/4725 using interactive regions of the image, e.g. hot spots details of information retrieval from the Internet by using URLs G06F16/955; processing chained hypermedia data for information retrieval G06F16/94
• CPC Class Codes: H04N 21/64322 IP
• CPC Class Codes: H04N 21/8545 for generating interactive applications

Abstract

In an image processing method, an object is located within an image. An area around the object is determined and divided into at least first and second portions based upon image information within the area. The object can then be classified based upon both image information in the first portion of the area and image information in the second portion of the area.

Claims

1. An image processing method comprising:

receiving an image;

locating a foreground object within a frame of the image, the foreground object being depicted by a plurality of pixels;

determining an area around the foreground object within the frame of the image;

dividing, using a processor, the area into at least first and second portions based upon image color information within the area, wherein the first portion comprises a first group of the pixels including a portion of the foreground object and the second portion comprises a second group of the pixels including a different portion of the foreground object, the first group of the pixels having a first color characteristic and the second group of the pixels having a second color characteristic different from the first color characteristic; and

after the dividing, classifying the foreground object within the frame of the image based upon the first color characteristic of the first portion of the area and the second color characteristic of the second portion of the area, the classifying being performed by comparing the first color characteristic with first stored color characteristic information and comparing the second color characteristic with second stored color characteristic information.

2. The method of claim 1, wherein receiving the image comprises receiving a video image.

3. The method of claim 2, wherein receiving the image comprises receiving an internet protocol television (IPTV) image.

4. The method of claim 1, wherein dividing the area into at least first and second portions comprises locating a line at a color boundary within the area.

5. The method of claim 4, wherein the area comprises a rectangle and the line comprises a line such that the area is divided into only first and second portions.

6. The method of claim 1, wherein the first portion is represented by a first color histogram and the second portion is represented by a second color histogram and wherein the area is divided into the first and second portions based upon the first and second color histograms.

7. The method of claim 6, wherein dividing the area into at least first and second portions comprises locating a line at a color boundary utilizing a Bhattacharyya distance.

8. The method of claim 7, wherein locating the line comprises utilizing a weighted Bhattacharyya distance.

9. The method of claim 6, wherein the area comprises a plurality of rows and wherein dividing the area into at least first and second portions comprises calculating a plurality of row color histograms for rows in the area and comparing each row color histograms with first and/or second color histograms to determine a location of a boundary between the first portion and the second portion.

10. The method of claim 9, wherein the area comprises a rectangle and wherein calculating a plurality of row color histograms comprises calculating row color histograms for only a small number of rows, the small number of rows comprising less than ten percent of a total number of rows in the rectangle.

11. The method of claim 1, wherein the foreground object comprises a sports player having a jersey and shorts and wherein classifying the foreground object comprises classifying the foreground object based upon a color of the jersey and a color of the shorts.

12. The method of claim 1, wherein classifying the foreground object comprises determining an affiliation of the foreground object with an organization.

13. A method for classifying individuals in a video, the method being performed on a processor and comprising:

receiving a video image;

locating a player/referee within the video image, the player/referee comprising a plurality of pixels in a frame of the video image;

determining an area around the player/referee, wherein the area distinguishes the player/referee from other players/referees in the video image;

dividing the area into an upper portion that includes a first group of the pixels that depict an upper portion of the player/referee and a lower portion that includes a second group of the pixels that depict a lower portion of the player/referee;

determining color information for the upper portion and color information for the lower portion;

performing a first comparison of the color information for the upper portion with known top color information;

performing a second comparison of the color information for the lower portion with known bottom color information; and

identifying a characteristic of the player/referee based upon the first comparison and the second comparison, the characteristic related to affiliation.

14. The method of claim 13, wherein the video image is a sports video, wherein the known top color information comprises jersey color information and wherein the known bottom color information comprises shorts color information.

15. The method of claim 14, wherein identifying the characteristic of the player/referee comprises determining a team of the player/referee.

16. The method of claim 13, wherein identifying the characteristic of the player/referee comprises determining an identity of the player/referee.

17. The method of claim 13, wherein receiving the video image comprises receiving a broadcast video image and wherein identifying the characteristic of the player/referee is performed substantially in real-time with the broadcast video image.

18. An interactive television system comprising:

an authoring tool configured to

receive a broadcast video image,

locate a plurality of pixels depicting an object within the broadcast video image,

determine an area around the object that segments the object from other objects in the broadcast video image,

divide the area around the object into a first group of the pixels including a portion of the object and a second group of the pixels including a different portion of the object,

determine first color information related to the first group of the pixels and second color information related to the second group of the pixels, and

generate metadata including an affiliation of the object based upon the first color information within the first group of the pixels and the second color information within the second group of the pixels;

an aggregator configured to receive the broadcast video image and metadata and generate a video stream that is enhanced with the metadata; and

a delivery system configured to transmit the video stream that is enhanced with the metadata.

19. The interactive television system of claim 18, further comprising an interactive television client coupled to receive the video stream that is enhanced with the metadata from the delivery system.

20. The interactive television system of claim 18, wherein the broadcast video image comprises an IPTV image.

21. An apparatus comprising:

a microprocessor; and

a non-transitory computer-readable storage medium with an executable program stored thereon, wherein the executable program is configured to instruct the microprocessor to;

locate a foreground object within a frame of an image, the foreground object being depicted by a plurality of pixels;

determine an area around the foreground object within the frame of the image;

divide the area into at least first and second portions based upon image color information within the area, wherein the first portion comprises a first group of the pixels including a portion of the foreground object and the second portion comprises a second group of the pixels including a different portion of the foreground object, the first group of the pixels having a first color characteristic and the second group of the pixels having a second color characteristic different from the first color characteristic; and

classify the foreground object within the frame of the image based upon the first color characteristic of the first portion of the area and the second color characteristic of the second portion of the area, wherein classifying the foreground object comprises comparing the first color characteristic with first stored color characteristic information and comparing the second color characteristic with second stored color characteristic information.