Image recognition system and image recognition program

• US 7,515,735 B2
• Filed: 04/11/2003
• Issued: 04/07/2009
• Est. Priority Date: 04/11/2003
• Status: Expired due to Fees

• Current Assignee: National Institute of Information and Communications Technology
• Original Assignee: National Institute of Information and Communications Technology
• Inventors: Miyamori, Hisashi
• Primary Examiner(s): Bella; Matthew C.
• Assistant Examiner(s): LIEW, ALEX KOK SOON
• Application Number:
  US10/552,143
• Publication Number:
  US 20070104368A1
• Time in Patent Office: 2,188 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

An image recognizing device for recognizing the motion of a player on the basis of a content containing a sport in which players play in zones demarcated with an obstacle such as a net, including a video information acquiring section for acquiring video information representing the motion of the player in play from the content, a hidden state judging unit for judging whether or not a use tool such as a ball moving between the zones and being an object of scoring is in a hidden state by a predetermined object, a hit time information determining section for determining the time at which the use tool hits on the basis of the hiding start time at which the use tool is judged to be hidden by the hidden state judging section and the hiding stop time at which the hiding is judged to be stopped, a rule information storage section in which rule information on the sport is stored, and an image content recognizing section for recognizing the image content including the motion of the player represented by the video information on the basis of the video information acquired by the video information acquiring section, the position of the use tool at the hit time determined by the hit time information determining section, and the rule information stored in the rule information storage section.

Claims

1. An image recognition system that recognizes motions of players of a sport playing between areas partitioned by an obstacle such as a net from a content of its on-air program, of its material video before broadcasted, or recorded by a recording media such as a VTR, and characterized by comprising

a visual information obtaining device that obtains visual information on which a motion of the player in at least one area during a match is shown from the content,

an occlusion state determining device that determines whether or not a used material such as a ball that moves between the areas included in the visual information obtained by the visual information obtaining device and that is an object to count the score of the relevant sport is in a state being hidden by a predetermined object body,

an impact time information specifying device that specifies an impact time when the used material is hit based on an occlusion start time when the occlusion state determining device determines that the used material changes its state from not being hidden by the object body to being hidden by the object and an occlusion release time when the occlusion state determining device determines that the used material changes its state from being hidden by the object body to not being hidden by the object,

a rule information storing device that stores rule information to conduct the relevant sport, and

an image content recognizing device that recognizes an image content including the motion of the player shown by the visual information based on the visual information obtained by the visual information obtaining device, a position of the used material at the impact time specified by the impact time information specifying device and the rule information stored in the rule information storing device.

2. The image recognition system described in claim 1, and characterized by that the occlusion state determining device comprises a distance determining device that determines whether or not the used material locates within a predetermined distance from the object body and an occlusion start and release time specifying device that specifies a moment when the distance determining device determines that the used material locates within the predetermined distance from the object body and the used material changes its state from not being hidden by the object body to being hidden by the object body as the occlusion start time and that specifies a moment when the distance determining device determines that the used material locates within the predetermined distance from the object body and the used material changes its state from being hidden by the object body to not being hidden by the object body as the occlusion release time.

3. The image recognition system described in claim 1, and characterized by that the impact time shown by t_a, the occlusion start time shown by t0 and the occlusion release time shown by t1 have a relationship shown by the following expression (Expression 1)

t_a=a×

t0+(1−

a)×

t1 

 

(Expression

     1)

where coefficient a is 0≦

a≦

1.

4. The image recognition system described in claim 1, and characterized by that the visual information obtaining device comprises a domain element extracting device that extracts used facilities information such as the obstacle like the net or a boundary line showing a boundary between the areas and outside the areas, player position information showing a position of the player and used material information showing the used material that moves between the areas and that becomes the object to count the score of the relevant sport from the visual information.

5. The image recognition system described in claim 4, and characterized by that the player position information is position information showing a region including the player and a tool that the player always holds and uses during the match.

6. The image recognition system described in claim 4, and characterized by that the domain element extracting device extracts the player position information from the visual information based on the used facilities information extracted by the domain element extracting device.

7. The image recognition system described in claim 4, and characterized by that the domain element extracting device extracts the used material information from the visual information based on the used facilities information and the player position information extracted by the domain element extracting device.

8. The image recognition system described in claim 4, and characterized by that the used facilities information, the player position information, the used material information and the rule information are based on knowledge regarding a sport item to be a target to extract the image.

9. The image recognition system described in claim 1, and characterized by comprising an audio information obtaining device that obtains audio information synchronous with the visual information, such as an impact sound generating at a moment when the used material such as the ball that moves between the areas and that is an object to count the score of the relevant sport is hit from the content, wherein the impact time information specifying device specifies the impact time based on a combination of the occlusion start time and the occlusion release time and the audio information obtained by the audio information obtaining device.

10. The image recognition system described in claim 9, and characterized by that the impact time information specifying part specifies a time when the audio information shows a value bigger than a predetermined level as the impact time.

11. The image recognition system described in claim 9, and characterized by that the audio information obtaining device comprises a filtering device that passes a predetermined frequency band and the audio information is the information that has passed through the filtering device.

12. The image recognition system described in claim 11, and characterized by the filtering device consists of a band-pass filter.

13. The image recognition system described in claim 9, and characterized by that the impact time information specifying device specifies the impact time based on an impact sound candidate data having a predetermined time including the impact sound extracted from the audio information.

14. The image recognition system described in claim 9, and characterized by that multiple pieces of impact sound candidate data are extracted from the audio information so that an impact sound candidate data at one time and an impact sound candidate data at its subsequent time have a time that overlaps each other and the impact time information specifying device specifies the impact time based on the multiple pieces of the impact sound candidate data.

15. The image recognition system described in claim 14, and characterized by that each of the multiple pieces of the impact sound candidate data is arranged to have an identical data length and the multiple pieces of the impact sound candidate data are arranged to be extracted from the audio information at intervals of a certain period.

16. The image recognition system described in claim 9, and characterized by comprising an impact sound pattern information storing device that stores impact sound pattern information that is a patternized sound change due to a state under which the used material is hit by the tool such as a racket that the player always holds and uses during the match, wherein the impact time information specifying device specifies the impact time based on the impact sound pattern information stored in the impact sound pattern information storing device and the audio information.

17. An image recognition system that recognizes motions of players of a sport playing between areas partitioned by an obstacle such as a net from a content of its on-air program, of its material video before broadcasted, or recorded by a recording media such as a VTR, and characterized by comprising

a visual information obtaining device that obtains visual information on which a motion of the player in at least one area during a match is shown from the content,

an occlusion state determining device that determines whether or not a used material such as a ball that moves between the areas included in the visual information obtained by the visual information obtaining device and that is an object to count the score of the relevant sport is in a state being hidden by a predetermined object body,

an impact time information specifying device that specifies an impact time when the used material is hit based on an occlusion start time when the occlusion state determining device determines that the used material changes its state from not being hidden by the object body to being hidden by the object and an occlusion release time when the occlusion state determining device determines that the used material changes its state from being hidden by the object body to not being hidden by the object, and

an image content recognizing device that recognizes an image content including the motion of the player shown by the visual information based on the visual information obtained by the visual information obtaining device and a position of the used material at the impact time specified by the impact time information specifying device.

18. The image recognition system described in claim 17, and characterized by comprising an audio information obtaining device that obtains audio information synchronous with the visual information, such as an impact sound generating at a time when the used material such as the ball that moves between the areas and that is an object to count the score of the relevant sport is hit from the content, and the impact time information specifying device specifies the impact time based on a combination of the occlusion start time and the occlusion release time and the audio information obtained by the audio information obtaining device.

19. An image recognition program stored in a computer readable medium that is a program activating an image recognition system that recognizes motions of players of a sport playing between areas partitioned by an obstacle such as a net from a content of its on-air program, of its material video before broadcasted, or recorded by a recording media such as a VTR by operating a computer, and characterized by making the computer function as

a visual information obtaining means that obtains visual information on which a motion of the player in at least one area during a match is shown from the content,

an occlusion state determining means that determines whether or not a used material such as a ball that moves between the areas included in the visual information obtained by the visual information obtaining means and that is an object to count the score of the relevant sport is in a state being hidden by a predetermined object body,

an impact time information specifying means that specifies an impact time when the used material is hit based on an occlusion start time when the occlusion state determining means determines that the used material changes its state from not being hidden by the object body to being hidden by the object and an occlusion release time when the occlusion state determining means determines that the used material changes its state from being hidden by the object body to not being hidden by the object,

a rule information storing means that stores rule information to conduct the relevant sport, and

an image content recognizing means that recognizes an image content including the motion of the player shown by the visual information based on the visual information obtained by the visual information obtaining means, a position of the used material at the impact time specified by the impact time information specifying means and the rule information stored in the rule information storing means.

20. An image recognition program stored in a computer readable medium that is a program activating an image recognition system that recognizes motions of players of a sport playing between areas partitioned by an obstacle such as a net from content of its on-air program, of its material video before broadcasted, or recorded by a recording media such as a VTR by operating a computer, and characterized by making the computer to function as

a visual information obtaining means that obtains visual information on which a motion of the player in at least one area during a match is shown from the content,

an occlusion state determining means that determines whether or not a used material such as a ball that moves between the areas included in the visual information obtained by the visual information obtaining means and that is an object to count the score of the relevant sport is in a state being hidden by a predetermined object body,

an impact time information specifying means that specifies an impact time when the used material is hit based on an occlusion start time when the occlusion state determining means determines that the used material changes its state from not being hidden by the object body to being hidden by the object and an occlusion release time when the occlusion state determining means determines that the used material changes its state from being hidden by the object body to not being hidden by the object, and

an image content recognizing means that recognizes an image content including the motion of the player shown by the visual information based on the visual information obtained by the visual information obtaining means and a position of the used material at the impact time specified by the impact time information specifying means.