Remote multiplayer interactive physical gaming with mobile computing devices

• US 10,643,492 B2
• Filed: 06/19/2019
• Issued: 05/05/2020
• Est. Priority Date: 06/20/2018
• Status: Active Grant

• Current Assignee: NEX Team Inc.
• Original Assignee: NEX Team Inc.
• Inventors: Lee, Keng Fai, Zhang, Qi, Chan, Wing Hung, Ng, Wang Fai, Lam, Tin Yin, Wu, Alexander, Smith, Ian, Schmierer, David Richard, Farkas, Bradford Lewis, Law, Kin Ming
• Primary Examiner(s): Shah, Milap
• Application Number:
  US16/445,893
• Publication Number:
  US 20190392729A1
• Time in Patent Office: 321 Days
• CPC Class Codes: A63B 2071/0694 Visual indication, e.g. Indicia
• CPC Class Codes: A63B 2220/806 Video cameras
• CPC Class Codes: A63B 24/0062 Monitoring athletic performances, e.g. for determining the work of a user on an exercise apparatus, the completed jogging or cycling distance
• CPC Class Codes: A63B 69/0071 for basketball
• CPC Class Codes: A63B 71/0622 Visual, audio or audio-visual systems for entertaining, instructing or motivating the user
• CPC Class Codes: A63B 71/0669 Score-keepers or score display devices
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/332 using wireless networks, e.g. cellular phone networks
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: G06F 16/353 into predefined classes
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/42 of sport video content
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G09B 19/0038 Sports
• CPC Class Codes: G09B 5/04 with audible presentation of the material to be studied sound recording or reproducing G11B

Abstract

Methods and systems for remote real-time or asynchronous multiplayer gameplay using mobile devices are disclosed. The methods and systems include steps of mounting mobile computing devices in gaming areas associated with users; initiating a game between the users; analyzing movements of the users, ball(s), and/or other game equipment of the game by performing a computer vision algorithm on one or more frames of an input video of the game; and providing feedback to the users, where feedback may include instructions to be performed by the users and player analytics. Embodiments of the present invention use computer vision techniques to enable a resource-limited mobile device such as a smartphone to conduct the aforementioned steps.

Claims

1. A method for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, comprising:

joining a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;

capturing a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on a first mobile computing device in a first gaming area associated with the first player;

extracting a first posture flow of the first player from the first game video, by performing a computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;

generating a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;

receiving a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of the second player in a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and

generating a feedback to the first player based on at least one of the first player analytic and the second player analytic.

2. The method of claim 1, wherein the second gaming area is geographically different from the first gaming area.

3. The method of claim 1, wherein the first play and the second play of the multiplayer interactive physical sports game are asynchronous in time.

4. The method of claim 1, further comprising:

receiving a user input from the first player to initiate the multiplayer interactive physical sports game, wherein the user input is selected from the group consisting of a voice input of a predetermined phrase and a gesture input of a predetermined gesture sequence to the first mobile computing device.

5. The method of claim 1, further comprising:

detecting a shot attempt from the first game video of the multiplayer interactive physical sports game, wherein the multiplayer interactive physical sports game is a ball game, and wherein the generating of the first player analytic is based on an outcome of the shot attempt.

6. The method of claim 1, further comprising:

providing the feedback to the first player using an audio output.

7. The method of claim 1, wherein the generating of the feedback is live during the first play of the multiplayer interactive physical sports game.

8. The method of claim 1, wherein the feedback to the first player are instructions for actions to be performed by the first player.

9. The method of claim 1, further comprising:

determining a foot location of the first player in the first gaming area from the first game video, wherein the generating of the feedback is further based on the foot location of the first player.

10. A system for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, comprising:

at least one processor on a first mobile computing device; and

a non-transitory physical medium for storing program code accessible by the at least one processor, the program code when executed by the processor causes the processor to;

join a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;

capture a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on the first mobile computing device in a first gaming area associated with the first player;

extract a first posture flow of the first player from the first game video, by performing a first computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;

generate a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;

receive a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and

generate a feedback to the first player based on at least one of the first player analytic and the second player analytic.

11. The system of claim 10, wherein the second gaming area is geographically different from the first gaming area.

12. The system of claim 10, wherein the first play and the second play of the multiplayer interactive physical sports game are asynchronous in time.

13. The system of claim 10, wherein the program code when executed by the processor further causes the processor to:

receive a user input from the first player to initiate the multiplayer interactive physical sports game, wherein the user input is selected from the group consisting of a voice input of a predetermined phrase and a gesture input of a predetermined gesture sequence to the first mobile computing device.

14. The system of claim 10, wherein the program code when executed by the processor further causes the processor to:

detect a shot attempt from the first game video of the multiplayer interactive physical sports game, wherein the multiplayer interactive physical sports game is a ball game, and wherein the generating of the first player analytic is based on an outcome of the shot attempt.

15. The system of claim 10, wherein the program code when executed by the processor further causes the processor to:

provide the feedback to the first player using an audio output.

16. The system of claim 10, wherein the generating of the feedback is live during the first play of the multiplayer interactive physical sports game.

17. The system of claim 10, wherein the feedback to the first player are instructions for actions to be performed by the first player.

18. A non-transitory physical storage medium for remote multiplayer interactive physical sports gaming with at least two mobile computing devices selected from the group consisting of a mobile phone and a tablet, the storage medium comprising program code stored thereon, and the program code when executed by a processor causes the processor to:

join a multiplayer interactive physical sports game between at least a first player and a second player, wherein the first player and the second player compete in completing a multiplayer interactive physical sports gaming challenge;

capture a first game video of the first player in a first play of the multiplayer interactive physical sports game, using a first camera on a first mobile computing device in a first gaming area associated with the first player;

extract a first posture flow of the first player from the first game video, by performing a first computer vision algorithm on one or more frames of the first game video, wherein the computer vision algorithm comprises a Convolutional Neural Network (CNN) module, wherein the CNN module detects one or more body key points of the first player in an image plane of the first game video, and wherein the CNN module has been trained using one or more prior input videos;

generate a first player analytic based on the first posture flow, wherein the first player analytic is not identical to the first posture flow;

receive a second player analytic, wherein the second player analytic was generated based on a second posture flow of the second player, and wherein the second posture flow was extracted from a second game video of a second play of the multiplayer interactive physical sports game, captured by a second camera on a second mobile computing device in a second gaming area associated with the second player, and wherein the second player analytic is not identical to the second posture flow; and

generate a feedback to the first player based on at least one of the first player analytic and the second player analytic.