Capturing smart playable device and gestures

• US 10,097,959 B2
• Filed: 10/18/2016
• Issued: 10/09/2018
• Est. Priority Date: 07/13/2016
• Status: Expired due to Fees

• Current Assignee: Play Impossible Corporation
• Original Assignee: Play Impossible Corporation
• Inventors: Monnin, Brian, Amit, Gadi, Langdon, Kevin
• Primary Examiner(s): He, Yingchun
• Application Number:
  US15/297,015
• Publication Number:
  US 20180020328A1
• Time in Patent Office: 721 Days
• Field of Search: None
• CPC Class Codes: A63B 23/00 Exercising apparatus specially adapted for particular parts of the body A63B22/00 takes precedence; force-resisting aspects A63B21/00; electric or electronic controls therefor A63B24/00; devices for exercising or strengthening of fingers, or arms in teaching operation of keyboards G09B15/06
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/533 for prompting the player, e.g. by displaying a game menu
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/92 Video game devices specially adapted to be hand-held while playing
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H01Q 1/22 by structural association with other equipment or articles
• CPC Class Codes: H01Q 1/24 : with receiving set
• CPC Class Codes: H01Q 1/243 : with built-in antennas
• CPC Class Codes: H01Q 1/27 : Adaptation for use in or on movable bodies H01Q1/08, H01Q1/12, H01Q1/18 take precedence
• CPC Class Codes: H01Q 1/38 : formed by a conductive layer on an insulating support patch antennas H01Q9/0407; microstrip dipole antennas H01Q9/065; microstrip slot antennas H01Q13/106; transmission line microstrip antennas H01Q13/206; manufacturing reflecting surfaces using insulating material for supporting the reflecting surface H01Q15/142
• CPC Class Codes: H04L 67/306 : User profiles
• CPC Class Codes: H04M 1/72403 : with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/72412 : using two-way short-range wireless interfaces
• CPC Class Codes: H04M 2250/52 : including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04W 4/027 : using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/70 : Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 76/14 : Direct-mode setup

Abstract

This disclosure is generally directed to capturing aspects of a smart playable device. The playable device can include any device that is suitable for sports, games, or play, such as balls, discs, staffs, clubs, and the like. The playable device can include rapid charging circuits to receive power from a remote charger. The playable device may transmit sensor data including motion data to a computing device. Sensor data may be interpreted by the computing device as gestures of the playable device for interacting with the computing device. Sensor data can also be interpreted as motion data during gameplay and notifications can be provided to a user based on the gameplay. In some instances, the computing device can capture image data of the playable device and identify the playable device and annotate image data based on the sensor data from the playable device.

Claims

1. A computer-implemented method comprising:

establishing a wireless communication between a smart playable device and a computing device;

receiving sensor data from the smart playable device;

determining that the sensor data corresponds to a motion of the smart playable device;

determining a first velocity of the smart playable device based on the sensor data;

receiving image data from an imaging device, the image data including at least a representation of at least a portion of the smart playable device;

performing first image analysis on the image data to identify an object that may represent the smart playable device based at least in part on a size or a shape of the representation of the at least the portion of the smart playable device;

performing second image analysis on the image data to determine a second velocity of the object;

determining that the first velocity is within a threshold amount of the second velocity;

determining that the object represents the smart playable device;

presenting, as displayed image data, at least a portion of the image data on a display of the computing device; and

annotating the displayed image data based at least in part on the sensor data of the smart playable device.

2. The computer-implemented method as recited in claim 1, further comprising:

mapping the sensor data to a range of colors associated with a velocity of the smart playable device;

determining a color of the range of colors associated with the first velocity; and

highlighting the object in the displayed image data with the color.

3. The computer-implemented method as recited in claim 1, further comprising:

determining that the motion of the smart playable device is associated with historical motion data, the historical motion data including at least a historical maximum height of the smart playable device resulting from a throw;

determining a height of the smart playable device based at least in part on the sensor data;

determining that the height is above the historical maximum height; and

annotating the displayed image data at a time in which the height of the smart playable device is greater than the historical maximum height.

4. The computer-implemented method as recited in claim 1, further comprising:

storing video data associated with gameplay of the smart playable device;

determining one or more game events based on the sensor data received from the smart playable device;

identifying, as identified video data, at least one portion of the video data based at least in part on the one or more game events; and

presenting the identified video data on the display of the computing device.

5. The computer-implemented method as recited in claim 1, further comprising:

extrapolating, as a predicted motion path, movement of the smart playable device based at least in part on the sensor data of the smart playable device;

determining that at least a portion of the predicted motion path extends beyond a frame of the imaging device; and

annotating the displayed image data with an indication to move the imaging device in a direction of the at least the portion of the predicted motion path extending beyond the frame of the imaging device.

6. The computer-implemented method as recited in claim 5, wherein the frame is a first frame, the computer-implemented method further comprising:

determining that the at least the portion of the predicted motion path is within a second frame of the imaging device; and

removing the indication from the displayed image data.

7. The computer-implemented method as recited in claim 1, further comprising:

determining a path of the smart playable device in the image data over time;

determining, based at least in part on the first velocity and the second velocity, velocity data associated with the smart playable device;

mapping the velocity data to a range of colors; and

highlighting one or more points along the path of the smart playable device in the image data with a color of the range of colors corresponding to a velocity of the smart playable device at the one or more points.

8. The computer-implemented method as recited in claim 1, wherein the object is a first object, the computer-implemented method further comprising:

performing third image analysis on the image data to identify a second object that may represent the smart playable device;

performing fourth image analysis on the image data to determine a third velocity of the object;

determining a difference between the first velocity and the third velocity;

determining that the difference meets or exceeds a difference threshold; and

determining that the second object does not represent the smart playable device.

9. The computer-implemented method as recited in claim 1, further comprising annotating at least a portion of the displayed image data with an animation based on gameplay associated with a game application selected by a user and operating in conjunction with the smart playable device.

10. A system comprising:

one or more processors; and

memory to store computer-executable instructions that, when executed, cause the one or more processors to perform acts comprising;

establishing a wireless communication between a smart playable device and a computing device;

receiving sensor data from the smart playable device;

determining that the sensor data corresponds to a motion of the smart playable device;

determining a first velocity of the smart playable device based on the sensor data;

receiving image data including at least a representation of at least a portion of the smart playable device;

performing first image analysis on the image data to identify an object that may represent the smart playable device based at least in part on a size or a shape of the representation of the at least the portion of the smart playable device;

performing second image analysis on the image data to determine a second velocity of the object;

determining that the first velocity is within a threshold amount of the second velocity;

determining that the object represents the smart playable device;

presenting, as displayed image data, at least a portion of the image data on a display of the computing device; and

annotating the displayed image data based at least in part on the sensor data of the smart playable device.

11. The system as recited in claim 10, the acts further comprising:

mapping the sensor data to a range of colors associated with a velocity of the smart playable device;

determining a color of the range of colors associated with the velocity; and

highlighting the object in the displayed image data with the color.

12. The system as recited in claim 10, the acts further comprising:

determining that the motion of the smart playable device is associated with historical motion data, the historical motion data including at least a historical maximum height of the smart playable device resulting from a throw;

determining a height of the smart playable device based at least in part on the sensor data;

determining that the height is above the historical maximum height; and

annotating the displayed image data at a time in which the height of the smart playable device is greater than the historical maximum height.

13. The system as recited in claim 10, the acts further comprising:

storing video data associated with gameplay of the smart playable device;

determining one or more game events based on the sensor data received from the smart playable device;

identifying, as identified video data, at least one portion of the video data based at least in part on the one or more game events; and

presenting the identified video data on the display of the computing device.

14. The system as recited in claim 10, the acts further comprising:

annotating at least a portion of the displayed image data with an animation based on gameplay associated with a game application selected by a user and operating in conjunction with the smart playable device.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause one or more processors to perform operations comprising:

establishing a wireless communication between a smart playable device and a computing device;

receiving sensor data from the smart playable device;

determining that the sensor data corresponds to a motion of the smart playable device;

determining a first velocity of the smart playable device based on the sensor data;

receiving image data including at least a representation of at least a portion of the smart playable device;

performing first image analysis on the image data to identify an object that may represent the smart playable device based at least in part on a size or a shape of the representation of the at least the portion of the smart playable device;

performing second image analysis on the image data to determine a second velocity of the object;

determining that the first velocity is within a threshold amount of the second velocity;

determining that the object represents the smart playable device;

causing, as displayed image data, at least a portion of the image data to be presented on a display of the computing device; and

annotating the displayed image data based at least in part on the sensor data of the smart playable device.

16. The non-transitory computer-readable medium of claim 15, the operations further comprising:

mapping the sensor data to a range of colors associated with a velocity of the smart playable device;

determining a color of the range of colors associated with the velocity; and

highlighting the object in the displayed image data with the color.

17. The non-transitory computer-readable medium of claim 15, the operations further comprising:

determining that the motion of the smart playable device is associated with historical motion data, the historical motion data including at least a historical maximum height of the smart playable device resulting from a throw;

determining a height of the smart playable device based at least in part on the sensor data;

determining that the height is above the historical maximum height; and

annotating the displayed image data at a time in which the height of the smart playable device is greater than the historical maximum height.

18. The non-transitory computer-readable medium of claim 15, the operations further comprising:

storing video data associated with gameplay of the smart playable device;

determining one or more game events based on the sensor data received from the smart playable device;

identifying, as identified video data, at least one portion of the video data based at least in part on the one or more game events; and

presenting the identified video data on the display of the computing device.

19. The non-transitory computer-readable medium of claim 15, the operations further comprising:

annotating at least a portion of the displayed image data with an animation based on gameplay associated with a game application selected by a user and operating in conjunction with the smart playable device.