Interactive system and method

• US 10,092,843 B1
• Filed: 10/20/2017
• Issued: 10/09/2018
• Est. Priority Date: 06/24/2010
• Status: Active Grant

• Current Assignee: Steven Mark Hoffberg
• Original Assignee: Steven Mark Hoffberg
• Inventors: Rabin, Gregory S., Hoffberg, Steven M.
• Primary Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US15/789,621
• Time in Patent Office: 354 Days
• Field of Search: None
• CPC Class Codes: A63F 13/30 Interconnection arrangements between game servers and game devices; Interconnection arrangements between game devices; Interconnection arrangements between game servers
• CPC Class Codes: A63F 13/57 Simulating properties, behaviour or motion of objects in the game world, e.g. computing tyre load in a car race game A63F13/56 takes precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/812 Ball games, e.g. soccer or baseball
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/30221 Sports video; Sports image
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00

Abstract

A system and method for receiving an ordered set of images and analyzing the images to determine at least one position in space and at least one motion vector in space and time for at least one object represented in the images is disclosed. Using these vectors, a four dimensional model of at least a portion of the information represented in the images is formulated. This model generally obeys the laws of physics, though aberrations may be imposed. The model is then exercised with an input parameter, which, for example, may represent a different perspective than the original set of images. The four dimensional model is then used to produce a modified set of ordered images in dependence on the input parameter and optionally the set of images, e.g., if only a portion of the data represented in the images is modeled. The set of images may then be rendered on a display device.

Claims

1. A method, comprising:

receiving at least one stream of live action video images comprising a representation of a plurality of moving physical three dimensional objects in space subject to at least one collision interaction;

processing the at least one stream of live action video images, to define object parameters for each respective three dimensional object according to a computer-implemented physical model comprising at least a velocity, an acceleration, and a mass, and to define collision parameters for each respective collision interaction according to the laws of physics;

defining a three dimensional physically accurate graphic model of each respective three dimensional object and a movement of the respective three dimensional object;

receiving as a user input, a set of parameters comprising at least one modification parameter, representing at least one of an artificial velocity, an artificial acceleration, and an artificial mass, distinct from the defined object parameters and the defined collision parameters;

determining a conflict between a movement of respective three dimensional objects according to the at least one modification parameter, representing a modified collision interaction; and

synthesizing at least one stream of artificial action video images comprising the plurality of moving physical three dimensional objects, according to the at least one modification parameter, the defined object parameters which are consistent with the at least one modification parameter, and the defined three dimensional physically accurate graphic model of each respective three dimensional object comprising at least the velocity, the acceleration, and the mass, modified by the at least one modification parameter, and the modified collision interaction,

wherein the at least one collision interaction in the at least one stream of live action video images is altered in the at least one stream of artificial action video images.

2. The method according to claim 1, wherein the three dimensional physically accurate graphic model of each respective three dimensional object comprises object-specific information retrieved from a database.

3. The method according to claim 1, wherein said synthesized at least one stream of artificial action video images comprises a representation of a surface of at least one object which is not represented in the at least one stream of live action video images, the representation of the surface of at least one object which is not represented in the at least one stream of live action video images being derived from the three dimensional physically accurate graphic model of the respective three dimensional object.

4. The method according to claim 1, wherein said synthesized at least one stream of artificial action video images comprises a representation of a surface of at least one object which is not represented in a corresponding portion of the at least one stream of live action video images, the representation of the surface of at least one object which is not represented in the corresponding portion of the at least one stream of live action video images being derived from at least one non-corresponding portion of the at least one stream of live action video images.

5. The method according to claim 1, wherein said processing the at least one stream of live action video images comprises:

determining a projection transformation between each physical three dimensional object in space and a respective image of the at least one stream of live action video images;

defining a three dimensional spatial model of the plurality of moving physical three dimensional objects;

determining spatial coordinates, rotation vectors, and motion vectors for each respective three dimensional object represented in the at least one stream of live action video images; and

inferring the velocity, the acceleration, and the mass relationships of each respective object, and to define the collision parameters for each respective collision interaction according to the law of physics.

6. The method according to claim 1, wherein said synthesizing the at least one stream of artificial action video images comprises texture mapping surfaces of the defined three dimensional physically accurate graphic model of each respective three dimensional object based on a persistently stored database representing the hidden surfaces of at least each respective three dimensional object.

7. The method according to claim 1, wherein said processing the at least one stream of live action video images, to define object parameters for each respective three dimensional object according to the computer-implemented physical model comprising at least the velocity, the acceleration, and the mass comprises acquiring images from at least two different perspectives.

8. The method according to claim 1, wherein said processing the at least one stream of live action video images comprises determining an affine representation of at least one point in a three dimensional space.

9. The method according to claim 8, wherein said synthesizing comprises determining at least one affine disparity relating to the at least one point in the three dimensional space.

10. The method according to claim 1, said synthesizing comprises representing at least one collision of the plurality of moving physical three dimensional objects that is not present in the at least one stream of live action video images.

11. The method according to claim 1, wherein the plurality of moving physical three dimensional objects comprises a human athlete, and the at least one stream of live action video images comprises a video of a live sports competition.

12. The method according to claim 11, further comprising analyzing the at least one stream of live action video images to determine a compliance with the set of sports competition rules.

13. The method according to claim 1, wherein at least one synthesized stream of artificial action video images is generated to represent a different perspective than any of the at least one stream of live action video images.

14. A system for processing live action video images, comprising:

a user input port, configured to receive a set of parameters comprising at least one modification parameter;

an video input port, configured to receive at least one stream of live action video images comprising a representation of a plurality of moving physical three dimensional objects in space subject to at least one collision interaction;

at least one automated processor, configured to;

process the at least one stream of live action video images, to define object parameters for each respective three dimensional object according to a computer-implemented physical model comprising at least a velocity, an acceleration, and a mass, and to define collision parameters for each respective collision interaction according to the laws of physics;

define a three dimensional physically accurate graphic model of each respective three dimensional object and a movement of the respective three dimensional object;

determine a conflict between a movement of respective three dimensional objects according to the at least one modification parameter and the respective motion vector, representing a modified collision interaction;

synthesize at least one stream of artificial action video images comprising the plurality of moving physical three dimensional objects comprising at least the velocity, the acceleration, and the mass, modified by the at least one modification parameter, and the modified collision interaction, according to;

the at least one modification parameter, wherein the at least one modification parameter represents at least one of an artificial velocity, an artificial acceleration, and an artificial mass, distinct from the defined object parameters and the defined collision parameters,

the defined object parameters which are consistent with the at least one modification parameter, and

the defined three dimensional physically accurate graphic model of each respective three dimensional object,

wherein the at least one collision interaction in the at least one stream of live action video images is altered in the at least one stream of artificial action video images to the modified collision interaction dependent on at least the at least one modification parameter.

15. The system according to claim 14, wherein the at least one automated processor comprises a graphic processing unit (GPU) comprising at least one Single Instruction, Multiple Data (SIMD) processor.

16. The system according to claim 14, further comprising a database which stores an object-specific three dimensional physically accurate graphic model of each respective three dimensional object.

17. The system according to claim 14, wherein the at least one automated processor is further configured to:

determine a projection transformation between each physical three dimensional object in space and a respective image of the at least one stream of live action video images;

define a three dimensional spatial model of the plurality of moving physical three dimensional objects;

determine spatial coordinates, rotation vectors, and motion vectors for each respective three dimensional object represented in the at least one stream of live action video images; and

infer the velocity, the acceleration, and the mass relationships of each respective object, and to define the collision parameters for each respective collision interaction according to the law of physics.

18. The system according to claim 14, wherein the at least one automated processor is further configured to process the at least one stream of live action video images to determine an affine representation of at least one point in a three dimensional space, and determine an affine disparity dependent on the affine representation.

19. The system according to claim 14, wherein the plurality of moving physical three dimensional objects comprises a human athlete, and the at least one stream of live action video images comprises a video of a live sports competition, and the at least one automate processor is further configured to determine a compliance with the set of sports competition rules.

20. A system for processing live action video images, comprising:

a user input port, configured to receive a set of parameters comprising at least one modification parameter through a game controller;

a port, configured to receive an electronic representation of a plurality of moving physical three dimensional objects in space subject to at least one collision interaction, representing at least one human competitor;

at least one automated processor, configured to;

process the electronic representation, to define object parameters for each respective three dimensional object according to a computer-implemented physical model comprising at least a velocity, an acceleration, and a mass, and to define collision parameters for each respective collision interaction according to the laws of physics and a motion vector describing a movement of the respective three dimensional object;

apply a set of competition rules defining acceptable interaction between the plurality of moving physical three dimensional objects in space;

define a three dimensional graphic model of each respective three dimensional object and a movement of the respective three dimensional object;

determine a conflict between a movement of respective three dimensional objects according to the at least one modification parameter, representing a modified collision interaction;

synthesize at least one artificial stream comprising the plurality of moving physical three dimensional objects comprising at least the velocity, the acceleration, and the mass, according to;

the at least one modification parameter and the modified collision interaction, wherein the at least one modification parameter represents at least one of an artificial velocity, an artificial acceleration, and an artificial mass, distinct from the defined object parameters and the defined collision parameters,

the defined object parameters which are not inconsistent with the at least one modification parameter, and

the defined three dimensional physically accurate graphic model of each respective three dimensional object,

wherein the at least one collision interaction is altered to the modified collision interaction dependent on at least the at least one modification parameter.