Systems and methods for virtually displaying real movements of objects in a 3D-space by means of 2D-video capture

• US 9,360,932 B1
• Filed: 05/23/2013
• Issued: 06/07/2016
• Est. Priority Date: 08/29/2012
• Status: Active Grant

• Current Assignee: Intellect Motion, LLC
• Original Assignee: Intellect Motion, LLC
• Inventors: Lukashevich, Uladzislau Anatolievich
• Primary Examiner(s): Bost, Dwayne
• Assistant Examiner(s): Ritchie, Darlene M
• Application Number:
  US13/900,551
• Time in Patent Office: 1,111 Days
• Field of Search: 345/419, 482/142, 340/573.1, 375/240.08, 715/757, 733/790.4, 356/614, 348/207.99, 463/8
• US Class Current: 1/1
• CPC Class Codes: A61B 2505/09 Rehabilitation or training
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/1127 using markers
• CPC Class Codes: A61B 5/4887 Locating particular structures in or on the body
• CPC Class Codes: A63B 2024/0015 Comparing movements or motion sequences with computerised simulations of movements or motion sequences, e.g. for generating an ideal template as reference to be achieved by the user
• CPC Class Codes: A63B 24/0006 Computerised comparison for qualitative assessment of motion sequences or the course of a movement
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

The disclosed method is intended for virtually reproducing real movements of an individual by 2D measurement of positions of two markers one of which is located, e.g., on a body trunk and another is held in a hand. A human being is selected as an object for implementation of the method based on specificity of movements of the body parts as a reaction of the muscle-joint sense in response to the visual information obtained through the visual channel. Video catching is performed by detecting the detectable elements such as LEDs attached to the markers by means of a web camera with visual matrix, and processing the video catching in a CPU into movement of a single cursor on the display. The method may find use in fitness, gaming, simulation, manipulation of antropometric robots, sport, medicine, etc.

Claims

1. A computer-implemented method for virtually displaying real movements of objects in a 3D-space by means of 2D-video capture, the method being performed in connection with a system comprising:

at least a first marker and a second marker, both markers having detectable elements laying in planes;

a web camera that has a visual matrix visual matrix sensor that detects the detectable elements for creating a 2D video capture information about movements of said first marker and second marker, the visual matrix visual matrix sensor having a visual matrix having a plane with a pixel coordinate system onto which the images of the detectable elements are projected;

a display having a display screen with an image of at least one target; and

a central processing unit that communicates with the web camera and with the display and that processes the 2D capture information into 3D video capture signals, the method comprising;

providing an object having at least a first part having a predetermined mobility and a second part having greater mobility than the predetermined mobility of the first part, both parts being independently moveable;

allocating a first marker on the first part and the second marker on the second part of the object;

arranging the first part and the second part in positions where the first marker and the second marker fall into the vision field of the web camera;

moving the first part and a second part with respective markers in space in an X₁, Y₁, Z₁ coordinate system, where axes X₁ and Y₁ form an X₁-Y₁ plane parallel to the plane of the respective marker, and axis Z₁ is directed from the X₁-Y₁ plane toward the visual matrix sensor of the web camera;

performing video catching of the movements of the first marker and of the second marker by detecting the detectable elements of the first marker and the second marker thus producing video catching information;

sending the video catching information to the central processing unit and processing the video catching information by converting the movement of the first marker and the second marker into movement of a single cursor on the display;

moving the parts of the object with markers for placing the cursor to a desired position relative to the target;

selecting an arbitrary point on each marker as a point of marker that is in a constant position relative to the marker and the detectable elements of the marker;

assuming the center of the visual matrix as a center point of the matrix for subsequent calculations;

determining pixel coordinates of the point of marker relative to the center point of the visual matrix;

sending the obtained pixel coordinates of the point of marker from the visual matrix to the central processing unit for calculating the coordinates of the point of marker with the use of the algorithm and with a predetermined frequency thus obtaining the trajectories of movement of the points of markers in the pixel coordinates of the visual matrix;

using the pixel coordinate system of the visual matrix as a common coordinate system;

forming motion vectors from the center point of the matrix coordinate system to the current point on the trajectory of movement of the points of markers;

summing the motion vectors of the points of markers into a common vector; and

assigning the movement of the common vector to the cursor on the screen of the display so that the movements of the first and the second markers are presented on the screen of the display as movements of the cursor relative to the target; and

wherein at least one marker has at least three detectable elements located in predetermined positions unchangeable with respect to the marker so that when the respective marker moves in the direction of axis Z1 the distances between the detectable elements projected onto the visual matrix change, and these changes are used for determining coordinates of the marker on the Z1 axis; and

when the marker is turned from the X1-Y1 plane, the distances between the detectable elements in the visual matrix change and these changes can be presented as turns of the marker from the X1-Y1 plane.

2. The computer-implemented method according to claim 1, wherein the object is a human being having a visual channel that obtains visual information and a channel of a muscle-joint sense, the first part is a body part other than a hand, and second part is a hand, the human being selected based on specificity of a movements of body parts as a reaction of the muscle-joint sense in response to the visual information obtained through the visual channel.

3. The computer-implemented method according to claim 2, wherein the first marker comprises at least one detectable element, and the second marker comprises at least three detectable elements.

4. The computer-implemented method according to claim 3, comprising a step of selecting detectable elements from the group consisting of a light source, a mirror, a characteristic part of a human body, a characteristic part of a human garment, and a characteristic object attached to a human garment.

5. The computer-implemented method according to claim 1, wherein in the course of the movement of the markers the pixel coordinates of the detectable elements of each marker on the visual matrix are associated with a X, Y, Z system, where the axes X and Y lay in the matrix plane, and the axis Z is perpendicular to the X-Y plane and has the same scale as the scale on the axes X and Y.

6. A computer-implemented method for virtually displaying real movements of an individual having a first body part of a predetermined mobility and a second body part having mobility higher than the predetermined mobility of the first body part, the individual performing real movements in a 3D-space, the method being performed in connection with a system comprising:

at least a first marker and a second marker, both markers having detectable elements located on each marker in predetermined positions unchangeable with respect to the marker;

a web camera that has a visual matrix sensor that detects the detectable elements for creating a 2D video capture information about movements of said first marker and second marker, the visual matrix sensor having a visual matrix having a plane with a pixel coordinate system onto which the images of the detectable elements are projected;

a display having a display screen with an image of at least one target; and

a central processing unit that communicates with the web camera and with the display and that processes the 2D capture information into 3D video capture signals, the method comprising;

providing allocating the first marker on the first part and the second marker on the second part of the individual;

arranging the first part and the second part in positions where the first marker and the second marker fall into the vision field of the web camera;

moving the first body part and a second body part with respective markers in space in an X₁, Y₁, Z₁ coordinate system, where axes X₁ and Y₁ form an X₁-Y₁ plane parallel to the plane of the respective marker, and axis Z₁ is directed from the X₁-Y₁ plane toward the visual matrix sensor of the web camera;

performing video catching of the movements of the first marker and of the second marker by detecting the detectable elements of the first marker and the second marker thus producing video catching information;

sending the video catching information to the central processing unit and processing the video catching information by converting the movement of the first marker and the second marker into movement of a single cursor on the display;

moving the parts of the object with markers for placing the cursor to a desired position relative to the target;

selecting an arbitrary point on each marker as a point of marker that is in a constant position relative to the marker and the detectable elements of the marker;

assuming the center of the visual matrix as a center point of the matrix for subsequent calculations;

determining pixel coordinates of the point of marker relative to the center point of the visual matrix;

sending the obtained pixel coordinates of the point of marker from the visual matrix to the central processing unit for calculating the coordinates of the point of marker with the use of the algorithm and with a predetermined frequency thus obtaining the trajectories of movement of the points of markers in the pixel coordinates of the visual matrix;

using the pixel coordinate system of the visual matrix as a common coordinate system;

forming motion vectors from the center point of the matrix coordinate system to the current point on the trajectory of movement of the points of markers;

summing the motion vectors of the points of markers into a common vector; and

assigning the movement of the common vector to the cursor on the screen of the display so that the movements of the first and the second markers are presented on the screen of the display as movements of the cursor relative to the target; and

wherein at least one marker has at least three detectable elements located in predetermined positions unchangeable with respect to the marker so that when the respective marker moves in the direction of axis Z1 the distances between the detectable elements projected onto the visual matrix change, and these changes are used for determining coordinates of the marker on the Z1 axis; and

when the marker is turned from the X1-Y1 plane, the distances between the detectable elements in the visual matrix change and these changes can be presented as turns of the marker from the X1-Y1 plane.

7. The computer-implemented method according to claim 6, wherein the first body part is a part other than a hand, and a second body part is a hand, the individual is selected based on specificity of a movements of the body parts as a reaction of the muscle-joint sense in response to the visual information obtained through the visual channel.

8. The computer-implemented method according to claim 7, wherein the first marker and the second marker each comprises at least three detectable elements.

9. The computer-implemented method according to claim 8, comprising a step of selecting detectable elements from the group consisting of a light source, a mirror, a characteristic part of a human body, a characteristic part of a human garment, and a characteristic object attachable to a human garment.

10. The computer-implemented method according to claim 6, wherein in the course of the movement of the markers the pixel coordinates of the detectable elements of each marker on the visual matrix are associated with a X, Y, Z system, where the axes X and Y lay in the matrix plane, and the axis Z is perpendicular to the X-Y plane and has the same scale as the scale on the axes X and Y.