3D digital painting

• US 10,521,951 B2
• Filed: 02/01/2019
• Issued: 12/31/2019
• Est. Priority Date: 06/01/2010
• Status: Active Grant

• Current Assignee: Vladimir Vaganov
• Original Assignee: Vladimir Vaganov
• Inventors: Vaganov, Vladimir
• Primary Examiner(s): Lewis, David L
• Assistant Examiner(s): Hoel, Matthew D
• Application Number:
  US16/265,767
• Publication Number:
  US 20190206112A1
• Time in Patent Office: 333 Days
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: B44D 2/002 Kits for drawing or painting
• CPC Class Codes: B44F 7/00 Designs imitating three-dimensional effects
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06T 15/02 Non-photorealistic rendering
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/20 Image signal generators
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/31 using parallax barriers
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 2213/008 : Aspects relating to glasses for viewing stereoscopic images

Abstract

A method of digital continuous and simultaneous three-dimensional painting and three-dimensional drawing with steps of providing a digital electronic display capable of presenting two pictures for a right eye and a left eye; providing means for creating a continuous 3D virtual canvas by digitally changing a value and sign of horizontal disparity between two images for the right eye and the left eye and their scaling on the digital electronic display corresponding to instant virtual distance between the user and an instant image within the virtual 3D canvas; providing at least one multi-axis input control device allowing digital painting or drawing on the digital electronic display; painting within virtual 3D canvas by providing simultaneous appearance of a similar stroke on the images for the right eye and the left eye on the digital electronic display.

Claims

1. A method of digital continuous and simultaneous three-dimensional painting, three-dimensional. drawing, and three-dimensional cursor (object, image) navigating, said method comprising:

providing a digital electronic display having a physical surface and a geometrical surface and configured for presenting two images;

one for a right eye and the other for a left eye of a user in front of the digital electronic display;

providing means for three-dimensional digital vision;

providing means for three-dimensional image presentation comprising a processor;

providing means for creating a continuous 3D virtual canvas comprising the geometrical surface of the digital electronic display and a virtual volume that includes the geometrical surface of the digital electronic display in said 3D virtual canvas by digitally changing a value and a sign of horizontal disparity between two images for the right eye and the left eye and their scaling on the digital electronic display corresponding to instant virtual distance between the user'"'"'s eyes and an instant (3D) image within the virtual 3D canvas;

wherein a resolution Δ

of continuity of changing of the virtual distance Z between the user and the virtual images within 3D virtual canvas is defined by a size p of a pixel on the digital electronic display in horizontal direction and by a distance d between pupils of the user'"'"'s eyes according to an expression;

Δ

≈

2p Z/d;

providing at least one input control device comprising;

a system of sensors that provide an input information about free 3D motion of at least one part of the user'"'"'s body into the at least one input control device for digital painting or drawing within 3D virtual canvas;

providing at least one kind of a coupling between at least part of the at least one input control device and the at least one part of the user'"'"'s body, said coupling chosen from a group consisting of;

mechanical coupling, optical coupling, electromagnetic coupling, sound coupling, ultrasound coupling, and a combination of two or more thereof;

moving the at least one part of the user'"'"'s body while the system of sensors within the at least one input control device is providing information for recording change of vectors of mechanical motion parameters of the at least one part of the user'"'"'s body, said system of sensors provide simultaneous appearance of similar scaled strokes or lines on the images for the right and the left eye for any instant position within 3D virtual canvas;

wherein a simultaneousness of appearance of said similar scaled strokes or lines on the images for the right and the left eye is limited by a smallest time interval equal to an inverted frequency of refreshment of frames on the digital electronic display and wherein a motion for making strokes or lines in all three dimensions is provided simultaneously and continuously in all directions of a 3D virtual space by free moving the at least one part of the user'"'"'s body.

2. A method according to claim 1, wherein free linear and angular motion of the at least one part of the user'"'"'s body for making simultaneous and continuous strokes or lines in all three-dimensions is detected by the system of sensors providing an input information about free 3D motion into the at least one input control device for digital painting or drawing within 3D virtual canvas, wherein the system of sensors are chosen from a group of sensors consisting of:

a one, two or three-axis force sensor;

a pressure sensor;

a one, two or three-axis tactile sensor;

a one, two or three-axis linear accelerometer;

a one or two-axis tilt sensor;

a one, two or three-axis gyro sensor;

a one, two or three-axis angular accelerometer;

a one, two or three-axis magnetometer;

6D or 9D Inertial Measurement Unit (IMU);

an altimeter;

an optical sensor;

a photo-sensor array;

an electromagnetic sensor;

an ultrasound sensor;

a microphone; and

. combinations, which provide a multi-axis sensor system for determining linear and angular motion parameters of the at least one part of the user'"'"'s body, which has been moved between recording of two virtual positions of the at least one input control device within 3D virtual canvas.

3. A method according to claim 1, wherein the means for three-dimensional image presentation comprising the processor provide:

means for receiving inputs from at least one multi-axis input control device for three-dimensional image presentation for painting of lines or strokes along and around three axes X, Y and Z within XYZ virtual space and wherein at least one input is used by the means for three-dimensional image presentation for painting or drawing lines or strokes along axis from the user'"'"'s eyes toward the digital electronic display and beyond by changing a horizontal disparity between images for the right eye and the left eye and their scaling, making three-dimensional paintings or drawings equally continuous in XY plane and XZ and YZ virtual planes; and

means for using a motion captured by the at least one input control device for making lines or strokes, and the motion is provided simultaneously and continuously in all three dimensions by linear and angular movements of the at least one part of the user'"'"'s body; and

means for description in digital format of images for the right eye and the left eye for every and all instant positions within 3D virtual canvas;

means for complete two-dimensional images of a painting for the right eye and the left eye at any stage of painting as accumulation of all lines and strokes made up to that stage; and

means for complete three-dimensional images of a painting including its linear and angular position within 3D virtual canvas.

4. A method according to claim 1, wherein the continuous 3D virtual canvas is presented as a temporary 3D virtual grid created by a system of lines related to a chosen system of coordinates within said 3D virtual canvas, and wherein said temporary 3D virtual grid or at least one selected portion of it can be deformed, scaled, changed, adjusted, rotated or shifted relative to the user, wherein the temporary 3D virtual grid allows the user to navigate placement and scale the size of strokes or 3D sketching within initially empty 3D virtual space and also place strokes to sides, to back, to top or bottom of a painted three-dimensional object, when a front view or an entire sketch of the painted three-dimensional object is already defined and wherein the temporary 3D virtual grid can be turned off and on at any time in a process of painting.

5. A method according to claim 4, wherein the at least one selected portion of the temporary 3D virtual grid is dedicated as a toolbox for arranging and organizing multiple parameters and characteristics of the painting tools, color palette, painting process and its different steps, visual effects, setups, commands, wherein a selectable 3D toolbox is chosen from a group:

rolodex, 3D stack of sliding flat windows, rotating in different directions geometrical shapes like cube, cylinder, prism, pyramid, sphere, ellipsoid, the surface of which is used for positioning the icons activating corresponding actions, 3D tree with a trunk, branches, sub-branches and leaves corresponding to desirable toolbox organizational structure.

6. A method according to claim 1, wherein the method is used for painting:

panoramic 3D pictures, wherein 3D canvas is surrounding the user and the user can paint either by walking with 3D headset viewer within 3D canvas and painting with input control devices (“

walking mode”

) or by sitting in a chair in front of a steady display and virtually rotating a grid of the 3D canvas around him sector by sector while painting portions of the 3D panoramic pictures (“

turntable mode”

) and

localized 3D pictures, wherein 3D canvas is localized within certain area and the user can paint either by walking with 3D headset viewer around 3D canvas and painting with the input control devices (“

walking mode”

) or by sitting in the chair in front of the steady display and rotating the grid of the 3D canvas around some vertical axis located within the 3D canvas verifying this rotation by the means for three-dimensional digital vision while painting side by side localized 3D picture (“

turntable mode”

).

7. A method according to claim 1 further comprising:

providing zooming in or out of a selected portion of 3D virtual canvas at any stage of a painting process providing higher accuracy of detail painting and better perspective of how it looks like from different distance for a viewer;

freezing a position of a brush within interested area of 3D virtual canvas, when the brush is moved;

providing switching accuracy of the brush to a higher value by either switching the brush or adjusting input control on the brush, which provides required accuracy;

providing a firm and stable surface for supporting a hand with the brush, which provides more accurate motion of the hand and the hand'"'"'s fingers;

providing turning on the brush'"'"'s motion and continue to paint within interested area of three-dimensional virtual space.

8. A method according to claim 1, wherein the user is using multiple input control devices coupled to different parts of his body by different kind of coupling for controlling functions of a digital painting process chosen from a group consisting of:

action, turning on and off, navigation of a cursor (object, image) within virtual painting space, scrolling, scaling, zooming, shadowing, screening, selecting, moving, rotating, deleting, restoring, saving, opening, closing, searching, setting up, previewing, undoing, clearing, repeating, pasting, finding, replacing, inserting, formatting, grouping, ungrouping, color selection, color mixing, assigning line or stroke width, assigning brush size, assigning swatch size, assigning sponge size, assigning eraser size, assigning a canvas virtual position, assigning depth of a focal plane around an instant virtual position of the stroke within 3D virtual canvas, creating special effects, and a combination thereof.

9. A method according to claim 1, wherein the method is used for editing, enhancing, filtering and modifying three-dimensional photo images, for development of new 3D games, based on collective collaborative painting, for personalization of existing game characters by adjusting, editing, correcting or complete redrawing or repainting the existing game characters to player'"'"'s taste within games, said method additionally comprising:

providing a three-dimensional photo image;

using this 3D image, as a basis of system of coordinates, defining the 3D virtual canvas, within which photo image supposed to be edited;

painting, drawing, editing, enhancing, filtering or modifying selected areas within 3D virtual canvas with the at least one input control device and verifying the position with the means for three-dimensional digital vision;

providing complete modified and edited two-dimensional images of the 3D photo image for the right eye and the left eye;

using left and right images for presentation of edited three-dimensional photo image.

10. A method according to claim 1 further comprising multiple three-axis input control devices, multiple means for three-dimensional digital vision, multiple means for three-dimensional image presentation comprising the processor, all connected into a network of multiple users collectively and simultaneously participating in three-dimensional painting, drawing, 3D photo images editing, gaming, studying, brainstorming, and researching, wherein working with three-dimensional imaging is required.

12. A system according to claim 1, wherein said digital electronic display configured for presenting two pictures for the right eye and the left eye is chosen from a group consisting of:

projection screen for digital projection, 3D television (TV), digital autostereoscopic displays including lenticular lens or parallax barrier display, computer monitor, display of a mobile device, eye glasses or headset with two micro-displays for the right eye and the left eye for a direct projection of right and left images into a retina of an eye.

11. A system for digital continuous and simultaneous three-dimensional painting, three-dimensional drawing, three-dimensional cursor (object, image) navigating, and digital recording of three-dimensional painted and three-dimensional drawn images, said system comprising:

a digital electronic display having a physical surface and a geometrical surface and configured for presenting two pictures;

one for a right eve and the other for a left eye of a user in front of the digital electronic display;

means for three-dimensional digital vision, through which the user sees 3D images;

at least one multi-axis input control device comprising a system of sensors, which provide an input information about free 3D motion of at least one part of the user'"'"'s body into the at least one multi-axis input control device for digital painting, or drawing or navigating 3D images;

means for at least one kind of a coupling between at least part of the at least one multi-axis input control device and the at least one part of the user'"'"'s body, said coupling chosen from a group consisting of;

mechanical coupling, optical coupling, electromagnetic coupling, sound coupling, ultrasound coupling, and a combination of two or more thereof;

means for three-dimensional image presentation comprising a processor;

wherein inputs of the at least one multi-axis input control device are used by the means for three-dimensional image presentation for painting of lines or strokes along and around three axes X, Y and Z within XYZ virtual space,

means for creating a continuous 3D virtual canvas comprising the geometrical surface of the digital electronic display and a virtual volume that includes the geometrical surface of the digital electronic display within said 3D virtual canvas by digitally changing a value and sign of horizontal disparity between two images for the right eye and the left eye and their scaling on the digital electronic display corresponding to instant virtual distance between the user'"'"'s eyes and an instant (3D) image within the virtual 3D canvas;

wherein a resolution Δ

of continuity of changing of the virtual distance Z between the user and the virtual images within 3D virtual canvas is defined by a size p of a pixel on the digital electronic display in horizontal direction and by a distance d between pupils of the user'"'"'s eyes according to an expression;

Δ

≈

2p Z/d;

wherein a motion captured by the at least one multi-axis input control device is used by the means for three-dimensional image presentation for making lines or strokes, and the motion is provided simultaneously and continuously in all three dimensions by movements of the at least one part of the user'"'"'s body;

wherein a simultaneousness of appearance of similar scaled strokes or lines on the images for the right and the left eye is limited by a smallest time interval equal to an inverted frequency of refreshment of frames on the digital electronic display and wherein a motion for making strokes or lines in all three dimensions is provided simultaneously and continuously in all directions within 3D virtual canvas by free moving the at least one part of the user'"'"'s body;

wherein the means for three-dimensional image presentation provides a description in digital format of images for the right eye and the left eye on the digital electronic display for every and all instant images made within 3D virtual canvas;

wherein the means for three-dimensional image presentation provides complete two-dimensional images of a painting for the right eye and the left eye at any stage of painting as accumulation of all lines and strokes made up to that stage; and

wherein the means for three-dimensional image presentation provides complete three-dimensional images of a painting including linear and angular position within 3D virtual canvas as accumulation of all lines and strokes made to the stage of completion and the means for three-dimensional image presentation provides two-dimensional left and right images to the means for three-dimensional digital vision for presentation of three-dimensional painting.

13. A system according to claim 11, wherein said means for three-dimensional digital vision are chosen from a group consisting of:

polarized glasses for viewing only one out of two images by each eye, images are projected on a screen or a monitor with corresponding polarization for each eye;

color glasses for viewing two corresponding images;

shutter glasses, time-splitting images on two images and synchronously with shutter glasses switching images between left and right eye;

autostereoscopic, glassless systems;

eye glasses or headset with two micro-displays for a right and the left, eye for a direct projection of right and left images into a retina of a corresponding eye.

14. A system according to claim 11, wherein sensors for the system of sensors are chosen from a group of sensors consisting of:

a one, two or three-axis force sensor a pressure sensor;

a one, two or three-axis tactile sensor;

a one, two or three-axis linear accelerometer;

one or two-axis tilt sensor;

a one, two or three-axis gyro sensor;

a one, two or three-axis angular accelerometer;

a one, two or three-axis magnetometer;

a 6D or 9D Inertial Measurement Unit (IMU);

an altimeter;

an optical sensor;

a photo-sensor array;

an electromagnetic sensor;

an ultrasound sensor;

a microphone; and

combinations thereof, which provide a multi-axis sensor system for determining linear and angular motion parameters of the at least one part of the user'"'"'s body, which has been moved between recording of two positions of the at least one multi-axis input control device within 3D virtual painting space (canvas).

15. A system according to claim 11,

wherein the means for three-dimensional image presentation comprising the processor are configured to:

provide receiving inputs from the at least one multi-axis input control device for three-dimensional image presentation for painting of lines or strokes along and around three axes X, Y and Z within XYZ virtual space and wherein at least one input is used by the means for three-dimensional image presentation for painting or drawing lines or strokes along axis from the user'"'"'s eyes toward the digital electronic display and beyond by changing a horizontal disparity between images for the right eye and the left eye and by scaling the images, making three-dimensional paintings or drawings equally continuous in XY plane and XZ and YZ virtual planes; and

provide using a motion captured by the at least one multi-axis input control device for making lines or strokes, and the motion is provided simultaneously and continuously in all three dimensions by linear and angular movements of the at least one part of the user'"'"'s body; and

provide the description in digital format of images for the right eye and the left eye for every and all instant virtual positions of the strokes within 3D virtual canvas;

provide complete two-dimensional images of a painting for the right eye and the left eye at any stage of painting as accumulation of all lines and strokes made up to that stage; and

provide complete three-dimensional images of a painting including the painting'"'"'s linear and angular position within 3D virtual canvas.

16. A system according to claim 13, wherein the eye glasses or the headset with two micro-displays for a right and the left eye for the direct projection of right and left images into a respective retina of the right eye the left, eye, additionally comprises sensors and actuators chosen from a group consisting of:

multi-axis mechanical motion sensors, one or two photo-sensitive arrays for measuring movement of eye-balls of the user, three-axis magnetometer for determining relative angular position of the user and a painted three-dimensional image;

altimeter;

optical sensor;

one or two photo-camera arrays for making 3D photo-pictures or 3D video;

electromagnetic sensor and transmitter;

ultrasound sensor and transmitter;

microphone;

speakerphone; and

a combination of the sensors and the actuators, wherein the sensors and the actuators provide an expanded functionality of the eye glasses and headsets as either additional input control devices or part of other input control devices for digital continuous and simultaneous 3D painting, drawing and navigating.

17. A system according to claim 16, wherein the eye glasses or the headset with two micro-displays for a right and the left eye for the direct projection of right and left images into the respective retina of the left eye and the right eye comprises a smart phone, which provides two micro-displays for a right and the left eye within common display of the smart phone and which also has two photo-cameras on an opposite side to a display side of the smart phone, said two photo-cameras are positioned along longer side of the smart phone at a distance from each other corresponding to an average distance between human eyes and wherein said eye glasses or said headset has optical lenses for viewing 3D photo-picture or 3D video including viewing in the process of shooting and recording the 3D photo-picture or 3D video.

18. A system according to claim 17, wherein said system is configured to provide, after taking 3D stereo photo-picture or 3D video, editing, enhancing, filtering and modifying three-dimensional photo images, and to provide transmitting an edited 3D images via standard. channel by the smart phone used within the system, wherein a recipient can immediately watch 3D images using his/her cell phone that doesn'"'"'t have two photo-cameras.

19. A system according to claim 16, wherein the eye glasses or the headset with two micro-displays for the right eye and the left eye for the direct projection of right and left images into the respective retina of the left eye and the right eye comprises a universal tool, which provides two micro-displays for a right and the left eye within common display of the universal tool and which also has two photo-cameras on an opposite side to a display side of the universal tool, said two photo-cameras are positioned along longer side of the universal tool at a distance from each other corresponding to an average distance between human eyes and wherein said eye glasses or said headset has optical lenses for 3D stereo photo-picture or 3D video and wherein a narrower size of the universal tool compared to a regular smart phone allows position of the universal tool with the digital electronic display and lenses for viewing the pictures on a display such that, they would not obstruct eyes from viewing surrounding.

20. A system according to claim 11 further comprising multiple multi-axis input control devices, computers and digital electronic canvases connected into a network of multiple users collectively and simultaneously participating in three-dimensional painting, drawing, 3D photo images editing, gaming, studying, brainstorming, or researching, wherein working with three-dimensional imaging is required.