Stroke operation prediction for three-dimensional digital content

• US 10,627,910 B2
• Filed: 02/21/2017
• Issued: 04/21/2020
• Est. Priority Date: 02/21/2017
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Inc.
• Inventors: Lu, Jingwan, DiVerdi, Stephen J., Kim, Byungmoon, Xing, Jun
• Primary Examiner(s): Patel, Nitin
• Assistant Examiner(s): Onyekaba, Amy
• Application Number:
  US15/438,276
• Publication Number:
  US 20180239434A1
• Time in Patent Office: 1,155 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 11/203 Drawing of straight lines or curves
• CPC Class Codes: G06T 15/02 Non-photorealistic rendering
• CPC Class Codes: G06T 17/30 Polynomial surface description
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics

Abstract

Stroke operation prediction techniques and systems for three-dimensional digital content are described. In one example, stroke operation data is received that describes a stroke operation input via a user interface as part of the three-dimensional digital content. A cycle is generated that defines a closed path within the three-dimensional digital content based on the input stroke operation and at least one other stroke operation in the user interface. A surface is constructed based on the generated cycle. A predicted stroke operation is generated based at least in part on the constructed surface. The predicted stroke operation is then output in real time in the user interface as part of the three-dimensional digital content as the stroke operation data is received.

Claims

1. In a digital medium environment to generate three-dimensional digital content, a method implemented by at least one computing device configured to be worn by a user, the method comprising:

receiving, by the at least one computing device, stroke operation data describing an input stroke operation received via a user interface with respect to the three-dimensional digital content, the input stroke operation detected by a sensor of the at least one computing device as a hand gesture of the user, and the three-dimensional digital content defining at least a portion of an augmented or virtual reality environment;

generating, by the at least one computing device, a cycle that defines a closed path within the three-dimensional digital content based on the input stroke operation and at least one other stroke operation in the user interface;

constructing, by the at least one computing device, a surface based on the generated cycle;

generating, by the at least one computing device, a predicted stroke operation based at least in part on the constructed surface; and

outputting, by an output device associated with the at least one computing device, the predicted stroke operation in real time in the user interface as part of the three-dimensional digital content as the stroke operation data is received, the predicted stroke operation being user selectable within a set amount of time to cause the predicted stroke operation to be included as part of the three-dimensional digital content.

2. The method as described in claim 1, wherein:

the stroke operation data includes a plurality of operation data samples; and

the generating of the cycle and the constructing of the surface are performed in real time as the plurality of operation data samples are received.

3. The method as described in claim 2, wherein each of the plurality of operation data samples includes a spatial parameter, an appearance parameter, and a temporal parameter.

4. The method as described in claim 1, wherein the generating of the cycle is based on the input stroke operation and at least one constraint used to select the at least one other stroke operation in the user interface from a plurality of other stroke operations in the user interface.

5. The method as described in claim 1, wherein the generating of the cycle is based on the input stroke operation and at least one constraint used to identify the at least one other stroke operation in the user interface, the at least one constraint defining a threshold spatial or temporal distance between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content.

6. The method as described in claim 1, wherein the generating of the cycle is based on the input stroke operation and at least one constraint used to identify the at least one other stroke operation in the user interface, the at least one constraint defining a direction between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content.

7. The method as described in claim 6, wherein the direction is defined through a relationship of a surface normal of the input stroke operation and a surface normal of the at least one other stroke operation.

8. The method as described in claim 1, wherein the generating of the cycle is based on the input stroke operation and at least one constraint used to identify the at least one other stroke operation in the user interface, the at least one constraint defining a minimum amount of the input stroke operation to be used as part of the generated cycle as defined between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content.

9. The method as described in claim 1, wherein the generating of the cycle is based on the input stroke operation and at least one constraint used to identify the at least one other stroke operation in the user interface, the at least one constraint specifying that the at least one other stroke operation does not loop back to a starting point at which the at least one other stroke operation intersects the input stroke operation.

10. The method as described in claim 1, wherein the generating of the cycle includes constructing the at least one other stroke operation automatically and without user intervention to form the cycle, the at least one other stroke operation not included in the user interface before the constructing of the at least one other stroke operation.

11. The method as described in claim 1, wherein the generating of the cycle includes generating a plurality of candidate cycles based on the input stroke operation and selecting the cycle from the plurality of candidate cycles.

12. The method as described in claim 1, wherein the constructing of the surface includes generating mesh vertices of the surface based on the generated cycle using bilinear interpolation.

13. The method as described in claim 1, wherein the generating of the predicted stroke operation includes:

projecting the stroke operation data into a two-dimensional stroke operation projection based on the constructed surface and depth information defining a distance of the two-dimensional stroke operation projection to the constructed surface;

calculating a two-dimensional stroke operation prediction based on the two-dimensional stroke operation projection; and

projecting the two-dimensional stroke operation projection into three dimensions to generate the predicted stroke operation based on the distance to the constructed surface.

14. The method as described in claim 1, wherein the generating of the predicted stroke operation is based on high-level structure exhibited by existing stroke operations in the user interface.

15. In a digital medium environment to generate three-dimensional digital content, a system comprising:

a cycle generation module implemented at least partially in hardware of a computing device configured to be worn by a user to generate a cycle that defines a closed path within the three-dimensional digital content based on an input stroke operation, at least one other stroke operation in a user interface, and at least one constraint, the input stroke operation detected by a sensor of the computing device as a gesture of the user that defines a line in three dimensions in the user interface, and the three-dimensional digital content defining at least a portion of an augmented or virtual reality environment;

a surface construction module implemented at least partially in the hardware of the computing device to construct a surface based on the generated cycle; and

a stroke prediction module implemented at least partially in the hardware of the computing device to generate a predicted stroke operation based at least in part on the constructed surface, the predicted stroke operation being user selectable to cause the predicted stroke operation to be included as part of the three-dimensional digital content.

16. The system as described in claim 15, wherein the at least one constraint defines:

a threshold temporal or spatial distance between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content;

a direction between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content;

or

a minimum amount of the input stroke operation to be used as part of the generated cycle as defined between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content.

17. In a digital medium environment to generate three-dimensional digital content, a system comprising:

means for generating a cycle that defines a closed path within the three-dimensional digital content based on at least one constraint, an input stroke operation, and at least one other stroke operation in a user interface used to generate the three-dimensional digital content, the input stroke operation detected by a sensor of a computing device, and the three-dimensional digital content defining at least a portion of an augmented or virtual reality environment;

means for constructing a surface based on the generated cycle; and

means for displaying a predicted stroke operation in the user interface based at least in part on the constructed surface, for a set amount of time and if the predicted stroke operation is not selected within the set amount of time, then the predicted stroke operation is automatically removed from the user interface.

18. The system as described in claim 17, wherein the at least one constraint defines:

a threshold temporal or spatial distance between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content;

a direction between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content;

or

a minimum amount of the input stroke operation to be used as part of the generated cycle as defined between the input stroke operation and the at least one other stroke operation within the three-dimensional digital content.

19. The system as described in claim 17, wherein the input stroke operation is detected by the sensor of the computing device as a gesture of a user that defines a line in three dimensions in the user interface.

20. The system as described in claim 17, further comprising means for generating mesh vertices of the surface based on the generated cycle using bilinear interpolation.