Method and system for ink data generation, ink data rendering, ink data manipulation and ink data communication

• US 9,766,804 B2
• Filed: 04/14/2016
• Issued: 09/19/2017
• Est. Priority Date: 11/19/2013
• Status: Active Grant

• Current Assignee: Wacom Co Limited (Wacom)
• Original Assignee: Wacom Co Limited (Wacom)
• Inventors: Angelov, Branimir, Nakayama, Yoshitaka
• Primary Examiner(s): Duong, Hien
• Application Number:
  US15/099,311
• Publication Number:
  US 20160253090A1
• Time in Patent Office: 523 Days
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 2203/04105 Pressure sensors for measuring the pressure or force exerted on the touch surface without providing the touch position
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/0414 using force sensing means to determine a position
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/0447 Position sensing using the local deformation of sensor cells
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services
• CPC Class Codes: G06T 11/203 Drawing of straight lines or curves
• CPC Class Codes: G06V 30/1423 the instrument generating sequences of position coordinates corresponding to handwriting preprocessing or recognising digital ink G06V30/32
• CPC Class Codes: G06V 30/347 Sampling; Contour coding; Stroke extraction

Abstract

A method implemented by a transmission device to communicate with multiple reception devices that respectively share a drawing area with the transmission device is provided. The transmission device transmits to the multiple reception devices vector-data ink data representative of traces of input operation detected by an input sensor of the transmission device. The method includes: (a) an ink data generation step of generating fragmented data of a stroke object, wherein the stroke object contains multiple point objects to represent a trace formed by a pointer, the fragmented data being generated per defined unit T, and generating a drawing style object; (b) a message formation step of generating messages including the drawing style object and the fragmented data; and (c) a transmission step of transmitting the messages.

Claims

1. A method implemented by a transmission device to communicate with multiple reception devices that respectively share a drawing area with the transmission device, wherein the transmission device transmits to the multiple reception devices vector-data ink data representative of traces of input operation detected by an input sensor of the transmission device, the method comprising:

(a) an ink data generation step that comprises;

(i) continuously inputting pen even data (INPUT 1) generated according to movement of a pointer, and generating fragmented data of a stroke object, wherein the stroke object contains multiple point objects to represent a trace of said movement of the pointer, the fragmented data being generated in units of T, wherein T is a defined unit used to fragment data to generate the fragmented data, and

(ii) generating a drawing style object, which defines a rendering configuration to be applied to the whole of said trace of the stroke object based on context information (INPUT 2), at a pen down time corresponding to generation of the pen event data at a beginning point of said trace;

(b) a message formation step that comprises;

(i) generating a first message that includes the drawing style object, and

(ii) generating one or more second messages subsequent to the first message, the one or more second messages including the fragmented data to be rendered with the rendering configuration defined by the drawing style object in the first message; and

(c) a transmission step of transmitting the first message including the drawing style object and the one or more second messages in sequence according to a defined communications protocol, wherein

the transmission step starts transmitting the first message at the pen down time.

2. The method of claim 1, wherein the defined unit T is selected from a group consisting of a unit time and a unit data size.

3. The method of claim 1, wherein said rendering configuration of the trace of the movement of the pointer is selected from a group consisting of a color of the trace, opacity of the trace, transparency of the trace, width of the trace, and texture of the trace.

4. The method of claim 3, wherein in step (b)(i) the first message includes the drawing style object and at least a first fragmented data, and in step (b)(ii) the second message includes the fragmented data generated subsequent to the at least first fragmented data included in the first message and does not include the drawing style object.

5. The method of claim 1, wherein step (a)(ii) of generating the drawing style object is triggered by a pen down event which comprises inputting of the pen event data at the beginning point of said trace.

6. The method of claim 1, wherein the ink data generation step of (a) further comprises (iii) generating a metadata object at a pen up time corresponding to generation of the pen event data at an ending point of said trace, and the message formation step of (b) further comprises (iii) including the metadata object in a last one of the one or more second messages.

7. The method of claim 6, wherein the first message including the drawing style object is configured to request updating of the drawing area.

8. The method of claim 7, wherein the message formation step of (b) further comprises (iv) when receiving a message indicating that said request for updating of the drawing area was rejected, aborting transmission of the one or more second messages.

9. The method of claim 1, wherein the defined unit T is smaller than a unit corresponding to said trace and is greater than a minimum unit that can be processed to draw a curve segment by interpolating point objects included in the fragmented data.

10. The method of claim 1, further comprising (d) a negotiation step, to be performed prior to the message formation step of (b), the negotiation step of (d) comprising determining a communications parameter value, which becomes a basis for determining said defined unit T of the fragmented data.

11. The method of claim 10, wherein said communications parameter value determines whether said defined unit T of the fragmented data equals, or is smaller than, a unit corresponding to said trace.

12. The method of claim 11, wherein the message formation step of (b) comprises forming the first and second messages using a first format when said communications parameter value determines said defined unit T of the fragmented data to be smaller than the unit corresponding to said trace, and forming the first and second messages using a second format different from the first format when said communications parameter value determines said defined unit T of the fragmented data to be equal to the unit corresponding to said trace.

13. The method of claim 12, wherein said first format includes a field that indicates whether a message includes an ending point of said trace represented by said stroke object, while said second format does not include such field.

14. The method of claim 12, wherein said defined unit T that is smaller than the unit corresponding to said trace is selected from a group consisting of:

(a) a unit based on one of the multiple point objects that form the stroke object, (b) a unit based on a byte that is smaller than a minimum unit defined by the communications protocol, and (c) a unit of time equal to or less than 100 msec.

15. The method of claim 10, further comprising (e) a negotiation step, to be performed prior to the message formation step of (b), the negotiation step of (e) comprising determining a communications parameter value, and the transmission step of (c) comprises transmitting the first message and the one or more second messages using a communication packet of the defined communications protocol specified by said communications parameter value.

16. The method of claim 15, wherein the communications parameter value specifies one of a first protocol, which includes a function of detecting loss of the communication packet, and a second protocol, which does not include the function of detecting loss of the communication packet, and the message formation step of (b) comprises switching between generating the first and second messages using the first protocol and generating the first and second messages using the second protocol, based on said communications parameter value.

17. The method of claim 16, wherein the message formation step of (b) comprises, when the second protocol is used, generating each of the first and second messages that includes a sequence ID to define an order of the first and second messages, and when the first protocol is used, generating each of the first and second messages without including a sequence ID.

18. The method of claim 17, further comprising (g) a message retransmission step of retransmitting a message whose sequence ID is indicated as being lost and its subsequent messages.

19. The method of claim 18, further comprising (h) a drawing area sharing step, to be performed prior to the message formation step of (b), the drawing area sharing step comprising transmitting latest information regarding the drawing area.

20. The method of claim 19, wherein the drawing area sharing step of (h) comprises generating the latest information regarding the drawing area using a first encoding method, and the message formation step of (b) comprises generating the first and second messages using a second encoding method having a smaller compression rate than the first encoding method.

21. The method of claim 20, wherein the message formation step of (b) further comprises determining a stroke ID that uniquely identifies the stroke object that is being generated amongst multiple stroke objects present within the drawing area, and including said stroke ID in at least the first message.

22. The method of claim 21, wherein the message formation step of (b) further comprises including a device ID of the transmission device in at least the first message.

23. The method of claim 19, wherein the message formation step of (b) further comprises including reproduction timing information, which indicates a time interval at which the fragmented data of the stroke object are to be reproduced by the reception device, in said stroke object.

24. The method of claim 19, wherein the ink data generation step of (a) comprises generating, based on said context information at the beginning point of said trace, variability information that indicates whether at least one of a width and a color of said trace varies according to said movement of the pointer, and the message formation step of (b) comprises including said variability information in the first message or a first of the one or more second messages.

25. The method of claim 24, wherein said width extends perpendicularly to a direction of said trace, and said variability information is a parameter that defines a range of the width in terms of at least one of a minimum width value and a maximum width value.

26. The method of claim 19, further comprising (i) a sharing step, to be performed prior to the ink data generation step of (a), the sharing step of (i) comprising transmitting a set of drawing parameters of said drawing style object to the multiple reception devices, and the ink data generation step of (a) comprises generating the drawing style object using an identifier of the set of the drawing parameters.

27. The method of claim 26, wherein the set of drawing parameters defines a set of pen-tip attributes applicable in said drawing area.

28. The method of claim 27, wherein the set of drawing parameters includes drawing resolution information that specifies accuracy of drawing coordinate positions, and the step of ink data generation of (a) comprises generating drawing coordinate positions for the point objects to represent said trace using said drawing resolution information.

29. The method of claim 28, wherein the step of ink data generation of (a) comprises scaling a vector from a first pen coordinate position of the pen event data corresponding to said beginning point of said trace to a second pen coordinate position of the pen event data corresponding to an ending point of said trace, using said drawing resolution information.

30. The method of claim 29, wherein said sharing step of (i) comprises transmitting a drawing coordinate position of an origin of the drawing area, and said ink data generation step of (a) comprises encoding a drawing coordinate position corresponding to said beginning point of said trace relative to said origin.

31. The method of claim 30, wherein the sharing step of (i) comprises transmitting multiple sets of drawing parameters of said drawing style object to the multiple reception devices, and subsequently transmitting an identifier of a selected one of the multiple sets of drawing parameters to the multiple reception devices.

32. The method of claim 1, wherein the defined unit T is defined by an application that uses the ink data.

33. The method of claim 32, wherein when the application uses audio in synchronization with the ink data, the defined unit is smaller than a unit corresponding to said trace when the ink data is transmitted with audio.

34. The method of claim 32, wherein when the application is used with a recognition engine that forms the ink data in a unit based on semantics, the defined unit T is greater than a unit corresponding to said trace.

35. The method of claim 32, further comprising the step of varying the frequency with which to include absolute coordinates in the point objects, wherein the point objects normally include relative coordinates, according to a type of protocol to be used for transmission.

36. The method of claim 32, further comprising the step of including an absolute value in a beginning point object of each of the one or more second messages and including an absolute value in a non-beginning point object of each of the one or more second messages.