3D document editing system

• US 10,817,126 B2
• Filed: 09/20/2016
• Issued: 10/27/2020
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kim, Seung Wook
• Primary Examiner(s): Tan, Alvin H
• Application Number:
  US15/271,196
• Publication Number:
  US 20180081519A1
• Time in Patent Office: 1,498 Days
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A 3D document editing system and graphical user interface (GUI) that includes a virtual reality and/or augmented reality device and an input device (e.g., keyboard) that implements sensing technology for detecting gestures by a user. Using the system, portions of a document can be placed at or moved to various Z-depths in a 3D virtual space provided by the VR device to provide 3D effects in the document. The sensing technology may allow the user to make gestures while entering text via a keypad, thus allowing the user to specify 3D effects in the document while typing. The system may also monitor entries made using the keypad, apply rules to the entries to detect particular types of entries such as URLs, and automatically shift the detected types of entries forward or backward on the Z axis relative to the rest of the content in the document.

Claims

1. A system, comprising:

an input device comprising;

a keypad for inputting text; and

a sensing region configured to sense gestures according to sensing technology;

a virtual reality (VR) device configured to;

display a document including text in a 3D virtual space for viewing by a user;

receive text input to the keypad by the user and display the received text in a plurality of text areas in the document, wherein each of the text areas comprises a plurality of text portions;

receive indications of a gesture input to the sensing region of the input device by the user;

interpreting the gesture to;

determine whether the gesture corresponds to a first set of gestures associated with the plurality of text areas or a second set of gestures associated with the plurality of text portions of the document, wherein the first set of gestures is different from the second set of gestures; and

determine a direction to move a selected one of the plurality of text areas or a selected one of the plurality of text portions forward or backward based on a direction of the gesture; and

in response to the gesture, add 3D effects to the text in the selected text area or the selected text portion of the document based on the gesture corresponding to the first set of gestures or the second set of gestures, wherein the gesture moves the selected text area or the selected text portion of the document in the determined direction on a Z axis relative to the document in the 3D virtual space.

2. The system as recited in claim 1, wherein the gesture includes one of:

a first gesture to move a text area of the document forward on the Z axis relative to the document;

a second gesture to move a text area of the document backward on the Z axis relative to the document, wherein the first set of gestures comprises the first gesture and the second gesture;

a third gesture to move a text portion of the document forward relative to other text in the document;

or

a fourth gesture to move a text portion in the document backward on the Z axis relative to other text the document, wherein the second set of gestures comprises the third gesture and the fourth gesture.

3. The system as recited in claim 2, wherein a text area is one of a paragraph, a column, a section, a text field, or a text box.

4. The system as recited in claim 2, wherein a text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).

5. The system as recited in claim 1, wherein the keypad is in a first area on a surface of the input device, and wherein the sensing region is in a second area on the surface of the input device at a position relative to the keypad on the input device that allows the user to perform the gestures with their thumbs while inputting text to the document with their fingers via the keypad.

6. The system as recited in claim 1, wherein the sensing technology includes one or more of touch sensing, motion sensing, or force sensing technology.

7. The system as recited in claim 1, wherein the VR device is further configured to:

apply one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically move a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

8. A device, comprising:

a controller; and

a projector configured to display a document including text in a 3D virtual space for viewing by a user under control of the controller, wherein the text is displayed in a plurality of text areas comprising a plurality of text portions;

wherein the controller is configured to;

detect a gesture by the user;

interpreting the gesture to;

determine whether the gesture corresponds to a first set of gestures associated with the plurality of text areas or a second set of gestures associated with the plurality of text portions of the document, wherein the first set of gestures is different from the second set of gestures; and

determine a direction to move a selected text area of the plurality of text areas or a selected text portion of the plurality of text portions forward or backward based on a direction of the gesture; and

in response to the gesture, add a 3D effect to the selected text area or the selected text portion of the text the document based on the gesture corresponding to the first set of gestures or the second set of gestures, wherein the gesture moves the selected text area or the selected text portion in the determined direction on a Z axis relative to the document in the 3D virtual space.

9. The device as recited in claim 8, wherein, to detect the gesture, the controller is configured to receive an indication of the gesture from an input device configured to sense gestures of the user according to sensing technology.

10. The device as recited in claim 9, wherein the input device comprises:

a keypad for inputting text; and

a sensing region configured to sense the gestures of the user according to the sensing technology, wherein the keypad is in a first area on a surface of the input device, and wherein the sensing region is in a second area on the surface of the input device at a position relative to the keypad on the input device that allows the user to perform the gestures with their thumbs while inputting text to the document with their fingers via the keypad.

11. The device as recited in claim 10, wherein the controller is further configured to:

apply one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically move a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

12. The device as recited in claim 9, wherein the sensing technology includes one or more of touch sensing, motion sensing, or force sensing technology.

13. The device as recited in claim 8, wherein the gesture is one of:

a first gesture to move a text area of the document forward on the Z axis relative to the document;

a second gesture to move a text area of the document backward on the Z axis relative to the document, wherein the first set of gestures comprises the first gesture and the second gesture;

a third gesture to move a text portion of the document forward relative to other text in the document;

or

a fourth gesture to move a text portion in the document backward on the Z axis relative to other text the document, wherein the second set of gestures comprises the third gesture and the fourth gesture.

14. The device as recited in claim 13, wherein a text area is one of a paragraph, a column, a section, a text field, or a text box.

15. The device as recited in claim 13, wherein a text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).

16. The device as recited in claim 8, wherein the projector is one of a near-eye VR projector that projects frames including left and right images on screens that are viewed by the user, or a direct retinal VR projector that scans frames including left and right images, pixel by pixel, directly to the user'"'"'s eyes.

17. A method, comprising:

performing, by a virtual reality (VR) device;

displaying a document including text in a 3D virtual space for viewing by a user, wherein the text is displayed in a plurality of text areas comprising a plurality of text portions;

receiving an indication of a gesture by the user from an input device, the gesture corresponds to a selected text portion;

interpreting the gesture to;

determine whether the gesture corresponds to a first set of gestures associated with the plurality of text areas or a second set of gestures associated with the plurality of text portions of the document, wherein the first set of gestures is different from the second set of gestures; and

determine a direction to move the selected text area or the selected text portion forward or backward based on a direction of the gesture; and

in response to the gesture, adding 3D effects to text in the document based on the gesture corresponding to the first set of gestures or the second set of gestures, wherein the gesture moves the selected text area or the selected text portion in the determined direction on a Z axis relative to the document in the 3D virtual space.

18. The method as recited in claim 17, wherein the input device comprises a keypad for inputting text and a sensing region for detecting gestures of the user, wherein the keypad is in a first area on a surface of the input device, and wherein the sensing region is in a second area on the surface of the input device at a position relative to the keypad on the input device that allows the user to perform the gesture with their thumbs while inputting text to the document with their fingers via the keypad, and wherein the method further comprises the sensing region detecting gestures made by the user with their thumbs while the user is inputting text to the document via the keypad.

19. The method as recited in claim 18, further comprising:

applying one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically moving a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

20. The method as recited in claim 17, wherein interpreting the gesture to determine whether the gesture corresponds to the first set of gestures or the second set of gestures comprises:

determining whether the gesture was made with two thumbs to move the selected text area of the document forward or backward on the Z axis relative to the document, wherein the respective text area is one of a paragraph, a column, a section, a text field, or a text box; and

determining whether the gesture was made with one thumb to move the respective text portion of the document forward relative to other text in the document, wherein the selected text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).