3D DOCUMENT EDITING SYSTEM

• US 20180081519A1
• Filed: 09/20/2016
• Published: 03/22/2018
• Est. Priority Date: 09/20/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Kim, Seung Wook
• Granted Patent: US 10,817,126 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A 3D document editing system and graphical user interface (GUI) that includes a virtual reality and/or augmented reality device and an input device (e.g., keyboard) that implements sensing technology for detecting gestures by a user. Using the system, portions of a document can be placed at or moved to various Z-depths in a 3D virtual space provided by the VR device to provide 3D effects in the document. The sensing technology may allow the user to make gestures while entering text via a keypad, thus allowing the user to specify 3D effects in the document while typing. The system may also monitor entries made using the keypad, apply rules to the entries to detect particular types of entries such as URLs, and automatically shift the detected types of entries forward or backward on the Z axis relative to the rest of the content in the document.

Claims

1. A system, comprising:

an input device comprising;

a keypad for inputting text; and

a sensing region configured to sense gestures according to sensing technology;

a virtual reality (VR) device configured to;

display a document including text in a 3D virtual space for viewing by a user;

receive text input to the keypad by the user and display the received text in the document;

receive indications of gestures input to the sensing region of the input device by the user; and

move text areas or text portions of the document forward or backward on a Z axis relative to the document in the 3D virtual space according to the gestures.

2. The system as recited in claim 1, wherein the gestures include one or more of:

a first gesture to move a text area of the document forward on the Z axis relative to the document;

a second gesture to move a text area of the document backward on the Z axis relative to the document;

a third gesture to move a text portion of the document forward relative to other text in the document;

or

a fourth gesture to move a text portion in the document backward on the Z axis relative to other text the document.

3. The system as recited in claim 2, wherein a text area is one of a paragraph, a column, a section, a text field, or a text box.

4. The system as recited in claim 2, wherein a text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).

5. The system as recited in claim 1, wherein the sensing region is located at a position relative to the keypad on the input device that allows the user to perform the gestures with their thumbs while inputting text to the document via the keypad.

6. The system as recited in claim 1, wherein the sensing technology includes one or more of touch sensing, motion sensing, or force sensing technology.

7. The system as recited in claim 1, wherein the VR device is further configured to:

apply one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically move a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

8. A device, comprising:

a controller; and

a projector configured to display a document including text in a 3D virtual space for viewing by a user under control of the controller;

wherein the controller is configured to;

detect a gesture by the user; and

move an area or portion of the text in the document forward or backward on a Z axis relative to the document in the 3D virtual space based at least in part on the detected gesture.

9. The device as recited in claim 8, wherein, to detect the gesture, the controller is configured to receive an indication of the gesture from an input device configured to sense gestures of the user according to sensing technology.

10. The device as recited in claim 9, wherein the input device comprises:

a keypad for inputting text; and

a sensing region configured to sense the gestures of the user according to the sensing technology, wherein the sensing region is located at a position relative to the keypad on the input device that allows the user to perform the gestures with their thumbs while inputting text to the document via the keypad.

11. The device as recited in claim 10, wherein the controller is further configured to:

apply one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically move a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

12. The device as recited in claim 9, wherein the sensing technology includes one or more of touch sensing, motion sensing, or force sensing technology.

13. The device as recited in claim 8, wherein the gesture is one of:

a first gesture to move a text area of the document forward on the Z axis relative to the document;

a second gesture to move a text area of the document backward on the Z axis relative to the document;

a third gesture to move a text portion of the document forward relative to other text in the document;

or

a fourth gesture to move a text portion in the document backward on the Z axis relative to other text the document.

14. The device as recited in claim 13, wherein a text area is one of a paragraph, a column, a section, a text field, or a text box.

15. The device as recited in claim 13, wherein a text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).

16. The device as recited in claim 8, wherein the projector is one of a near-eye VR projector that projects frames including left and right images on screens that are viewed by the user, or a direct retinal VR projector that scans frames including left and right images, pixel by pixel, directly to the user'"'"'s eyes.

17. A method, comprising:

performing, by a virtual reality (VR) device;

displaying a document including text in a 3D virtual space for viewing by a user;

receiving indications of gestures by the user from an input device; and

moving, by the VR device, an area or portion of the text in the document forward or backward on a Z axis relative to the document in the 3D virtual space according to the gestures.

18. The method as recited in claim 17, wherein the input device comprises a keypad for inputting text and a sensing region for detecting gestures of the user, wherein the method further comprises the sensing region detecting gestures made by the user with their thumbs while the user is inputting text to the document via the keypad.

19. The method as recited in claim 18, further comprising:

applying one or more rules to text input to the document via the keypad to detect particular types of keypad entries as specified by the rules; and

automatically moving a text area or text portion of the document that includes a keypad entry detected according to a respective rule forward or backward on the Z axis relative to the document in the 3D virtual space as specified by the respective rule.

20. The method as recited in claim 17, wherein receiving indications of gestures by the user from an input device comprises:

receiving indications of gestures made with two thumbs to move a text area of the document forward or backward on the Z axis relative to the document, wherein a text area is one of a paragraph, a column, a section, a text field, or a text box; and

receiving indications of gestures made with one thumb to move a text portion of the document forward relative to other text in the document, wherein a text portion is one of a word, a sentence, a title, a heading, or a uniform resource locator (URL).