Systems and methods for gesture handling in a three-dimensional visual scene

• US 10,459,526 B2
• Filed: 07/12/2017
• Issued: 10/29/2019
• Est. Priority Date: 07/12/2017
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Gontovnik, Danil, Ng, Yu Hang, Hanchar, Siarhei, Slater, Michael, Anpilov, Sergei Viktorovich
• Primary Examiner(s): Duffy, David
• Assistant Examiner(s): Doshi, Ankit B
• Application Number:
  US15/648,031
• Publication Number:
  US 20190018494A1
• Time in Patent Office: 839 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/00 Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

In one embodiment, a method includes displaying, on a client system, a visual scene with one or more first objects and one or more second objects. The one or more second objects are associated with an augmented reality context. A first touch event handler, associated with an operating system running on the client system, may receive a set of touch events and send the set of touch events to a second touch event handler running on the client system. The second touch event handler may detect a first subset of touch events relating to the one or more second objects. The second touch event handler may process the first subset of touch events and send a second subset of touch events relating to the one or more first objects to the first touch event handler. The first touch event handler may process the second subset of touch events.

Claims

1. A method comprising:

displaying, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;

receiving, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;

sending, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;

detecting, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;

processing, by the second touch event handler, the first subset of touch events relating to the one or more second objects;

sending, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;

processing, by the first touch event handler, the second subset of touch events relating to the one or more first objects;

sending, to the client system associated with the user, instructions for updating the visual scene in response to the processing of the first subset of touch events or the processing of the second subset of touch events; and

updating, on the client system associated with the user, the visual scene for display in response to receiving the instructions.

2. The method of claim 1, wherein processing of the second subset of touch events relating to the one or more first objects by the first touch event handler is done synchronously.

3. The method of claim 2, wherein the first touch event handler is an OS touch event handler.

4. The method of claim 1, wherein processing of the first subset of touch events relating to the one or more second objects by the second touch event handler is done asynchronously.

5. The method of claim 4, wherein the second touch event handler is associated with an application configured to generate the AR context.

6. The method of claim 1, wherein the detecting of the first subset of touch events relating to the one or more second objects is performed using a ray casting technique.

7. The method of claim 1, wherein the one or more second objects are three-dimensional (3D) objects that are overlaid on top of a real-world scene captured by a camera of the client system.

8. The method of claim 1, wherein the one or more first objects are system-level user interface objects.

9. The method of claim 1, wherein the set of touch events comprises one or more of a tap gesture, a drag gesture, a touch-up gesture to scroll upwards, a touch-down gesture to scroll downwards, a pinch-to-zoom gesture, or a swipe gesture.

10. The method of claim 1, wherein the sending of the set of touch events to the second touch event handler comprises:

injecting, by the first touch event handler, the set of touch events and information associated with each touch event in a data structure.

11. The method of claim 10, wherein the information associated with a touch event in the data structure comprises position or location coordinates where the touch event is performed and type of gesture associated with the touch event.

12. The method of claim 10, wherein the sending of the second subset of touch events to the first touch event handler comprises:

re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler into the data structure.

13. The method of claim 12, wherein re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler comprises:

re-creating the second subset of touch events for processing by the first touch event handler.

14. The method of claim 13, wherein re-injecting the second subset of touch events that are not processed or consumed by the second touch event handler comprises:

determining to re-create the second subset of touch events based on a type of the OS running on the client system.

15. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

display, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;

receive, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;

send, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;

detect, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;

process, by the second touch event handler, the first subset of touch events relating to the one or more second objects;

send, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;

process, by the first touch event handler, the second subset of touch events relating to the one or more first objects;

send, to the client system associated with the user, instructions for updating the visual scene in response to the processing of the first subset of touch events or the processing of the second subset of touch events; and

update, on the client system associated with the user, the visual scene for display in response to receiving the instructions.

16. The media of claim 15, wherein the first touch event handler is an OS touch event handler that is configured to process the second subset of touch events synchronously.

17. The media of claim 15, wherein the second touch event handler is associated with an application configured to generate the AR context, and wherein the second touch event handler is configured to process the first subset of touch events asynchronously.

18. A system comprising:

one or more processors; and

a non-transitory memory coupled to the processors comprising instructions executable by the processors, the processors operable when executing the instructions to;

display, on a client system associated with a user, a visual scene comprising one or more first objects and one or more second objects, wherein the one or more second objects are associated with an augmented reality (AR) context in the scene;

receive, by a first touch event handler associated with an operating system (OS) running on the client system, a set of touch events;

send, by the first touch event handler to a second touch event handler running on the client system, the set of touch events;

detect, by the second touch event handler, a first subset of touch events in the set of touch events relating to the one or more second objects that are associated with the AR context in the scene;

process, by the second touch event handler, the first subset of the touch events relating to the one or more second objects;

send, by the second touch event handler to the first touch event handler, a second subset of touch events in the set of touch events relating to the one or more first objects;

process, by the first touch event handler, the second subset of touch events relating to the one or more first objects;

send, to the client system associated with the user, instructions for updating the visual scene in response to the processing of the first subset of touch events or the processing of the second subset of touch events; and

update, on the client system associated with the user, the visual scene for display in response to receiving the instructions.

19. The system of claim 18, wherein the first touch event handler is an OS touch event handler that is configured to process the second subset of touch events synchronously.

20. The system of claim 18, wherein the second touch event handler is associated with an application configured to generate the AR context, and wherein the second touch event handler is configured to process the first subset of touch events asynchronously.