Systems and Methods for Interacting with a Projected User Interface

• US 20150089453A1
• Filed: 09/25/2014
• Published: 03/26/2015
• Est. Priority Date: 09/25/2013
• Status: Abandoned Application

• Current Assignee: Aquifi, Inc.
• Original Assignee: Aquifi, Inc.
• Inventors: Dal Mutto, Carlo, Rafii, Abbas, Hummel, Britta
• Application Number:
  US14/497,090
• Publication Number:
  US 20150089453A1
• Time in Patent Office: Days
• US Class Current: 715/852
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

A system and method for providing a 3D gesture based interaction system for a projected 3D user interface is disclosed. A user interface display is projected onto a user surface. Image data of the user interface display and an interaction medium are captured. The image data includes visible light data and IR data. The visible light data is used to register the user interface display on the projected surface with the Field of View (FOV) of at least one camera capturing the image data. The IR data is used to determine gesture recognition information for the interaction medium. The registration information and gesture recognition information is then used to identify interactions.

Claims

1. A processing system configured to conduct Three Dimensional (3D) gesture based interactive sessions for a projected user interface display comprising:

a memory containing an image processing application; and

a processor directed by the image processing application read from the memory to;

receive image data that includes visible light image data and Infrared (IR) image data,

obtain visible light image data from the image data,

generate registration information for a user interface display on a projected surface with a field of view of one or more image capture devices using the visible light image data,

obtain IR image data from the image data, and

generate gesture information for an interaction medium using the IR data, and

identify an interaction with an interactive object with the user interface display using the gesture information and the registration information.

2. The processing system of claim 1 wherein the generating of the registration information includes determining geometric relationship information that relates the FOV of the at least one camera to the user interface display on the projection surface.

3. The processing system of claim 2 wherein the geometric relationship is the homography between the FOV of the at least one camera and the user interface display on the projection surface.

4. The processing system of claim 3 wherein the geometric relationship information is determined based upon AR tags in the projected user interface display.

5. The processing system of claim 4 wherein the projected user interface display includes at least four AR tags.

6. The processing system of claim 4 wherein the AR tags are interactive objects in the user interface display.

7. The processing system of claim 1 wherein the generating of the registration information includes determining 3D location information for the projection surface indicating a position of the projection surface in 3D space.

8. The processing system of claim 7 wherein the 3D location information is determined based upon fiducials within the user interface display.

9. The processing system of claim 8 wherein the user interface display includes at least 3 fiducials.

10. The processing system of claim 3 wherein each fiducial in the user interface display is an interactive object in the user interface display.

11. The processing system of claim 1 wherein the interaction medium is illuminated with an IR illumination source.

12. The processing system of claim 1 wherein the visible light image data is obtained from images captured by the at least one camera that include only the projected user interface display on the projection surface.

13. The process system of claim 1 wherein the visible light image data is obtained from images captured by the at least one camera that include the interaction medium and the projected user interface display on the projection surface.

14. The processing system of claim 1 wherein the IR image data is obtained from images captured by the at least one camera that include the interaction medium and the projected user interface display on the projected surface.

15. The processing system of claim 1 wherein the image data is captured using at least one depth camera.

16. A method for providing a Three Dimensional (3D) gesture interactive sessions for a projected user interface display comprising:

generating a user interface display including interactive object using a processing system;

projecting the user interface display onto a projection surface using a projector;

capturing image data of the projected user interface display on the projection surface using at least one camera;

obtaining visible light image data from the image data using the processing system;

generating registration information for the user interface display on the projected surface with the field of view of one or more image capture devices providing the image data from the visible light data using the processing system;

obtaining the IR image data from the image data using the processing system; and

generating gesture information for an interaction medium in the image data from the IR image data using the processing system; and

identifying an interaction with an interactive object with the user interface display using the gesture information and the registration information.

17. The method of claim 16 wherein the generating of the registration information includes determining geometric relationship information that relates the FOV of the at least one camera to the user interface display on the projection surface using the processing system.

18. The method of claim 17 wherein the geometric relationship is the homography between the FOV of the at least one camera and the user interface display on the projection surface.

19. The method of claim 18 wherein the geometric relationship information is determined based upon AR tags in the projected user interface display.

20. The method of claim 19 wherein the projected user interface display includes at least four AR tags.

21. The method of claim 19 wherein the AR tags are interactive objects in the user interface display.

22. The method of claim 16 wherein the generating of the registration information includes determining 3D location information for the projection surface indicating a location of the projection surface in 3D space using the processing system.

23. The method of claim 22 wherein the 3D location information is determined based upon fiducials within the user interface display.

24. The method of claim 23 wherein the user interface display includes at least 3 fiducials.

25. The method of claim 23 wherein each fiducial in the user interface display is an interactive object in the user interface display.

26. The method of claim 16 further comprising:

emitting IR light towards the projected surface using at least one IR emitter to illuminate the interaction medium.

27. The method of claim 16 wherein the visible light image data is obtained from images captured by the at least one camera that include only the projected user interface display on the projection surface.

28. The method of claim 16 wherein the visible light image data is obtained from images captured by the at least one camera that include the interaction medium and the projected user interface display on the projection surface.

29. The method of claim 16 wherein the IR image data is obtained from images captured by the at least one camera that include the interaction medium and the projected user interface display on the projection surface.

30. The method of claim 16 wherein the image data is captured using at least one depth camera.