Gesture-controlled augmented reality experience using a mobile communications device

• US 9,983,687 B1
• Filed: 01/05/2018
• Issued: 05/29/2018
• Est. Priority Date: 01/06/2017
• Status: Active Grant

• Current Assignee: Lumini Corporation
• Original Assignee: Adtile Technologies Inc.
• Inventors: Forsblom, Nils, Metti, Maximilian, Scandaliato, Angelo
• Primary Examiner(s): Dicke, Chad
• Application Number:
  US15/863,775
• Time in Patent Office: 144 Days
• Field of Search: None
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

A method for producing a gesture-controlled augmented reality experience using a first mobile communications device includes receiving a motion sensor input on a motion sensor input modality of the first mobile communications device, calculating a trajectory of a camera of the first mobile communications device in response to the received motion sensor input, receiving a visual input captured by the camera of the first mobile communications device, translating a gesture of a user into a set of quantified values based on the received visual input and the calculated trajectory of the camera, and controlling an augmented reality object within a three-dimensional virtual environment in response to a substantial match between the set of quantified values and a set of predefined values.

Claims

1. A method for producing a gesture-controlled augmented reality experience using a first mobile communications device, the method comprising:

receiving a motion sensor input on a motion sensor input modality of the first mobile communications device from a user;

calculating a trajectory of a camera of the first mobile communications device in response to the received motion sensor input;

receiving one or more images of a person other than the user captured by the camera of the first mobile communications device;

translating a gesture of the person other than the user into a set of quantified values based on the received one or more images and the calculated trajectory of the camera, the set of quantified values being calculated by a removal of the calculated trajectory from the received one or more images of the person other than the user; and

controlling an augmented reality object within a three-dimensional virtual environment in response to a substantial match between the set of quantified values and a set of predefined values.

2. The method of claim 1, further comprising:

displaying the augmented reality object on the first mobile communications device.

3. The method of claim 2, wherein said displaying the augmented reality object includes displaying a movable-window view of the three-dimensional virtual environment on the first mobile communications device.

4. The method of claim 1, further comprising:

outputting, on the first mobile communications device, at least one of visual, auditory, and haptic feedback in response to a substantial match between the set of quantified values and a set of predefined values.

5. The method of claim 1, further comprising:

receiving an auditory input captured by a microphone of the first mobile communications device;

wherein said controlling includes controlling the augmented reality object within the three-dimensional virtual environment in response to the received auditory input.

6. The method of claim 5, further comprising:

detecting a signature of music in response to the received auditory input;

wherein said controlling includes controlling the augmented reality object within the three-dimensional virtual environment in response to the detected signature.

7. The method of claim 1, further comprising:

transmitting object control data to a second mobile communications device held by the person other than the user and communicatively coupled to the first mobile communications device, the object control data defining one or more movements of the augmented reality object within the three-dimensional virtual environment in response to said controlling.

8. The method of claim 7, further comprising:

receiving a set of secondary quantified values from the second mobile communications device;

wherein said controlling includes controlling the augmented reality object within the three-dimensional virtual environment in response to a substantial match between the set of secondary quantified values and the set of predefined values.

9. The method of claim 8, wherein the set of secondary quantified values is derived from a motion sensor input on a motion sensor input modality of the second mobile communications device.

10. The method of claim 8, wherein the set of secondary quantified values is derived from a one or more images captured by a camera of the second mobile communications device.

11. The method of claim 8, wherein the set of secondary quantified values is derived from an auditory input captured by a microphone of the second mobile communications device.

12. The method of claim 1, further comprising:

transmitting feedback data to a second mobile communications device held by the person other than the user and communicatively coupled to the first mobile communications device, the feedback data defining at least one of visual, auditory, and haptic feedback to be output by the second mobile communications device in response to a substantial match between the set of quantified values and a set of predefined values.

13. The method of claim 1, wherein the motion sensor input modality of the first mobile communications device includes at least one of an accelerometer, a gyroscope, and a magnetometer integrated into the first mobile communications device.

14. A system comprising a non-transitory program storage medium readable by a first mobile communications device, the medium tangibly embodying one or more programs of instructions executable by the device to perform operations for producing a gesture-controlled augmented reality experience, the operations comprising:

receiving a motion sensor input on a motion sensor input modality of the first mobile communications device from a user;

calculating a trajectory of a camera of the first mobile communications device in response to the received motion sensor input;

receiving one or more images of a person other than the user captured by the camera of the first mobile communications device;

translating a gesture of the person other than the user into a set of quantified values based on the received one or more images and the calculated trajectory of the camera, the set of quantified values being calculated by a removal of the calculated trajectory from the received one or more images of the person other than the user; and

controlling an augmented reality object within a three-dimensional virtual environment in response to a substantial match between the set of quantified values and a set of predefined values.

15. The system of claim 14, further comprising:

the first mobile communications device;

wherein the first mobile communications device includes a processor or programmable circuitry for executing the one or more programs of instructions.

16. The system of claim 15, further comprising:

a second mobile communications device held by the person other than the user and communicatively coupled to the first mobile communications device;

wherein the operations further comprise transmitting object control data to the second mobile communications device, the object control data defining one or more movements of the augmented reality object within the three-dimensional virtual environment in response to said controlling.

17. The system of claim 16, wherein:

the second mobile communications device transmits a set of secondary quantified values to the first mobile communications device; and

said controlling includes controlling the augmented reality object within the three-dimensional virtual environment in response to a substantial match between the set of secondary quantified values and the set of predefined values.

18. The system of claim 17, wherein the second mobile communications device derives the set of secondary quantified values from a motion sensor input on a motion sensor input modality of the second mobile communications device, one or more images captured by a camera of the second mobile communications device, and/or an auditory input captured by a microphone of the second mobile communications device.

19. A mobile communications device operable to produce a gesture-controlled augmented reality experience, the mobile communications device comprising:

a motion sensor for receiving a motion sensor input from a user;

a camera for capturing a one or more images of a person other than the user; and

a processor for calculating a trajectory of the camera in response to the received motion sensor input, translating a gesture of person other than the user into a set of quantified values based on the received one or more images and the calculated trajectory of the camera, the set of quantified values being calculated by a removal of the calculated trajectory from the received one or more images of the person other than the user, and controlling an augmented reality object within a three-dimensional virtual environment in response to a substantial match between the set of quantified values and a set of predefined values.

20. The mobile communications device of claim 19, further comprising:

a display;

wherein the processor controls the display to display the augmented reality object within a movable-window view of the three-dimensional virtual environment.