HIGH EFFICIENCY INPUT APPARATUS AND METHOD FOR VIRTUAL REALITY AND AUGMENTED REALITY

• US 20190324525A1
• Filed: 09/07/2018
• Published: 10/24/2019
• Est. Priority Date: 04/19/2018
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Perdices-Gonzalez, Sergio
• Granted Patent: US 10,877,554 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0202 Constructional details or processes of manufacture of the input device
• CPC Class Codes: G06F 3/023 Arrangements for converting discrete items of information into a coded form, e.g. arrangements for interpreting keyboard generated codes as alphanumeric codes, operand codes or instruction codes
• CPC Class Codes: G06F 3/0308 comprising a plurality of distinctive and separately oriented light emitters or reflectors associated to the pointing device, e.g. remote cursor controller with distinct and separately oriented LEDs at the tip whose radiations are captured by a photo-detector associated to the screen
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03543 Mice or pucks G06F3/03541 takes precedence
• CPC Class Codes: G06F 3/03547 Touch pads, in which fingers can move on a surface
• CPC Class Codes: G06F 3/044 by capacitive means
• CPC Class Codes: G06F 3/046 by electromagnetic means
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An electronic device, method, and computer readable medium for high efficiency input in virtual reality and augmented reality are provided. The electronic device includes a (wired or wireless) transceiver, and a processor coupled to the transceiver. The processor receives, using the transceiver, information from at least one of a sensing element on an input element of a physical device or a tracking element on the physical device; provides, in a virtual environment and based on the information received from the at least one of the sensing element or the tracking element, a virtual device corresponding to the physical device; detects, based on the information received from the at least one of the sensing element or the tracking element, an interaction with the physical device; and provides, in association with the virtual device in the virtual environment, an indication of the interaction with the physical device.

Claims

1. An electronic device comprising:

a transceiver; and

a processor coupled to the transceiver, the processor configured to;

receive, using the transceiver, information from at least one of a sensing element on an input element of a physical device or a tracking element on the physical device;

provide, in a virtual environment and based on the information received from the at least one of the sensing element or the tracking element, a virtual device corresponding to the physical device;

detect, based on the information received from the at least one of the sensing element or the tracking element, an interaction with the physical device; and

provide, in association with the virtual device in the virtual environment, an indication of the interaction with the physical device.

2. The electronic device of claim 1, wherein:

the processor is further configured to determine at least one of a type or a size of the physical device based on the tracking element on the physical device, and

the virtual device is provided in the virtual environment based on the at least one of the type or the size of the physical device.

3. The electronic device of claim 1, wherein:

the interaction with the physical device is detected based on information received from the sensing element,

the interaction indicates a location of an appendage of a user in relation to the input element of the physical device, and

the indication is a marker provided in relation to a virtual input element on the virtual device corresponding to the input element of the physical device.

4. The electronic device of claim 1, wherein:

the sensing element includes at least one of an electrode sensing element, a capacitive sensing element, or an electromagnetic interference (EMI) sensing element;

when the interaction with the physical device is detected based on information received from the sensing element, the indication provides a virtual appendage in relation to a virtual input element on the virtual device; and

a virtual contact or a virtual distance between the virtual appendage and the virtual input element corresponds to a contact or a distance between an appendage of a user and the input element of the physical device.

5. The electronic device of claim 1, wherein the processor is further configured to:

provide the virtual environment via a head-mounted display (HMD);

determine, based on information received from the tracking element, at least one of a physical location or a physical orientation of the physical device relative to the HMD; and

control at least one of a virtual location or a virtual orientation of the virtual device in the virtual environment based on the at least one of the physical location or the physical orientation of the physical device.

6. The electronic device of claim 5, wherein, when the interaction with the physical device is detected based on information received from the tracking element, the indication is an adjustment of the at least one of the virtual location or the virtual orientation of the virtual device corresponding to the at least one of the physical location or the physical orientation of the physical device.

7. The electronic device of claim 5, wherein:

the tracking element includes a set of one or more transceivers on the physical device, and

at least one of the physical location or the physical orientation of the physical device relative to the HMD is determined based on the set of one or more transceivers.

8. The electronic device of claim 5, wherein:

the HMD includes a camera, and

at least one of the physical location or the physical orientation of the physical device relative to the HMD is determined based on object recognition technology applied to one or more images captured by the camera.

9. The electronic device of claim 1, wherein the processor is further configured to:

detect, based on information received from the sensing element, a gesture over the physical device from at least one of a hand or an appendage of a user;

determine a function corresponding to the gesture; and

performing the function in the virtual environment.

10. The electronic device of claim 1, wherein the processor is further configured to:

detect, based on information received from the sensing element, at least one of a physical location or a physical orientation of at least one of a hand or appendage in relation to the physical device; and

display at least one of a virtual hand or a virtual appendage in the virtual environment corresponding to the at least one of the physical location or the physical orientation of the hand or appendage.

11. A method, the method comprising:

receive information from at least one of a sensing element on an input element of a physical device or a tracking element on the physical device;

providing, in a virtual environment and based on the information received from the at least one of the sensing element or the tracking element, a virtual device corresponding to the physical device;

detecting, based on the information received from the at least one of the sensing element or the tracking element, an interaction with the physical device; and

providing, in association with the virtual device in the virtual environment, an indication of the interaction with the physical device.

12. The method of claim 11, further comprising:

determining at least one of a type or a size of the physical device based on the tracking element on the physical device,

wherein the virtual device is provided in the virtual environment based on the at least one of the type and the size of the physical device.

13. The method of claim 11, wherein:

the interaction with the physical device is detected based on information received from the sensing element,

the interaction indicates a location of an appendage of a user in relation to the input element of the physical device, and

the indication is a marker provided in relation to a virtual input element on the virtual device corresponding to the input element of the physical device.

14. The method of claim 11, wherein:

the sensing element includes at least one of an electrode sensing element, a capacitive sensing element, or an electromagnetic interference (EMI) sensing element;

when the interaction with the physical device is detected based on information received from the sensing element, the indication provides a virtual appendage in relation to a virtual input element on the virtual device; and

a virtual contact or a virtual distance between the virtual appendage and the virtual input element correspond to a contact or a distance between an appendage of a user and the input element of the physical device.

15. The method of claim 11, further comprising:

providing the virtual environment via a head-mounted display (HMD);

determining, based on information received from the tracking element, at least one of a physical location or a physical orientation of the physical device relative to the HMD; and

controlling at least one of a virtual location or a virtual orientation of the virtual device in the virtual environment based on the at least one of the physical location or the physical orientation of the physical device.

16. The method of claim 15, wherein, when the interaction with the physical device is detected based on the information received from the tracking element, the indication is an adjustment of the at least one of the virtual location or the virtual orientation of the virtual device corresponding to the at least one of the physical location or the physical orientation of on the physical device.

17. The method of claim 15, wherein:

the tracking element includes a set of one or more transceivers on the physical device, and

at least one of the physical location or the physical orientation of the physical device relative to the HMD is determined based on the set of the one or more transceivers.

18. The method of claim 15, wherein:

the HMD includes a camera, and

at least one of the physical location or the physical orientation of the physical device relative to the HMD is determined based on object recognition technology applied to one or more images captured by the camera.

19. A non-transitory computer readable medium embodying a computer program, the computer program comprising computer readable program code that when executed causes at least one processor to:

receive information from at least one of a sensing element on an input element of a physical device or a tracking element on the physical device;

provide, in a virtual environment and based on the information received from the at least one of the sensing element or the tracking element a virtual device corresponding to the physical device;

detect, based on the information received from that at least one of the sensing element or the tracking element an interaction with the physical device; and

provide, in association with the virtual device in the virtual environment, an indication of the interaction with the physical device.

20. The non-transitory computer readable medium of claim 19, wherein:

the interaction with the physical device is detected based on information from the sensing element,

the interaction indicates a location of an appendage of a user in relation to the input element of the physical device, and

the indication is a marker provided in relation to a virtual input element on the virtual device corresponding to the input element of the physical device.

21. The non-transitory computer readable medium of claim 19, wherein the processor is further configured to:

provide the virtual environment via a head-mounted display (HMD);

determine, based on information received from the tracking element, at least one of a physical location and orientation of the physical device relative to the HMD; and

control at least one of a virtual location or a virtual orientation of the virtual device in the virtual environment based on the at least one of the physical location or the physical orientation of the physical device.

22. The non-transitory computer readable medium of claim 21, wherein, when the interaction with the physical device is detected based on information received from the tracking element, the indication is an adjustment of the at least one of the virtual location or the virtual orientation of the virtual device corresponding to the at least one of the physical location or the physical orientation of the physical device.