Re-anchorable virtual panel in three-dimensional space

• US 9,529,446 B2
• Filed: 07/23/2015
• Issued: 12/27/2016
• Est. Priority Date: 12/25/2014
• Status: Active Grant

• Current Assignee: National Taiwan University (Government of The Republic of China)
• Original Assignee: National Taiwan University (Government of The Republic of China)
• Inventors: Hung, Yi-Ping, Lin, Shih-Yao
• Primary Examiner(s): Tzeng, Fred
• Application Number:
  US14/807,790
• Publication Number:
  US 20160187991A1
• Time in Patent Office: 523 Days
• Field of Search: 345/156
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G09G 5/18 Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12

Abstract

The present invention provides a re-anchorable virtual panel in a three-dimensional space, including a detection module, a recognition module and a display module. The detection module includes a finger position detection submodule and a hand feature extraction submodule. The finger position detection submodule extracts the position and direction of a user'"'"'s fingers, and transmits the estimated information of the finger'"'"'s position to the recognition module. The hand feature extraction submodule extracts the user'"'"'s hand features from the input image sequence. After extracting the hand features, the hand feature extraction submodule transmits the information to the recognition module. The recognition module recognizes the position and direction of the user'"'"'s fingers and hands, and determines the virtual panel'"'"'s location and direction through the information. The recognition module recognizes hand gestures for controlling the virtual panel. The display module provides virtual feedback when the virtual panel is operated.

Claims

1. A re-anchorable virtual panel in a three-dimensional space, comprising:

a detection module, detecting an image sequence, analyzing a feature information of a user in the image sequence, and generating a feature analysis information of the user according to the feature information, and transmitting the feature analysis information;

a recognition module, receiving the feature analysis information, recognizing whether the feature analysis information conforms to a setting virtual device feature and whether the setting virtual device feature lasts longer than a predetermined time or not, and transmitting the feature analysis information; and

a display module, receiving the feature analysis information, and displaying a virtual device for the user according to the feature analysis information;

wherein the virtual device is continuously displayed until the display module receives another feature analysis information, and the virtual device is re-displayed for the user according to the other feature analysis information.

2. The re-anchorable virtual panel as claimed in claim 1, wherein the detection module includes a finger position detection submodule and a hand feature extraction submodule, the finger position detection submodule separates a hand position of the user from the image sequence according to an information on depth, the finger position detection submodule further calculates a three-dimensional finger position information of the user and generates the feature analysis information, and the hand feature extraction submodule analyzes the feature information of the user in the image sequence and generates the feature analysis information.

3. The re-anchorable virtual panel as claimed in claim 2, wherein the hand feature extraction submodule is a RGB-D camera.

4. The re-anchorable virtual panel as claimed in claim 2, wherein the finger position detection submodule is a finger position extraction sensor (Leap Motion).

5. The re-anchorable virtual panel as claimed in claim 1, wherein the feature analysis information is one of a finger position information and a hand feature information of the user.

6. The re-anchorable virtual panel as claimed in claim 1, wherein the feature analysis information and the setting virtual device feature are an L-shaped hand gesture.

7. The re-anchorable virtual panel as claimed in claim 6, wherein the L shape hand gesture is formed by a thumb part and a forefinger part.

8. The re-anchorable virtual panel as claimed in claim 7, wherein the display module sets a generating position of the virtual device according to a position of a part between the thumb part and the forefinger part, determines a direction and an angle of the virtual device according to the positions of the thumb part and the forefinger part, and generates the virtual device.

9. The re-anchorable virtual panel as claimed in claim 1, wherein the display module determines the position and direction of the virtual device according to a gaze direction of the user and the position of the feature analysis information.

10. The re-anchorable virtual panel as claimed in claim 1, wherein the feature analysis information and the setting virtual device feature are the shape and color of the user'"'"'s wrist.

11. The re-anchorable virtual panel as claimed in claim 1, wherein the feature analysis information and the setting virtual device feature are a mark of a watch worn by the user.

12. The re-anchorable virtual panel as claimed in claim 11, wherein the mark of the watch is a QR code.

13. The re-anchorable virtual panel as claimed in claim 1, wherein the re-anchorable virtual panel is configured in a sensor.

14. The re-anchorable virtual panel as claimed in claim 13, wherein the sensor is configured in one of a head wear device, a home appliance device and a wall of an interior space.

15. A method of generating a re-anchorable virtual panel device, comprising the steps of:

detecting an image sequence via a detection module, analyzing a feature information of a user in the image sequence, generating a feature analysis information of the user according to the feature information, and transmitting the feature analysis information to a recognition module;

recognizing whether the feature analysis information is a setting virtual device feature by the recognition module;

recognizing whether the setting virtual device feature lasts longer than a predetermined time by the recognition module after recognizing that the feature analysis information is the setting virtual device feature;

transmitting the feature analysis information to a display module by the recognition module after recognizing that the setting virtual device feature lasts longer than the predetermined time; and

displaying a virtual device for the user by the display module according to the feature analysis information,

wherein the virtual device is continuously displayed until the display module receives another feature analysis information, and the virtual device is re-displayed according to the other feature analysis information.

16. The method of generating a re-anchorable virtual panel device as claimed in claim 15, wherein the feature analysis information is one of a finger position information and a hand feature information of the user.

17. The method of generating a re-anchorable virtual panel device as claimed in claim 15, wherein the feature analysis information and the setting virtual device feature are an L-shaped hand gesture.

18. The method of generating a re-anchorable virtual panel device as claimed in claim 17, wherein the L-shaped hand gesture is formed by a thumb part and a forefinger part.

19. The method of generating a re-anchorable virtual panel device as claimed in claim 15, wherein the feature analysis information and the setting virtual device feature are the user'"'"'s wrist shape and color.

20. The method of generating a re-anchorable virtual panel device as claimed in claim 15, wherein the feature analysis information and the setting virtual device feature are a mark of a watch worn by the user.

21. The method of generating a re-anchorable virtual panel device as claimed in claim 20, wherein the mark of the watch is a QR code.

22. The method of generating a re-anchorable virtual panel device as claimed in claim 15, further comprising an operation method of the re-anchorable virtual panel device, comprising the steps of:

putting down the hand that sets the virtual device and raises another hand'"'"'s forefinger after the virtual device is generated;

continuously tracing a fingertip of the other hand'"'"'s forefinger by the detection module and projecting a three-dimensional position of the fingertip of the forefinger on the virtual device in a two-dimensional space when the other hand'"'"'s forefinger is detected by the detection module; and

when the detection module detects that the forefinger performs a forward pressing action on the virtual device, the forward pressing action is being regarded as a click gesture.

23. The method of generating a re-anchorable virtual panel device as claimed in claim 22, wherein the forward pressing action is an accelerated motion with an acceleration greater than a threshold.