Three-dimensional interactive device and operation method thereof

• US 8,923,562 B2
• Filed: 05/16/2013
• Issued: 12/30/2014
• Est. Priority Date: 12/24/2012
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Hung-Wei, Wang, Hau-Wei, Dong, Shu-Ping
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US13/895,382
• Publication Number:
  US 20140177909A1
• Time in Patent Office: 593 Days
• Field of Search: 382/100, 382/103, 382/107, 382/181, 382/190, 382/195, 382/203, 348/135, 348169-172, 715/863
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass

Abstract

A three-dimensional (3D) interactive device and an operation method thereof are provided. The 3D interactive device includes a projection unit, an image capturing unit, and an image processing unit. The projection unit projects an interactive pattern to a surface of a body, so that a user performs an interactive trigger operation on the interactive pattern by a gesture. The image capturing unit captures a depth image within an image capturing range. The image processing unit receives the depth image and determines whether the depth image includes a hand region of the user. If yes, the image processing unit performs hand geometric recognition on the hand region to obtain gesture interactive semantics. According to the gesture interactive semantics, the image processing unit controls the projection unit and the image capturing unit. Accordingly, the disclosure provides a portable, contact-free 3D interactive device.

Claims

1. A three-dimensional interactive device comprising:

a projection unit projecting a first interactive pattern to a surface of a body, such that a user performs an interactive trigger operation on the first interactive pattern by a gesture, wherein the first interactive pattern is projected within a projection range;

an image capturing unit capturing a depth image within an image capturing range, wherein the image capturing range covers the projection range; and

an image processing unit connected to the projection unit and the image capturing unit, the image processing unit receiving the depth image and determining whether the depth image comprises a hand region of the user, if yes, the image processing unit performs hand geometric recognition on the hand region to obtain a gesture interactive semantics, and the image processing unit controls the projection unit and the image capturing unit according to the gesture interactive semantics.

2. The three-dimensional interactive device as recited in claim 1, wherein the projection unit projects a border marker symbol and a center marker symbol on at least one border point and a center of the projection range, respectively, so as to form a calibration pattern for calibration.

3. The three-dimensional interactive device as recited in claim 2, further comprising:

a coordinate calibration unit coupled to the projection unit and the image capturing unit, the coordinate calibration unit receiving a three-primary-color image captured by the image capturing unit, wherein the three-primary-color image covers the calibration pattern, the coordinate calibration unit analyzes a coordinate of the border marker symbol and a coordinate of the center marker symbol in the three-primary-color image through conducting an image comparison method to obtain a coordinate transformation between a projection coordinate of the projection unit and an image capturing coordinate of the image capturing unit.

4. The three-dimensional interactive device as recited in claim 2, wherein the border marker symbol projected by the projection unit is a pattern of a circular point.

5. The three-dimensional interactive device as recited in claim 2, wherein the center marker symbol projected by the projection unit is a hand-shaped pattern.

6. The three-dimensional interactive device as recited in claim 1, wherein the image processing unit further comprises:

a gesture recognition unit analyzing the depth image through conducting a histogram statistical method and recognizing the hand region of the user through a convex hull and a convex deficiency of the depth image.

7. The three-dimensional interactive device as recited in claim 6, wherein the gesture recognition unit further recognizes a location of a centroid point of the hand region of the user and outputs a coordinate of the centroid point corresponding to the location of the centroid point to the projection unit.

8. The three-dimensional interactive device as recited in claim 7, wherein the projection unit correspondingly adjusts a projection location of a second interactive pattern and a dimension of the second interactive pattern according to the coordinate of the centroid point.

9. The three-dimensional interactive device as recited in claim 6, wherein the gesture recognition unit further recognizes a depth position of at least one fingertip of the hand region of the user and tracks a motion trajectory of the at least one fingertip, and the gesture recognition unit recognizes the gesture interactive semantics represented by the motion trajectory through comparing a sample in a gesture interactive semantic database with the depth position of the at least one fingertip of the hand region of the user.

10. The three-dimensional interactive device as recited in claim 9, wherein the gesture recognition unit transmits the gesture interactive semantics and at least one three-dimensional interactive parameter obtained by analyzing the depth position of the at least one fingertip and the motion trajectory to the projection unit and the image capturing unit, so as to control the projection unit to project a second interactive pattern corresponding to the gesture interactive semantics and control the image capturing unit to continuously capture the depth image comprising the gesture.

11. The three-dimensional interactive device as recited in claim 1, wherein the gesture interactive semantics at least comprise a tap-to-select operation, a slide-to-scroll operation, a spin-to-scroll operation, a flick-to-nudge operation, a spread-to-enlarge operation, and a pinch-to-shrink operation.

12. The three-dimensional interactive device as recited in claim 1, wherein the projection unit is a pico-projector.

13. The three-dimensional interactive device as recited in claim 1, wherein the image capturing unit is a time-of-flight camera, a stereo vision depth camera, a laser speckle camera, or a laser tracking camera.

14. An operation method of a three-dimensional interactive device, the three-dimensional interactive device comprising a projection unit and an image capturing unit, the operation method comprising:

performing a coordinate calibration process on a projection coordinate of the projection unit and an image capturing coordinate of the image capturing unit;

projecting a first interactive pattern to a surface of a body by the projection unit, such that a user performs an interactive trigger operation on the first interactive pattern by a gesture, wherein the first interactive pattern is projected within a projection range;

capturing a depth image within an image capturing range by the image capturing unit, wherein the image capturing range covers the projection range;

determining whether the depth image comprises a hand region of the user;

if yes, performing a hand geometric recognition on the hand region to obtain a gesture interactive semantics; and

controlling the projection unit and the image capturing unit according to the gesture interactive semantics.

15. The operation method as recited in claim 14, wherein the coordinate calibration process comprises:

respectively projecting a border marker symbol and a center marker symbol on at least one border point and a center of the projection range by the projection unit, so as to form a calibration pattern;

capturing a three-primary-color image of the calibration pattern by the image capturing unit; and

analyzing a coordinate of the border marker symbol and a coordinate of the center marker symbol in the three-primary-color image through conducting an image comparison method to obtain a coordinate transformation between the projection coordinate and the image capturing coordinate.

16. The operation method as recited in claim 15, wherein the border marker symbol is a pattern of a circular point.

17. The operation method as recited in claim 15, wherein the center marker symbol is a hand-shaped pattern.

18. The operation method as recited in claim 14, wherein the step of determining whether the depth image comprises the hand region of the user to obtain the gesture interactive semantics comprises:

analyzing the depth image through conducting a histogram statistical method;

recognizing the hand region of the user through a depth value of the depth image as well as a convex hull and a convex deficiency of the depth image; and

recognizing a location of a depth of at least one fingertip of the hand region of the user, tracking a motion trajectory of the at least one fingertip, and recognizing the gesture interactive semantics represented by the motion trajectory through comparing a sample in a gesture interactive semantic database with the depth position of the at least one fingertip of the hand region of the user.

19. The operation method as recited in claim 18, after obtaining the gesture interactive semantics, the operation method further comprising:

transmitting at least one three-dimensional interactive parameter obtained by analyzing the depth position of the at least one fingertip and the motion trajectory to the projection unit and the image capturing unit, so as to control the projection unit to project a second interactive pattern corresponding to the gesture interactive semantics and control the image capturing unit to continuously capture the depth image comprising the gesture.

20. The operation method as recited in claim 19, further comprising:

recognizing a location of a centroid point of the hand region of the user and outputting a coordinate of the centroid point corresponding to the location of the centroid point to the projection unit; and

according to the coordinate of the centroid point, correspondingly adjusting a projection location of the second interactive pattern and a dimension of the second interactive pattern by the projection unit.

21. The operation method as recited in claim 14, wherein the gesture interactive semantics at least comprise a tap-to-select operation, a slide-to-scroll operation, a spin-to-scroll operation, a flick-to-nudge operation, a spread-to-enlarge operation, and a pinch-to-shrink operation.