APPARATUS, METHOD AND RECORDING MEDIUM FOR CONTROLLING USER INTERFACE USING INPUT IMAGE

• US 20150070272A1
• Filed: 09/05/2014
• Published: 03/12/2015
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: KIM, Jin-Yong, KANG, Ji-Young, KIM, Dae-Sung, LIM, Seung-Kyung, JEON, Jin-Young, KIM, Seok-Tae, LEE, Bo-Young
• Granted Patent: US 9,898,090 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions

Abstract

A method of controlling a user interface using an input image is provided. The method includes storing operation executing information of each of one or more gesture forms according to each of a plurality of functions, detecting a gesture form from the input image, and identifying the operation executing information mapped on the detected gesture form to execute an operation according to a function which is currently operated.

Claims

1. A method of controlling a user interface using an input image, the method comprising:

storing operation executing information of each of one or more gesture forms according to each of a plurality of functions;

detecting a gesture form from the input image; and

identifying the operation executing information mapped on the detected gesture form according to a function which currently operates and executing the operation of the operation executing information.

2. The method of controlling the user interface as claimed in claim 1, wherein the detecting of the gesture form from the input image comprises:

extracting a contour from the input image;

determining whether the extracted contour is matched with one of the one or more gesture forms; and

recognizing a matched form as the gesture form if it is determined that the extracted contour is matched with one of the one or more gesture forms.

3. The method of controlling the user interface as claimed in claim 1, wherein when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed, a predetermined metadata corresponding to the detected gesture form is tagged to the input image.

4. The method of controlling the user interface as claimed in claim 1, wherein when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed, a predetermined visible effect corresponding to the detected gesture form is displayed on the input image.

5. The method of controlling the user interface as claimed in claim 4, further comprising:

detecting at least one portion on a face in the input image by applying a face recognizing function to the input image,

wherein in the displaying of the predetermined visible effect corresponding to the detected gesture form on the input image, the predetermined visible effect is displayed on the detected face portion corresponding to the predetermined face portion in correspondence to the detected gesture form.

6. The method of controlling the user interface as claimed in claim 4, further comprising:

changing at least one of a type, a color and a shape of the visible effect according to a recognition of a predetermined voice instruction.

7. The method of controlling the user interface as claimed in claim 1, wherein the operating function corresponds to a camera function, and if the detected gesture form is a predetermined pointing gesture form, the identifying of the operation executing information mapped on the detected gesture form so as to execute the operation of the operation executing information comprises:

detecting a predetermined pointing position of the predetermined pointing gesture form from the input image; and

executing an auto focusing operation of the camera function by a reference of the detected pointing position.

8. The method of controlling the user interface as claimed in claim 1, wherein when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed, a predetermined texture corresponding to the detected gesture form is applied to the input image and displayed on a preview screen.

9. The method of controlling the user interface as claimed in claim 1, wherein if the detected gesture form is a predetermined pointing gesture form, the identifying the operation execution information mapped on the detected gesture form and executing the operation of the operation execution information comprises:

detecting at least one object located in a pointing direction of the pointing gesture form from the input image; and

applying a predetermined texture corresponding to the detected gesture form to the detected object and displaying it on a preview screen.

10. The method of controlling the user interface as claimed in claim 1, wherein the operating function corresponds to an augmented reality function, and when operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed, a predetermined function mapped on the detected gesture form is executed in the augmented reality image displayed on the preview screen according to an execution of the augmented reality function.

11. An apparatus for controlling a user interface using an input image, the apparatus comprising:

an image receiving unit configured to receive an input of an image;

a storage unit; and

a controller configured to enable the storage unit to store operation executing information of each of one or more gesture forms according to each of a plurality of functions, detect a gesture form from an image input through the image receiving unit, and identify and execute the operation executing information mapped on the detected gesture form according to a function which is currently operated.

12. The apparatus for controlling the user interface as claimed in claim 11, wherein the controller extracts a contour from the input image when the gesture form is detected from the input image, determines whether the extracted contour has a form matched with the gesture form, and determines the matched form as the gesture form and detects the gesture form if it is determined that the extracted contour has the form matched with the one or more gesture forms.

13. The apparatus for controlling the user interface as claimed in claim 11, wherein the controller tags a predetermined metadata corresponding to the detected gesture form to the input image when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed.

14. The apparatus for controlling the user interface as claimed in claim 11, further comprising:

a displaying unit,

wherein the controller displays the input image and a predetermined visible effect corresponding to the detected gesture form of the input image on the displaying unit, when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed.

15. The apparatus for controlling the user interface as claimed in claim 14, wherein the controller applies a face recognition function to the input image so as to detect at least one portion of a face from the input image, and displays the predetermined visible effect on the detected face portion corresponding to the predetermined face portion in correspondence to the detected gesture form.

16. The apparatus for controlling the user interface as claimed in claim 14, further comprising:

a microphone,

wherein the controller changes at least one of a type, a color and a shape of the visible effect according to recognition of a predetermined voice instruction input through the microphone.

17. The apparatus for controlling the user interface as claimed in claim 11, wherein the operating function corresponds to a camera function, and the controller detects a predetermined pointing position of a predetermined pointing gesture form from the input image if the detected gesture form is the predetermined pointing gesture form, and executes an auto focusing operation of the camera function by a reference of the detected pointing position.

18. The apparatus for controlling the user interface as claimed in claim 11, further comprising:

a displaying unit, wherein a predetermined texture corresponding to the detected gesture form is applied to the input image displayed on a preview screen of the displaying unit, when the operation executing information mapped on the detected gesture form is identified and the operation of the operation executing information is executed.

19. The apparatus for controlling the user interface as claimed in claim 11, further comprising:

a displaying unit, wherein the controller detects at least one object located in a pointing direction of the pointing gesture form from the input image displayed on a preview screen of the displaying unit if the detected gesture form is a predetermined pointing gesture form, and applies a predetermined texture corresponding to the detected gesture form to the detected object.

20. The apparatus for controlling the user interface as claimed in claim 11, further comprising:

a displaying unit, wherein the operating function is an augmented reality function, and the controller executes a predetermined function mapped on the detected gesture form in the augmented reality image displayed on a preview screen of the displaying unit according to an execution of the augmented reality function.

21. A recording medium for controlling a user interface using an input image, the recording medium comprising:

an image receiving unit configured to receive an input of an image;

a storage unit; and

programs which operate a controller for enabling the storage unit to store operation executing information of each of one or more gesture forms according to each of a plurality of functions, detecting a gesture form from an image input through the image receiving unit, and identifying the operation executing information mapped on the detected gesture form according to a function which is currently operated and executing the operation of the operation executing information.

22. A method of modifying an input image, the method comprising:

receiving a gesture form via an image receiving unit;

matching the gesture form to at least one of a plurality of stored gesture forms; and

applying an object associated with the matched gesture form to the input image and displaying the input image with the applied object.

23. The method of modifying the input image as in claim 22, wherein the applied object is on one of a visible object and a non-visible object associate with the matched gesture form.

24. The method of modifying the input image as in claim 23, wherein the non-visible object associated with the matched gesture form is metadata.

25. The method of modifying the input image as in claim 24, wherein the metadata defines a focus point for an image capture of a camera function.

26. The method of modifying the input image as in claim 23, wherein the visible object associated with the matched gesture form is an effect rendered from a shape of the matched gesture.

27. The method of modifying the input image as in claim 23, wherein the visible object associated with the matched gesture form is further associated with a feature of the input image, and the application of the visible object to the input image is performed in a manner that integrates the visible object to the feature of the input image.

28. The method of modifying the input image as in claim 27, wherein the visible object is a pair of glasses and the feature of the input image is a user'"'"'s face.

29. The method of modifying the input image as in claim 27, wherein the visible object is a texture and the feature of the input image is an object selectively chosen from a plurality of objects in the input image.

30. The method of modifying the input image as in claim 23, wherein when the input image is a sequence of input images of a video call, the visible object is a message text applied to each input image in the sequence of input images.