Apparatus, method and recording medium for controlling user interface using input image

• US 10,579,152 B2
• Filed: 02/14/2018
• Issued: 03/03/2020
• Est. Priority Date: 09/10/2013
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Kim, Jin-Yong, Kang, Ji-Young, Kim, Dae-Sung, Kim, Seok-Tae, Lee, Bo-Young, Lim, Seung-Kyung, Jeon, Jin-Young
• Primary Examiner(s): Eurice, Michael J
• Application Number:
  US15/896,751
• Publication Number:
  US 20180173321A1
• Time in Patent Office: 748 Days
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/60 Control of cameras or camera modules
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/675 comprising setting of focusing regions

Abstract

A method of controlling a user interface using an input image is provided. The method includes storing operation executing information of each of one or more gesture forms according to each of a plurality of functions, detecting a gesture form from the input image, and identifying the operation executing information mapped on the detected gesture form to execute an operation according to a function which is currently operated.

Claims

1. A mobile device comprising:

a camera;

a display; and

a controller configured to;

display a preview image obtained by the camera,

detect a gesture form in the preview image,

identify an object from the preview image based on the detected gesture,

identify a visible effect according to the identified object, and

display the preview image with the identified visible effect.

2. The mobile device as claimed in claim 1, wherein the controller is further configured to:

extract a contour from the preview image,

identify whether the extracted contour is matched with one of one or more predetermined gesture forms, and

identify the matched form as the gesture form if it is identified that the extracted contour is matched with one of the one or more predetermined gesture forms.

3. The mobile device as claimed in claim 1, wherein the controller is further configured to tag a predetermined metadata corresponding to the detected gesture form to the preview image.

4. The mobile device as claimed in claim 1, further comprising:

a microphone,

wherein the controller is further configured to change at least one of a type, a color and a shape of the visible effect according to recognition of a predetermined voice instruction input through the microphone.

5. The mobile device as claimed in claim 1,

wherein the detected gesture form is a pointing gesture form, and

wherein the controller is further configured to:

execute an auto focusing operation of a camera function by a reference of a detected pointing position of the detected gesture form.

6. The mobile device as claimed in claim 1,

wherein the controller is further configured to apply a predetermined texture to the identified object in the preview image.

7. A method of controlling a mobile device, the method comprising:

displaying a preview image obtained by a camera of the mobile device;

detecting a gesture form in the it preview image;

identifying an object from the preview image based on the detected gesture;

identifying a visible effect according to the identified object; and

displaying the preview image with the identified visible effect.

8. The method of controlling the mobile device as in claim 7, further comprising:

extracting a contour from the preview image;

identifying whether the extracted contour is matched with one of one or more predetermined gesture forms; and

identifying the matched form as the gesture form if it is identified that the extracted contour is matched with one of the one or more predetermined gesture forms.

9. The method of controlling the mobile device as in claim 7, further comprising:

tagging a predetermined metadata corresponding to the detected gesture form to the preview image.

10. The method of controlling the mobile device as in claim 7, further comprising:

changing at least one of a type, a color and a shape of the visible effect according to recognition of a predetermined voice instruction input through a microphone of the mobile device.

11. The method of controlling the mobile device as in claim 7, wherein the detected gesture form is a pointing gesture form, and

wherein the method further comprises:

executing an auto focusing operation of a camera function by a reference of a detected pointing position of the detected gesture form.

12. The method of controlling the mobile device as in claim 7, wherein the displaying of the preview image with the identified visible effect comprises:

applying a predetermined texture to the identified object in the preview image.

13. A recording medium having stored thereon instructions, when executed by at least one processor of a mobile device, causes the mobile device to:

display a preview image obtained by a camera of the mobile device;

detect a gesture form in the preview image;

identify an object from the preview image based on the detected gesture;

identify a visible effect according to the identified object; and

display the preview image with the identified visible effect.