Electronic apparatus having a sensing unit to input a user command and a method thereof

• US 10,423,284 B2
• Filed: 01/19/2018
• Issued: 09/24/2019
• Est. Priority Date: 12/26/2014
• Status: Active Grant

• Current Assignee: Seungman Kim
• Original Assignee: Seungman Kim
• Inventors: Kim, Seungman
• Primary Examiner(s): Sheng, Tom V
• Application Number:
  US15/875,367
• Publication Number:
  US 20180239494A1
• Time in Patent Office: 613 Days
• CPC Class Codes: G02B 13/00 Optical objectives specially designed for the purposes specified below with variable magnification in general G02B15/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

An electronic apparatus includes a housing, a display unit installed in the housing and having a display area to display an image or a menu icon, a sensing unit installed in the housing adjacent to the display unit, having a virtual area to sense an object which is disposed over the display area of the display unit, and having a first camera to photograph and display the virtual area in a photographing mode and a second camera to extract the object from the virtual area and display a sensed image of the extracted object in a sensing mode, and a controller to analyze status of the object according to the sensed image of the object, and to determine the analyzed status of the object as a user input in the sensing mode.

Claims

1. A mobile device, comprising:

a housing having a front side and a rear side;

an input and output installed in the housing, and comprising a panel on the front side of the housing to receive a user input and to display a photographing image or a menu;

a sensing unit installed on the front side of the housing, and having a first camera and a second camera respectively having a first virtual area and a second virtual area which correspond to a display area of the panel; and

a controller installed in the housing, configured to perform a photographing mode to control the first camera to photograph the first virtual area corresponding to the display area of the panel to generate a first photographing image, and to control the display unit to display the first photographing image in the display area of the panel, and configured to perform a sensing mode to control the second camera to photograph the second virtual area including an object disposed to face the display area of the panel to generate a second photographing image representing the object, to determine a state of the object of the second photographing image, and to perform a function of the mobile device according to the menu and the state of the object as an input to the menu,

wherein the second virtual area includes a plurality of sub-objects, one of which has a higher priority than the other one, and the controller is configured to determine a state of the one sub-object with the higher priority as the input to the menu.

2. The mobile device of claim 1, further comprising:

a memory to store information,

wherein the controller is configured to compare the information and the state of the object and to perform the function of the mobile device according to the menu and the comparison.

3. The mobile device of claim 1, wherein the controller is configured to analyze the object of the second photographing image with respect to the display area of the panel to sense the state of the object, and to perform the function of the mobile device according to the menu and the analysis.

4. The mobile device of claim 1, wherein the controller is configured to modify the object of the second photographing image with respect to the display area of the panel to sense the state of the object, and to perform the function of the mobile device according to the menu and the modification.

5. The mobile device of claim 1, wherein the second camera photographs the object of the second virtual area when the menu is displayed in the display area of the panel.

6. The mobile device of claim 1, wherein the second camera generates the second photographing image corresponding to the second virtual area, and the controller extracts the object from the second photographing image to sense the state of the object and performs the function of the mobile device according to the menu and the extraction.

7. The mobile device of claim 1, wherein the controller is configured to display an image corresponding to at least one of the state of the object and the state of the one sub-object over the menu of the display area of the panel.

8. The mobile device of claim 1, wherein the controller is configured to control the input and output unit to generate an output of the performed function of the mobile device, and the output comprises an image output through the display area of the panel, an audio output through an audio element, or a data output through an interface to an external device.

9. The mobile device of claim 1, further comprising:

an interface configured to communicate with an external device to transmit or receive data to perform the function of the mobile device,

wherein the controller is configured to perform the function of the mobile device according to the menu of the input and output unit, the state of the object as the user input to the menu, and the data of the external device.

10. The mobile device of claim 1, wherein:

the controller selects the one sub-object with the higher priority as the object corresponding to the menu; and

the controller performs the function according to at least one of the state of the object and the state of the one sub-object with the higher priority.

11. A method of a mobile device, the method comprising:

providing a housing having a front side and a rear side, an input and output installed in the housing and comprising a panel on the front side of the housing to receive a user input and to display a photographing image or a menu, and a sensing unit installed on the front side of the housing, and having a first camera and a second camera each respectively having a first virtual area and a second virtual area which correspond to a display area of the panel of the input and output unit;

using a controller processor installed in the housing, configured to perform a photographing mode to control the first camera to photograph the first virtual area corresponding to the display area of the panel to generate a first photographing image, and to control the display unit to display the first photographing image in the display area of the panel, and configured to perform a sensing mode to control the second camera to photograph the second virtual area including an object disposed to face the display area of the panel to generate a second photographing image representing the object, to determine a state of the object of the second photographing image, and to perform a function of the mobile device according to the menu and the state of the object as an input to the menu,

wherein the second virtual area includes a plurality of sub-objects, one of which has a higher priority than the other one, and the method further comprises determining a state of the one sub-object with the higher priority as the input to the menu.

12. The method of claim 11, wherein:

the mobile device further comprises a memory to store information; and

the method further comprises comparing the information and the state of the object and to perform the function of the mobile device according to the menu and the comparison.

13. The method of claim 11, wherein the method further comprises analyzing the object of the second photographing image to sense the state of the object, and performing the function of the mobile device according to the menu and the analysis.

14. The method of claim 11, wherein the method further comprises modifying the object of the second photographing image with respect to the display area of the panel to sense the state of the object, and performing the function of the mobile device according to the menu and the modification.

15. The method of claim 11, wherein the controlling the second camera comprises controlling the second camera to photograph the object of the second virtual area during displaying the menu in the display area of the panel.

16. The method of claim 11, further comprising:

displaying an image corresponding to at least one of the state of the object and the state of the one sub-object over the menu of the display area of the panel.

17. The method of claim 11, further comprising:

controlling the input and output unit to generate an output of the performed function of the mobile device,

wherein the output comprises an image output through the display area of the panel, an audio output through an audio element, or a data output through an interface to an external device.

18. The method of claim 11, further comprising:

an interface configured to communicate with an external device to transmit or receive data to perform a function of the mobile device,

wherein the controller is configured to perform the function of the mobile device according to the menu, the state of the object as the user input to the menu, and the data of the external device.

19. The method of claim 11, wherein:

the method further comprises selecting the one sub-object with the higher priority as the object corresponding to the menu; and

the method further comprises performing the function according to at least one of the state of the object and the state of the one sub-object with the higher priority.

20. A non-transitory computer-readable medium containing computer-readable codes as a program to perform the method of claim 11.