Electronic device and method for controlling of the same

• US 10,567,567 B2
• Filed: 08/10/2018
• Issued: 02/18/2020
• Est. Priority Date: 08/10/2017
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Seo, Kyunghye, Jung, Juhyun, Lee, Younghoon
• Primary Examiner(s): Taylor, Nathan S
• Application Number:
  US16/100,996
• Publication Number:
  US 20190075200A1
• Time in Patent Office: 557 Days
• Field of Search: 455420
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1671 Special purpose buttons or auxiliary keyboards, e.g. retractable mini keypads, keypads or buttons that remain accessible at closed laptop G06F1/1666 takes precedence
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/169 the I/O peripheral being an integrated pointing device, e.g. trackball in the palm rest area, mini-joystick integrated between keyboard keys, touch pads or touch stripes G06F1/1643 takes precedence; constructional details of pointing devices G06F3/033
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/1698 the I/O peripheral being a sending/receiving arrangement to establish a cordless communication link, e.g. radio or infrared link, integrated cellular phone details of antennas disposed inside a computer H01Q1/2266
• CPC Class Codes: G06F 1/266 Arrangements to supply power to external peripherals either directly from the computer or under computer control, e.g. supply of power through the communication port, computer controlled power-strips
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: G06F 2203/0338 Fingerprint track pad, i.e. fingerprint sensor used as pointing device tracking the fingertip image
• CPC Class Codes: G06F 2203/0339 Touch strips, e.g. orthogonal touch strips to control cursor movement or scrolling; single touch strip to adjust parameter or to implement a row of soft keys
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 1/72448 with means for adapting the functionality of the device according to specific conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion telephones including GPS signal receivers H04M2250/10
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04N 21/41265 having a remote control device for bidirectional communication between the remote control device and client device unidirectional remote control device H04N21/42204
• CPC Class Codes: H04N 21/42202 : environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
• CPC Class Codes: H04N 21/42208 : Display device provided on the remote control
• CPC Class Codes: H04N 21/42224 : Touch pad or touch panel provided on the remote control
• CPC Class Codes: H04N 21/42226 : Reprogrammable remote control devices

Abstract

A mobile terminal and a method for controlling the mobile terminal are disclosed. A mobile terminal according to the present invention may figure out a user'"'"'s intention for controlling an IoT device on the basis of at least one of a grip sensor, attitude detection sensor, camera image, and context awareness information without involving a process of executing a separate application for controlling the IoT device, thereby providing a control menu for controlling the IoT device more intuitively.

Claims

1. A mobile terminal, comprising:

a body;

a touchscreen coupled to a front side of the body;

a first sensor coupled to an edge of the body and configured to sense an applied grip;

a second sensor configured to detect attitude of the body;

an input unit being located at a rear side of the body;

a wireless communication unit configured to perform wireless communication with an external electronic device;

a first camera and a second camera having a wider field of view than the first camera; and

a controller configured to;

cause the mobile terminal to enter a remote control mode, for remotely controlling the external electronic device, when an input from the first sensor indicates a grip state that satisfies a first defined condition, or when an input from the second sensor satisfies a second defined condition;

after entering the remote control mode, control the external electronic device in response to a received input;

capture a first image through the first camera that is in an Always on Camera (AOC) mode when the touchscreen is in an inactive state;

capture a second image through the second camera, when only a part of the external electronic device is present in the first image;

recognize the external electronic device in the captured second image; and

display an extracted name corresponding to the external electronic device as being overlapped on a preview image.

2. The mobile terminal of claim 1, wherein the first defined condition comprises the applied grip being any of:

gripping more than a predetermined region of a lower edge of the first sensor,

gripping more than a predetermined region of a first side surface edge, or

gripping more than a predetermined region of a second side surface edge, and

wherein the controller is further configured to;

cause the mobile terminal to enter the remote control mode further when a touch input is received at the touchscreen for a period of time while the grip state continues to satisfy the first defined condition.

3. The mobile terminal of claim 1, wherein the first defined condition comprises the applied grip being any of:

gripping more than a predetermined region of a lower edge of the first sensor located along the edge of the body,

gripping more than a predetermined region of a first side surface edge, or

gripping more than a predetermined region of a second side surface edge, and

wherein the controller is further configured to;

cause the mobile terminal to enter the remote control mode further when a touch input is received at the input unit for a period of time while the grip state continues to satisfy the first defined condition.

4. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the mobile terminal to enter the remote control mode further when an input from the second sensor indicates a change of the attitude of the body and the change in the attitude is maintained for a defined period of time.

5. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the touchscreen to display a control menu for remotely controlling the external electronic device, according to a position of a touch input received at the touchscreen.

6. The mobile terminal of claim 5, wherein the controller is further configured to:

detect a power supply state of the external electronic device, the detected power supply state being one of a plurality of different states; and

configure the control menu according to which state, of the plurality of different states, the power supply state is detected.

7. The mobile terminal of claim 6, wherein the controller is further configured to:

cause the touchscreen to display a power button when the detected power supply state indicates that power has not been provided to the external electronic device; and

cause the touchscreen to display the control menu that includes the power button when the detected power supply state indicates that power has been provided to the external electronic device.

8. The mobile terminal of claim 1, wherein the controller is further configured to:

control the external electronic device in response to an input received at the input unit.

9. The mobile terminal of claim 8, wherein the controller is further configured to:

detect a power supply state of the external electronic device, the detected power supply state being one of a plurality of different states; and

control the external device according to which state, of the plurality of different states, the power supply state is detected.

10. The mobile terminal of claim 1, wherein the controller is further configured to:

obtain context awareness information; and

control a specific external electronic device corresponding to the obtained context awareness information, while in the remote control mode.

11. The mobile terminal of claim 10, further comprising:

a third sensor unit configured to sense an environment within which the mobile terminal is located, wherein the third sensor unit comprises at least one of a temperate sensor, a humidity sensor, a light sensor, an image sensor, or air quality measurement sensor, and wherein the controller is further configured to;

obtain the context awareness information based on an input received from the third sensor unit.

12. The mobile terminal of claim 10, wherein the context awareness information comprises a history of remote control of the external electronic device by the mobile terminal in the remote control mode.

13. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the touchscreen to display a control menu for controlling the external electronic device as a pop-up window, after entering the remote control mode.

14. The mobile terminal of claim 1, wherein the external electronic device comprises at least one of a refrigerator, a washing machine, a TV, an air conditioner, an air purifier, a humidifier, a dehumidifier, a drier, a robot vacuum cleaner, a speaker, or a car.

15. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the touchscreen to display a control menu for controlling the external electronic device while the external electronic device is in an inactive state, after entering the remote control mode from an Always on Display (AOD) mode in which the touchscreen is in an inactive state.

16. A method performed at a mobile terminal having a first camera and a second camera having a wider field of view than the first camera and a touchscreen to control an external electronic device, the method comprising:

causing the mobile terminal to enter a remote control mode, for remotely controlling the external electronic device via a network, when an input from a first sensor indicates a grip state that satisfies a first defined condition, or when an input from a second sensor satisfies a second defined condition as to attitude of the mobile terminal;

entering the remote control mode;

controlling the external electronic device in response to an input received at the mobile terminal,

capturing a first image through the first camera that is in an Always on Camera (AOC) mode when the touchscreen is in an inactive state;

capturing a second image through the second camera, when only a part of the external electronic device is present in the first image;

recognizing the external electronic device in the captured second image; and

displaying an extracted name corresponding to the external electronic device as being overlapped on a preview image.

17. A mobile terminal, comprising:

a body;

a touchscreen coupled to the body;

a first sensor coupled to the body and configured to measure an applied grip;

a second sensor configured to detect attitude of the body;

a wireless communication unit configured to perform wireless communication with an external electronic device;

a first camera and a second camera having a wider field of view than the first camera; and

a controller configured to;

cause the mobile terminal to enter a remote control mode, for remotely controlling the external electronic device, when measurement from the first sensor indicates a first defined condition or when an input from the second sensor indicates a second defined condition; and

control the external electronic device in response to a received input, after entering the remote control mode;

capture an image through the first camera that is in an Always on Camera (AOC) mode when the touchscreen is in an inactive state;

capture a second image through the second camera that is in an Always on Camera (AOC) mode when only a part of the external electronic device is present in the first image;

recognize the external electronic device in the captured second image; and

display an extracted name corresponding to the external electronic device as being overlapped on a preview image.