Mobile terminal and method for controlling the same

• US 10,389,863 B2
• Filed: 11/07/2016
• Issued: 08/20/2019
• Est. Priority Date: 11/11/2015
• Status: Active Grant

• Current Assignee: LG Electronics, Inc. (LG Corporation)
• Original Assignee: LG Electronics, Inc. (LG Corporation)
• Inventors: Jeon, Bongjeong, Kim, Insuk, Choi, Jian, Kim, Namki, Oh, Sesook
• Primary Examiner(s): Tran, Khoi H
• Assistant Examiner(s): Hannan, B M M
• Application Number:
  US15/345,361
• Publication Number:
  US 20170134553A1
• Time in Patent Office: 1,016 Days
• Field of Search: 455420
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: H04L 67/12 specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/125 involving control of end-device applications over a network
• CPC Class Codes: H04M 1/72412 using two-way short-range wireless interfaces
• CPC Class Codes: H04M 1/72415 for remote control of appliances
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04W 4/026 using orientation information, e.g. compass
• CPC Class Codes: H04W 4/08 User group management
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]
• CPC Class Codes: H04W 76/10 Connection setup

Abstract

A mobile terminal includes a camera unit configured to sense an image, a touch screen configured to output image information, a communication unit configured to perform wireless communication with at least one Internet of things (IOT) device, and a controller configured to, when the mobile terminal enters a specific indoor area, control IOT devices included in the sensed image or a previously stored image of the specific indoor area by using the image, wherein the controller stores a plurality of pieces of set information including a state in which at least some of the IOT devices are connected and operational states of the connected IOT devices, and when any one of the plurality of pieces of set information is selected, the controller controls at least some of the IOT devices to be connected to each other and operate according to the selected set information.

Claims

1. A mobile terminal comprising:

a camera;

a memory configured to store information;

a touchscreen configured to display information;

a transceiver configured to perform wireless communication with at least one device; and

a controller configured to;

cause the touchscreen to display an augmented reality (AR) image comprising a plurality of devices located in a specific indoor area where the mobile terminal is located and which can be changed according to an orientation direction of the camera, wherein the AR image comprises displayed information of each of the plurality of devices next to each corresponding displayed device in the AR image;

control two or more devices of the plurality of devices selected from the AR image to be connected to each other;

cause the memory to store set information in response to a storing command, the stored set information comprising connection information of the two or more devices and operational state information of the two or more devices when the connection information is stored; and

cause the two or more devices to operate according to the stored operational state information in response to a selection of the stored set information,

wherein the controller is further configured to perform authentication of the mobile terminal for controlling the plurality of devices, the authentication comprising;

establishing a connection with an access point via the transceiver, wherein the access point includes information related to the plurality of devices in the specific indoor area;

causing the touchscreen to display notification information indicating that the mobile terminal is not registered;

causing the transceiver to transmit a first authentication information received from a user when the first authentication information matches user authentication information previously registered at the access point;

causing the touchscreen to display an image sensed by the camera as a preview image comprising a specific device of the plurality of devices;

receiving second authentication information from the user different from the first authentication information in a state where the specific device of the plurality of devices is displayed on the preview image; and

obtaining authority to control the specific device displayed on the preview image when the second authentication information matches another user authentication information previously registered at the access point.

2. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the set information to be stored to be associated with a specific situational setting; and

cause the memory to differently store multiple set information associated with a same specific situational setting based on corresponding surrounding environment information detected by the mobile terminal when each of the multiple set information is stored.

3. The mobile terminal of claim 2, wherein the surrounding environment information comprises a current time and weather information.

4. The mobile terminal of claim 1, wherein the controller is further configured to cause the touchscreen to display information for changing an operational state of at least one of the two or more devices based on surrounding environment information detected by the mobile terminal.

5. The mobile terminal of claim 4, wherein the displayed information for changing the operational state of the at least one of the two or more devices is further based on the detected surrounding environment information and a current operational state of the two or more devices.

6. The mobile terminal of claim 5, wherein the controller is further configured to update the selected set information when an operational state of at least one of the two or more devices is changed according to the displayed information.

7. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the touchscreen to display graphic objects corresponding to each of the two or more devices; and

cause a selected one of the two or more devices to be disconnected when a graphic object corresponding to the selected one device is selected.

8. The mobile terminal of claim 1, wherein the controller is further configured to select the two or more devices from the plurality of devices for the stored set information according to a drag trace received via the touchscreen, wherein the drag trace forms a loop intersecting each of the two or more devices in the displayed image.

9. The mobile terminal of claim 1, wherein the controller is further configured to cause the set information to be stored when a preset period expires or when a connection state of one of the two or more devices is changed.

10. The mobile terminal of claim 1, wherein the controller is further configured to cause the touchscreen to display a control interface for each of the two or more devices, wherein the control interfaces are different based on a size of the corresponding device in the displayed image.

11. The mobile terminal of claim 10, wherein the size of the corresponding device in the displayed image is changed according to zooming-in or zooming-out of the image or according to movement of the mobile terminal within the specific indoor area.

12. The mobile terminal of claim 10, wherein the controller is further configured to:

calculate a ratio of the size of the corresponding device in the displayed image to a total size of a display of the touchscreen; and

change a displayed control interface based on the calculated ratio, wherein the displayed control interface is displayed to have a higher granularity of control of the corresponding device as a size of the corresponding device in the displayed image is increased.

13. The mobile terminal of claim 1, wherein the controller is further configured to:

cause the touchscreen to display a graphic object related to an operational state of a selected device of the two or more devices in response to a selection of the selected device in the displayed image; and

control the selected device based on a touch input to the displayed graphic object.

14. The mobile terminal of claim 13, wherein the controller is further configured to cause the touchscreen to display an effect on the displayed image according to a change in the operational state of one of the two or more devices.

15. The mobile terminal of claim 1, further comprising a sensor configured to detect a state in which the mobile terminal is gripped,

wherein the preview image obtained by the camera or a previously stored image is displayed based on the detected state in which the mobile terminal is gripped.

16. The mobile terminal of claim 15, wherein the controller is further configured to cause the touchscreen to display another previously stored image corresponding to another indoor area when a specific object included in the previously stored image is displayed on the touchscreen and a selection input is received to the displayed specific object,

wherein the displayed another previously stored image includes another plurality of devices.

17. The mobile terminal of claim 16, wherein the controller is further configured to connect one of the another plurality of devices with one of the two or more devices.

18. The mobile terminal of claim 16, wherein the controller is further configured to cause the touchscreen to display an indicator corresponding to a region of the specific object on the displayed image when the another previously stored image is stored.

19. The mobile terminal of claim 1, wherein the controller is further configured to:

determine that the mobile terminal has entered the specific indoor area when the mobile terminal is connected to the access point.

20. A method for controlling a mobile terminal, the method comprising:

displaying an augmented reality (AR) image comprising a plurality of devices located in a specific indoor area where the mobile terminal is located and which can be changed according to an orientation direction of a camera of the mobile terminal, wherein the AR image comprises displayed information of each of the plurality of devices next to each corresponding displayed device in the AR image; and

controlling two or more of the plurality of devices to be connected to each other;

storing a set information item in response to a storing command, the stored set information comprising connection information of the connected two or more of the plurality of devices and operational state information of the connected two or more of the plurality of devices when the connection information is stored;

displaying a plurality of set information items wherein each of the plurality of set information items are displayed differently based on surrounding environment information detected when each corresponding set information item was stored; and

causing at least part of the plurality of devices to operate according to the operation states when storing the set information item in response to a selection of the stored set information,

wherein the method further comprises performing authentication of the mobile terminal for controlling the plurality of devices, the authentication comprising;

establishing a connection with an access point, wherein the access point includes information related to the plurality of devices in the specific indoor area;

displaying notification information indicating that the mobile terminal is not registered;

transmitting a first authentication information received from a user when the first authentication information matches user authentication information previously registered at the access point;

displaying an image sensed by the camera as a preview image comprising a specific device of the plurality of devices;

receiving a second authentication information from the user different from the first authentication information in a state where the specific device of the plurality of devices is displayed on the preview image; and

obtain authority to control the specific device displayed on the preview image when the second authentication information matches another user authentication information previously registered at the access point.