OBJECT MANAGEMENT APPARATUS, MOBILE TERMINAL, AND OBJECT MANAGEMENT METHOD

• US 20090290014A1
• Filed: 04/29/2009
• Published: 11/26/2009
• Est. Priority Date: 05/23/2008
• Status: Active Grant

• Current Assignee: PFU Limited (Fujitsu Limited)
• Original Assignee: PFU Limited (Fujitsu Limited)
• Inventors: TAKATA, Hiroaki
• Granted Patent: US 8,094,242 B2
• Time in Patent Office: Days
• US Class Current: 348/61
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer

Abstract

An object management apparatus comprising an image photographing unit, a storage unit, and a control unit communicably connected to at least one operation target device and a mobile terminal including a displacement detecting unit that detects a displacement of a physical quantity caused when the mobile terminal moves and a display unit via a network, wherein the storage unit includes, a display object storing unit, an operation target object storing unit, and a processing command storing unit, and the control unit includes, a display object transmitting unit, a mobile terminal position acquiring unit, an region determining unit, an operation target object determining unit, a processing command determining physical quantity displacement receiving unit, a display object acquiring unit, a processing command searching unit, and a processing command executing unit.

Claims

1. An object management apparatus comprising an image photographing unit, a storage unit, and a control unit communicably connected to at least one operation target device and a mobile terminal including a displacement detecting unit that detects a displacement of a physical quantity caused when the mobile terminal moves and a display unit via a network, wherein

the storage unit includes:

a display object storing unit that stores a display object corresponding to processing data;

an operation target object storing unit that associates and stores an operation target object corresponding to the operation target device and a specific region on a work plane where the operation target object is present; and

a processing command storing unit that associates and stores position information of the mobile terminal, the displacement of the physical quantity of the mobile terminal, and a processing command for specifying processing to be performed on the operation target device using the display object, and

the control unit includes;

a display object transmitting unit that transmits the display object stored in the display object storing unit to the mobile terminal to cause the display unit to display the display object;

a mobile terminal position acquiring unit that acquires the position information of the mobile terminal on the work plane by comparing a first image on the work plane photographed by the image photographing unit and in which the operation target object is displayed in the specific region and a second image in which the mobile terminal is displayed on the work plane;

an region determining unit that determines whether the position information is included in the specific region by comparing the position information of the mobile terminal acquired by the mobile terminal position acquiring unit and the specific region of the operation target object stored in the operation target object storing unit;

an operation target object determining unit that determines the operation target object associated with the specific region from the operation target object storing unit when the region determining unit determines that the position information is included in the specific region;

a processing command determining physical quantity displacement receiving unit that receives the displacement of the physical quantity detected by the displacement detecting unit of the mobile terminal from the mobile terminal after the operation target object being determined by the operation target object determining unit;

a display object acquiring unit that acquires the display object transmitted to the mobile terminal by the display object transmitting unit and displayed on the display unit from the display object storing unit when the displacement of the physical quantity is received by the processing command determining physical quantity displacement receiving unit;

a processing command searching unit that searches for the processing command associated with the position information and the displacement of the physical quantity from the processing command storing unit using the position information of the mobile terminal acquired by the mobile terminal position acquiring unit and the displacement of the physical quantity received by the processing command determining physical quantity displacement receiving unit as search keys; and

a processing command executing unit that causes the operation target device corresponding to the operation target object determined by the operation target object determining unit to execute the processing command searched for by the processing command searching unit using the display object acquired by the display object acquiring unit.

2. The object management apparatus according to claim 1, wherein

the storing unit further includes:

an operation command storing unit that associates and stores the displacement of the physical quantity of the mobile terminal and an operation command for specifying operation processing on the display object displayed on the display unit of the mobile terminal, and

the control unit further includes;

an operation command determining physical quantity displacement receiving unit that receives the displacement of the physical quantity detected by the displacement detecting unit of the mobile terminal from the mobile terminal after being determined by the region determining unit that the position information is not included in the specific region;

an operation command searching unit that searches for the operation command associated with the displacement of the physical quantity from the operation command storing unit using the displacement of the physical quantity received by the operation command determining physical quantity displacement receiving unit as the search key; and

an operation command executing unit that executes the operation command searched for by the operation command searching unit on the display object.

3. The object management apparatus according to claim 2, wherein

the operation command specifies operation processing to extract, from the display object storing unit, another display object that is different from the display object displayed on the display unit of the mobile terminal and to transmit the extracted display object to the mobile terminal to cause the display unit to display the display object according to the displacement of the physical quantity of the mobile terminal.

4. The object management apparatus according to claim 1, wherein

the displacement detecting unit of the mobile terminal is a three-axis acceleration sensor and the physical quantity includes gravitational acceleration or acceleration.

5. The object management apparatus according to claim 1, wherein

the control unit further includes:

an image processing unit that performs image processing on the first image and the second image acquired by the image photographing unit so that the first image and the second image become images photographed from a front direction by performing projective transformation, wherein

the mobile terminal position acquiring unit

acquires the position information of the mobile terminal on the work plane by comparing the first image and the second image after the projective transformation on which image processing is performed by the image processing unit.

6. The object management apparatus according to claim 1, wherein

the operation target object contains characteristic information to determine at least one of a character, symbol, figure, photo, or model to identify the operation target device corresponding to the operation target object, wherein

the control unit further includes:

a specific region extracting unit that extracts the specific region on the work plane where the operation target object is present from the first image based on the characteristic information to associate the operation target object and the specific region extracted by the specific region extracting unit and to store the associated operation target object and the associated specific region in the operation target object storing unit.

7. A mobile terminal communicably connected to at least one operation target device and an object management apparatus via a network and comprising a displacement detecting unit that detects a displacement of a physical quantity caused when the mobile terminal moves, a display unit, an image photographing unit, a storage unit, and a control unit, wherein

the storage unit includes:

a display object storing unit that stores a display object corresponding to processing data;

an operation target object storing unit that stores an operation target object corresponding to the operation target device; and

a processing command storing unit that associates and stores the operation target object, the displacement of the physical quantity of the mobile terminal, and a processing command for specifying processing to be performed on the operation target device using the display object, and

the control unit includes;

a display object displaying unit that displays the display object stored in the display object storing unit on the display unit;

an operation target object extracting unit that extracts, based on at least a portion of images on a work plane photographed by the image photographing unit and in which the operation target object is displayed in the specific region, the operation target object in the images;

a matching determining unit that determines whether two operation target objects match by comparing the operation target object extracted by the operation target object extracting unit and the operation target object stored in the operation target object storing unit;

an operation target object determining unit that determines the matched operation target object from the operation target object storing unit after being determined by the matching determining unit that the two operation target objects match;

a processing command determining physical quantity displacement detecting unit that detects the displacement of the physical quantity by the displacement detecting unit after the operation target object being determined by the operation target object determining unit;

a display object acquiring unit that acquires the display object displayed on the display unit by the display object displaying unit from the display object storing unit when the displacement of the physical quantity is detected by the processing command determining physical quantity displacement detecting unit;

a processing command searching unit that searches for the processing command associated with the operation target object and the displacement of the physical quantity from the processing command storing unit using the operation target object determined by the operation target object determining unit and the displacement of the physical quantity detected by the processing command determining physical quantity displacement detecting unit as search keys; and

a processing command executing unit that causes the operation target device corresponding to the operation target object determined by the operation target object determining unit to execute the processing command searched for by the processing command searching unit using the display object acquired by the display object acquiring unit.

8. The mobile terminal according to claim 7, wherein

the storing unit further includes:

an operation command storing unit that associates and stores the displacement of the physical quantity of the mobile terminal and an operation command for specifying operation processing on the display object displayed on the display unit of the mobile terminal, and

the control unit further includes;

an operation command determining physical quantity displacement detecting unit that detects the displacement of the physical quantity by the displacement detecting unit after being determined by the matching determining unit that the two operation target objects do not match;

an operation command searching unit that searches for the operation command associated with the displacement of the physical quantity from the operation command storing unit using the displacement of the physical quantity detected by the operation command determining physical quantity displacement detecting unit as the search key; and

an operation command executing unit that executes the operation command searched for by the operation command searching unit on the display object.

9. The mobile terminal according to claim 8, wherein

the operation command specifies operation processing to extract, from the display object storing unit, another display object that is different from the display object displayed on the display unit of the mobile terminal and to cause the display unit to display the extracted display object according to the displacement of the physical quantity of the mobile terminal.

10. The mobile terminal according to claim 7, wherein

the displacement detecting unit is a three-axis acceleration sensor and the physical quantity includes gravitational acceleration or acceleration.

11. The mobile terminal according to claim 7, wherein

the control unit further includes:

an image processing unit that performs image processing on the image acquired by the image photographing unit so that the image becomes an image photographed from a front direction by performing projective transformation, wherein

the operation target object extracting unit

extracts the operation target object in the image based on the image after the projective transformation on which image processing is performed by the image processing unit.

12. The mobile terminal according to claim 7, wherein

the operation target object contains characteristic information to determine at least one of a character, symbol, figure, photo, or model to identify the operation target device corresponding to the operation target object, wherein

the operation target object extracting unit

extracts the operation target object from the image based on the characteristic information.

13. An object management method executed by an object management apparatus including an image photographing unit, a storage unit, and a control unit communicably connected to at least one operation target device and a mobile terminal including a displacement detecting unit that detects a displacement of a physical quantity caused when the mobile terminal moves and a display unit via a network, wherein

the storage unit includes:

a display object storing unit that stores a display object corresponding to processing data;

an operation target object storing unit that associates and stores an operation target object corresponding to the operation target device and a specific region on a work plane where the operation target object is present; and

a processing command storing unit that associates and stores position information of the mobile terminal, the displacement of the physical quantity of the mobile terminal, and a processing command for specifying processing to be performed on the operation target device using the display object, and

the method comprises;

a display object transmitting step of transmitting the display object stored in the display object storing unit to the mobile terminal to cause the display unit to display the display object;

a mobile terminal position acquiring step of acquiring the position information of the mobile terminal on the work plane by comparing a first image on the work plane photographed by the image photographing unit and in which the operation target object is displayed in the specific region and a second image in which the mobile terminal is displayed on the work plane;

an region determining step of determining whether the position information is included in the specific region by comparing the position information of the mobile terminal acquired at the mobile terminal position acquiring step and the specific region of the operation target object stored in the operation target object storing unit;

an operation target object determining step of determining the operation target object associated with the specific region from the operation target object storing unit after being determined that the position information is included in the specific region at the region determining step;

a processing command determining physical quantity displacement receiving step of receiving the displacement of the physical quantity detected by the displacement detecting unit of the mobile terminal from the mobile terminal after the operation target object being determined at the operation target object determining step;

a display object acquiring step of acquiring the display object transmitted to the mobile terminal at the display object transmitting step and displayed on the display unit from the display object storing unit when the displacement of the physical quantity is received at the processing command determining physical quantity displacement receiving step;

a processing command searching step of searching for the processing command associated with the position information and the displacement of the physical quantity from the processing command storing unit using the position information of the mobile terminal acquired at the mobile terminal position acquiring step and the displacement of the physical quantity received at the processing command determining physical quantity displacement receiving step as search keys; and

a processing command executing step of causing the operation target device corresponding to the operation target object determined at the operation target object determining step to execute the processing command searched for at the processing command searching step using the display object acquired at the display object acquiring step,

wherein the steps are executed by the control unit.

14. The object management method according to claim 13, wherein

the storing unit further includes:

an operation command storing unit that associates and stores the displacement of the physical quantity of the mobile terminal and an operation command for specifying operation processing on the display object displayed on the display unit of the mobile terminal, and

the method further comprises;

an operation command determining physical quantity displacement receiving step of receiving the displacement of the physical quantity detected by the displacement detecting unit of the mobile terminal from the mobile terminal after being determined that the position information is not included in the specific region at the region determining step;

an operation command searching step of searching for the operation command associated with the displacement of the physical quantity from the operation command storing unit using the displacement of the physical quantity received at the operation command determining physical quantity displacement receiving step as the search key; and

an operation command executing step of executing the operation command searched for at the operation command searching step on the display object.

15. The object management method according to claim 14, wherein

the operation command specifies operation processing to extract, from the display object storing unit, another display object that is different from the display object displayed on the display unit of the mobile terminal and to transmit the extracted display object to the mobile terminal to cause the display unit to display the display object according to the displacement of the physical quantity of the mobile terminal.

16. The object management method according to claim 13, wherein

the displacement detecting unit of the mobile terminal is a three-axis acceleration sensor and the physical quantity includes gravitational acceleration or acceleration.

17. The object management method according to claim 13, wherein

the method further comprises:

an image processing step of performing image processing on the first image and the second image acquired by the image photographing unit so that the first image and the second image become images photographed from a front direction by performing projective transformation, wherein

the mobile terminal position acquiring step includes

acquiring the position information of the mobile terminal on the work plane by comparing the first image and the second image after the projective transformation on which image processing is performed at the image processing step.

18. The object management method according to claim 13, wherein

the operation target object contains characteristic information to determine at least one of a character, symbol, figure, photo, or model to identify the operation target device corresponding to the operation target object, wherein

the method further comprises:

a specific region extracting step of extracting the specific region on the work plane where the operation target object is present from the first image based on the characteristic information to associate the operation target object and the specific region extracted at the specific region extracting step and to store the associated operation target object and the associated specific region in the operation target object storing unit.

19. An object management method executed by a mobile terminal communicably connected to at least one operation target device and an object management apparatus via a network and including a displacement detecting unit that detects a displacement of a physical quantity caused when the mobile terminal moves, a display unit, an image photographing unit, a storage unit, and a control unit, wherein

the storage unit includes:

a display object storing unit that stores a display object corresponding to processing data;

an operation target object storing unit that stores an operation target object corresponding to the operation target device; and

a processing command storing unit that associates and stores the operation target object, the displacement of the physical quantity of the mobile terminal, and a processing command for specifying processing to be performed on the operation target device using the display object, and

the method comprises;

a display object displaying step of displaying the display object stored in the display object storing unit on the display unit;

an operation target object extracting step of extracting, based on at least a portion of images on a work plane photographed by the image photographing unit and in which the operation target object is displayed in the specific region, the operation target object in the images;

a matching determining step of determining whether two operation target objects match by comparing the operation target object extracted at the operation target object extracting step and the operation target object stored in the operation target object storing unit;

an operation target object determining step of determining the matched operation target object from the operation target object storing unit after being determined that the two operation target objects match at the matching determining step;

a processing command determining physical quantity displacement detecting step of detecting the displacement of the physical quantity by the displacement detecting unit after the operation target object being determined at the operation target object determining step;

a display object acquiring step of acquiring the display object displayed on the display unit at the display object displaying step from the display object storing unit when the displacement of the physical quantity is detected at the processing command determining physical quantity displacement detecting step;

a processing command searching step of searching for the processing command associated with the operation target object and the displacement of the physical quantity from the processing command storing unit using the operation target object determined at the operation target object determining step and the displacement of the physical quantity detected at the processing command determining physical quantity displacement detecting step as search keys; and

a processing command executing step of causing the operation target device corresponding to the operation target object determined at the operation target object determining step to execute the processing command searched for at the processing command searching step using the display object acquired at the display object acquiring step,

wherein the steps are executed by the control unit.

20. The object management method according to claim 19, wherein

the storing unit further includes:

an operation command storing unit that associates and stores the displacement of the physical quantity of the mobile terminal and an operation command for specifying operation processing on the display object displayed on the display unit of the mobile terminal, and

the method further comprises;

an operation command determining physical quantity displacement detecting step of detecting the displacement of the physical quantity by the displacement detecting unit after being determined that the two operation target objects do not match at the matching determining unit;

an operation command searching step of searching for the operation command associated with the displacement of the physical quantity from the operation command storing unit using the displacement of the physical quantity detected at the operation command determining physical quantity displacement detecting step as the search key; and

an operation command executing step of executing the operation command searched for at the operation command searching step on the display object.

21. The object management method according to claim 20, wherein

the operation command specifies operation processing to extract, from the display object storing unit, another display object that is different from the display object displayed on the display unit of the mobile terminal and to cause the display unit to display the extracted display object according to the displacement of the physical quantity of the mobile terminal.

22. The object management method according to claim 19, wherein

the displacement detecting unit is a three-axis acceleration sensor and the physical quantity includes gravitational acceleration or acceleration.

23. The object management method according to claim 19, wherein

the method further comprises:

an image processing step of performing image processing on the image acquired by the image photographing unit so that the image becomes an image photographed from a front direction by performing projective transformation, wherein

the operation target object extracting step includes

extracting the operation target object in the image based on the image after the projective transformation on which image processing is performed at the image processing step.

24. The object management method according to claim 19, wherein

the operation target object contains characteristic information to determine at least one of a character, symbol, figure, photo, or model to identify the operation target device corresponding to the operation target object, wherein

the operation target object extracting step includes

extracting the operation target object from the image based on the characteristic information.