System and method for interacting with objects via a camera enhanced mobile device

• US 7,830,417 B2
• Filed: 08/15/2006
• Issued: 11/09/2010
• Est. Priority Date: 08/15/2006
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Yang, Jack, Zhou, Hanning, Denoue, Laurent, Chiu, Patrick, Liu, Qiong, Kimber, Donald
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US11/504,873
• Publication Number:
  US 20080200205A1
• Time in Patent Office: 1,547 Days
• Field of Search: 348/211.99, 725/105
• US Class Current: 348/211.99
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

Embodiments of the present invention enable an image based controller to control and manipulate objects with simple point-and-capture operations via images captured by a camera enhanced mobile device. Powered by this technology, a user is able to complete many complicated control tasks via guided control of objects without utilizing laser pointers, IR transmitters, mini-projectors, or bar code tagging and/or customized wall paper are not needed for the environment control. This description is not intended to be a complete description of, or limit the scope of, the invention. Other features, aspects, and objects of the invention can be obtained from a review of the specification, the figures, and the claims.

Claims

1. A system to support image based object controlling, comprising:

a camera enhanced mobile device operable to;

point to and capture one or more images of an object of interest; and

send the captured one or more images of the object to a processing center via a network;

a database of object features containing features of a plurality of objects;

said processing center operable to;

extract a plurality of features of the object from the one or more images captured by the camera enhanced mobile device;

identify the object the user points to based on the database of object features; and

manipulate and control the identified object via an interface unit; and

said interface unit operable to interact with both the processing center and the object.

2. The system according to claim 1, wherein:

the object is one of a visible networked device and a visible un-networked physical object.

3. The system according to claim 2, wherein:

the visible networked device is one of;

a video projector, a side or rear flat panel, a front or back room loudspeaker, a television, and an air conditioner.

4. The system according to claim 2, wherein:

the visible un-networked physical object is one of a paper document, a white board, a traditional instruments not connected to any network, and an audio signal recorded by a microphone attached to a mobile device.

5. The system according to claim 1, wherein:

the camera enhanced mobile device is one of a cell phone and a PDA.

6. The system according to claim 1, wherein:

the network is one of Internet and a wireless communication network.

7. The system according to claim 1, wherein:

each of the plurality of features of the object is one of;

a Scale Invariant Feature Transform (SIFT) feature, a Gabor feature, a Discrete Cosine Transform (DCT) coefficient, an Optical Character Recognition (OCR) results, and data captured by other sensors on the camera enhanced mobile device.

8. The system according to claim 1, wherein:

the processing center is one of a stand-alone computing unit and an embedded unit in a mobile device or a service center.

9. The system according to claim 1, wherein:

the interface unit has backward compatibility if the object is a traditional device.

10. The system according to claim 1, wherein:

The processing center is operable to manipulate the object by sending a command to the object if the object is on the same network or transforming the command into Infra-Red (IR) signals to control the object.

11. A method to support image based object controlling, comprising:

establishing a database containing features of a plurality of objects;

capturing one or more images of an object via a camera enhanced mobile device and sending the one or more images to a processing center via a network;

extracting a plurality of features of the object from the one or more images captured by the camera enhanced mobile device;

identifying the object the user points to based on the database of object features; and

manipulating and controlling the identified object via commands.

12. The method according to claim 11, further comprising:

defining a plurality of operation rules between the one or more images of the object and its manipulation actions.

13. The method according to claim 11, further comprising:

interacting with the object via an interface unit if the object is a traditional device.

14. The method according to claim 11, further comprising:

manipulating the object by sending a command to the object if the object is on the same network or transforming the command into Infra-Red (IR) signals to control the object.

15. The method according to claim 11, further comprising:

identifying the object in the one or more images via Scale Invariant Feature Transform (SIFT).

16. A non-transitory computer readable storage medium having instructions stored thereon that when executed cause a system to:

establish a database containing features of a plurality of objects;

capture one or more images of an object via a camera enhanced mobile device and send the one or more images to a processing center via a network;

extract a plurality of features of the object from the one or more images captured by the camera enhanced mobile device;

identify the object the user points to based on the database of object features; and

manipulate and control the identified object via commands.

17. A system to support image based object controlling, comprising:

means for establishing a database containing features of a plurality of objects;

means for capturing one or more images of an object via a camera enhanced mobile device and sending the one or more images to a processing center via a network;

means for extracting a plurality of features of the object from the one or more images captured by the camera enhanced mobile device;

means for identifying the object the user points to based on the database of object features; and

means for manipulating and controlling the identified object via commands.