Smartphone-based asset management system

• US 9,805,331 B2
• Filed: 02/09/2011
• Issued: 10/31/2017
• Est. Priority Date: 02/09/2011
• Status: Active Grant

• Current Assignee: Deutsche Telekom AG, Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Original Assignee: Deutsche Telekom AG, Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.)
• Inventors: Hsu, Cheng-Hsin, Kim, Kyu-Han, Singh, Jatinder Pal, Chen, David M., Tsai, Sam, Girod, Bernd
• Primary Examiner(s): CRAWLEY, TALIA F
• Application Number:
  US13/023,679
• Publication Number:
  US 20120202515A1
• Time in Patent Office: 2,456 Days
• Field of Search: 705 28, 707758
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders

Abstract

A method of tracking an inventory of objects via a mobile communications device includes acquiring an image of one or more of the objects via the mobile communications device, which also collects a location of the mobile communications device while acquiring the image of the one or more of the objects. The location and image are transferred from the mobile communications device to a remote server via a wireless network, such that the one or more of the objects are identified at the server based on the image, and the location and identity of the one or more objects are stored on a database associated with the server.

Claims

1. A computer-implemented method for image processing and object recognition, the method comprising:

receiving, by a remote image processing server, an image of a plurality of objects and location information corresponding to the image from a mobile communications device via a wireless network;

performing, by an image analyzer of the remote image processing server, segmentation to divide the received image into a plurality of regions, wherein performing segmentation comprises;

generating an edge map from the image;

removing components of the edge map that are smaller than a predetermined threshold size from the edge map;

determining a dominant angle for substantially parallel lines of the edge map; and

identifying the plurality of regions of the image, wherein the plurality of regions of the image correspond to regions of the edge map between lines of the edge map within a predetermined threshold angle of the dominant angle;

for at least one region of the plurality of regions, extracting, by the remote image processing server, features of the region and matching the features of the region with a database entry to determine an object identity of a single object corresponding to the region;

determining, by the remote image processing server, a localized position of the single object within a room based on the received location information and further based on previously acquired training information;

storing, by the remote image processing server, the localized position of the single object with the corresponding object identity in a database;

receiving, by the remote image processing server, a query for an object;

determining;

by the remote image processor server, in response to the query, a localized position for the object based on information stored in the database; and

outputting, by the remote image processor server, the localized position for the object.

2. The method according to claim 1, wherein the image received from mobile communications device was acquired via a built in camera of the mobile communications device.

3. The method according to claim 1, wherein the location information received from the mobile communications device was determined by the device based on local wireless signals.

4. The method according to claim 3, wherein the location information received from the mobile communications device was further determined based on sensor signals from the one or more accelerometers hosted within the mobile communications device.

14. The method according to claim 1, wherein the objects are books.

15. The method according to claim 14, wherein the image received from the mobile communications device is an image of the contents of a book shelf.

16. The method according to claim 14, wherein the image received from the mobile communications device is an image of the contents a single rack of a book shelf having multiple racks.

5. A non-transitory computer-readable medium, part of a remote image processing server, having processor-executable instructions stored thereon for image processing and object recognition, the processor-executable instructions, when executed by a processor, causing the following steps to be performed:

receiving an image of a plurality of objects and location information corresponding to the image from a mobile communications device via a wireless network;

performing segmentation to divide the received image into a plurality of regions, wherein performing segmentation comprises;

generating an edge map from the image;

removing components of the edge map that are smaller than a predetermined threshold size from the edge map;

determining a dominant angle for substantially parallel lines of the edge map; and

identifying the plurality of regions of the image, wherein the plurality of regions of the image correspond to regions of the edge map between lines of the edge map within a predetermined threshold angle of the dominant angle;

for at least one region of the plurality of regions, extracting features of the region and matching the features of the region with a database entry to determine an object identity of a single object corresponding to the region;

determining a localized position of the single object within a room based on the received location information and farther based on previously acquired training information; and

storing the localized position of the single object with the corresponding object identity in a database;

receiving a query for an object;

determining, in response to the query, a localized position for the object based o information stored in the database; and

outputting the localized position for the object.

6. The non-transitory computer-readable medium according to claim 5, wherein the received image was acquired via a built in camera of the mobile communications device.

7. The non-transitory computer-readable medium according to claim 5, wherein the location information was determined by the mobile communications device based on local wireless signals.

8. The non-transitory computer-readable medium according to claim 7, wherein the location information was further determined based on sensor signals from one or more accelerometers hosted within the mobile communications device.

9. The non-transitory computer-readable medium according to claim 5, wherein the objects are books.

10. The non-transitory computer-readable medium according to claim 9, wherein the received image is an image of the contents of a book shelf.

11. The non-transitory computer-readable medium according to claim 9, wherein the received image is an image of the contents of a single rack of a book shelf having multiple racks.

12. A remote image processing server, the server comprising:

a receiver, configured to receive an image of a plurality of objects and insertion information corresponding to the image from a mobile communications devise via a wireless network;

a processor, configured to;

perform segmentation to divide the received image into a plurality of regions, wherein performing segmentation comprises;

generating an edge map from the image;

removing components of the edge map that are smaller than a predetermined threshold size from the edge map;

determining a dominant angle for substantially parallel lines of the edge map; and

identifying the plurality of regions of the image, wherein the plurality of regions of the image correspond to regions of the edge map between lines of the edge map within a predetermined threshold angle of the dominant angle;

for at least one region of the plurality of regions extract features of the region and matching the features of the region with a database entry to determine an object identity of a single object corresponding to the region, wherein a Speeded Up Robust Features (SURF) algorithm is used to match the features of the region with the database entry; and

determine a localized position of the single object within a room leased on the received location information and farther based on previously acquired training information; and

a database, configured to store the localized position of the single object with the corresponding object identity;

wherein the receiver is further configured to receive a query for an object; and

wherein the processor is further configured to determine and output the localized position for the object based on information stored in the database.

13. The server of claim 12, wherein the localized position of the single object within the room is received from the mobile communications device.