Object information derived from object images

• US 10,635,714 B2
• Filed: 09/20/2019
• Issued: 04/28/2020
• Est. Priority Date: 11/06/2000
• Status: Expired due to Term

• Current Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Original Assignee: Nant Holdings IP, LLC (NantWorks, LLC)
• Inventors: Boncyk, Wayne C., Cohen, Ronald H.
• Primary Examiner(s): Sherali, Ishrat I
• Application Number:
  US16/578,231
• Publication Number:
  US 20200012679A1
• Time in Patent Office: 221 Days
• Field of Search: 382100, 382103, 382305, 705 261- 272
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/24554 Unary operations; Data partitioning operations
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/7335 Graphical querying, e.g. query-by-region, query-by-sketch, query-by-trajectory, GUIs for designating a person/face/object as a query predicate end-user interface involving hot spots associated with the video H04N21/4725; end-user interface for selecting a Region of Interest H04N21/4728
• CPC Class Codes: G06F 16/904 Browsing; Visualisation therefor for navigating the web G06F16/954; browsing optimisation for the web G06F16/957
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/9538 Presentation of query results
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06Q 20/10 specially adapted for electronic funds transfer [EFT] systems; specially adapted for home banking systems
• CPC Class Codes: G06Q 30/0261 based on user location
• CPC Class Codes: G06Q 30/0267 Wireless devices
• CPC Class Codes: G06Q 30/0269 based on user profile or attribute
• CPC Class Codes: G06Q 30/0601 : Electronic shopping [e-shopping]
• CPC Class Codes: G06Q 30/0613 : Third-party assisted
• CPC Class Codes: G06Q 30/0623 : Item investigation
• CPC Class Codes: G06Q 30/0625 : Directed, with specific intent or strategy
• CPC Class Codes: G06Q 30/0635 : Processing of requisition or of purchase orders
• CPC Class Codes: G06T 7/00 : Image analysis
• CPC Class Codes: G06V 10/10 : Image acquisition document image scanning and transmission H04N1/00; control of digital cameras H04N23/60
• CPC Class Codes: G06V 10/255 : Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 10/768 : using context analysis, e.g. recognition aided by known co-occurring patterns
• CPC Class Codes: G06V 20/00 : Scenes; Scene-specific elements control of digital cameras H04N23/60
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/46 : Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/80 : Recognising image objects characterised by unique random patterns
• CPC Class Codes: G06V 2201/07 : Target detection
• CPC Class Codes: G06V 40/16 : Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/18 : Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 23/80 : Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/44 : Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

An image-based transaction system includes a mobile device with an image sensor that is programmed to capture, via the image sensor, a digital images of a scene. The mobile device identifies a document using image characteristics from the captured images and acquires an image of at least a part of the document, and then identifies symbols in the image based on locations within the image of the document. The symbols can include alphanumeric symbols. The mobile device processes the symbols according to their type to obtain an address related to the document and the symbols and initiates a transaction associated with the identified document.

Claims

1. An image-based transaction system, comprising:

a mobile device having an image sensor wherein the mobile device, when software in the mobile device is executed, is caused to execute operations comprising;

digitally capturing images of a scene via the image sensor;

identifying an object using image characteristics from the digitally captured images;

automatically acquiring an image of at least part of the identified object in the scene;

identifying symbols, including alphanumeric symbols, in the image based on locations within the image;

processing the symbols according to their symbol type;

obtaining an address related to the identified object and the processed symbols; and

initiating a transaction associated with the identified object via a server, wherein the transaction is related to the address.

2. The system of claim 1, wherein the system further comprises the server.

3. The system of claim 2, wherein initiating the transaction occurs over a network.

4. The system of claim 1, wherein the mobile device comprises a portable telephone.

5. The system of claim 1, wherein the address comprises an information address.

6. The system of claim 1, wherein the address comprises a network address.

7. The system of claim 1, wherein the address references pertinent information related to the identified object.

8. The system of claim 1, wherein the processed symbols comprise decoded text information.

9. The system of claim 1, wherein the processed symbols comprise decoded numeric information.

10. The system of claim 1, wherein the transaction is with an account associated with a user.

11. The system of claim 1, wherein the symbols are determined based on at least one of a size of the symbols or a nature of the symbols.

12. The system of claim 1, wherein the symbols include at least one of the following symbol types:

a barcode, or a matrix code.

13. The system of claim 1, wherein the transaction includes transmitting the image to the server.

14. The system of claim 1, wherein the mobile device is further caused to execute operations comprising converting the image to a grayscale representation.

15. The system of claim 1, wherein the mobile device is further caused to execute operations comprising decomposing the image using at least one image processing algorithm.

16. The system of claim 15, wherein the symbols are discriminated based on a confidence level.

17. The system of claim 1, wherein the image comprises an RGB image.

18. The system of claim 1, wherein the mobile device is further caused to execute operations comprising receiving pertinent information based on the address.

19. The system of claim 1, wherein the mobile device is further caused to execute operations comprising automatically acquiring the image upon identification of the object.

20. The system of claim 1, wherein the mobile device is further caused to execute operations comprising automatically recognizing that the image is to be acquired.

21. An image-based transaction system, comprising:

a mobile device having an image sensor, wherein the mobile device, when software in the mobile device is executed, is caused to execute operations comprising;

digitally capturing images of a scene via the image sensor;

identifying an object using image characteristics from the digitally captured images;

acquiring an image of at least part of the identified object in the scene;

identifying symbols, including alphanumeric symbols, in the image based on locations within the image;

processing the symbols according to their symbol type;

obtaining an address related to the identified object and the processed symbols; and

initiating a transaction associated with the identified object via a server, wherein the transaction is related to the address.

22. The system of claim 21, wherein the system further comprises the server.

23. The system of claim 22, wherein initiating the transaction occurs over a network.

24. The system of claim 21, wherein the mobile device comprises a portable telephone.

25. The system of claim 21, wherein the address comprises an information address.

26. The system of claim 21, wherein the address comprises a network address.

27. The system of claim 21, wherein the address references pertinent information related to the identified object.

28. The system of claim 21, wherein the processed symbols comprise decoded text information.

29. The system of claim 21, wherein the processed symbols comprise decoded numeric information.

30. The system of claim 21, wherein the transaction is with an account associated with a user.

31. The system of claim 21, wherein the symbols are determined based on at least one of a size of the symbols or a nature of the symbols.

32. The system of claim 21, wherein the symbols include at least one of the following symbol types:

a barcode, or a matrix code.

33. The system of claim 21, wherein the transaction includes transmitting the image to the server.

34. The system of claim 21, wherein the mobile device is further caused to execute operations comprising converting the image to a grayscale representation.

35. The system of claim 21, wherein the mobile device is further caused to execute operations comprising decomposing the image using at least one image processing algorithm.

36. The system of claim 35, wherein the symbols are discriminated based on a confidence level.

37. The system of claim 21, wherein the image comprises an RGB image.

38. The system of claim 21, wherein the mobile device is further caused to execute operations comprising receiving pertinent information based on the address.

39. The system of claim 21, wherein the mobile device is further caused to execute operations comprising automatically acquiring the image.

40. The system of claim 39, wherein the mobile device is further caused to execute operations comprising automatically acquiring the image upon identification of the object.

41. The system of claim 21, wherein the mobile device is further caused to execute operations comprising automatically recognizing that the image is to be acquired.

42. A computer implemented method of initiating a transaction based on an image, the method comprising:

digitally capturing, via a mobile device, images of a scene via an image sensor in the mobile device;

identifying, via the mobile device, an object using image characteristics from the digitally captured images;

acquiring, via the mobile device, an image of at least part of the identified object in the scene;

identifying symbols, including alphanumeric symbols, in the image based on locations within the image;

processing the symbols according to their symbol type;

obtaining an address related to the identified object and the processed symbols; and

initiating a transaction associated with the identified object via a server, wherein the transaction is related to the address.

43. The method of claim 42, wherein the operation of acquiring the image includes automatically acquiring the image.

44. The method of claim 43, wherein the operation of acquiring the image includes automatically acquiring the image upon identification of the object.

45. The method of claim 42, further including automatically recognizing that the image is to be acquired.

46. The method of claim 42, wherein the transaction is with a bank account.

47. The method of claim 42, wherein the transaction is with at least one of the following types of accounts:

an account liked to a user, an account linked to the mobile device, or a credit card account.

48. The method of claim 42, wherein the symbols include at least one of the following symbol types:

text, numeric characters, a barcode, or a matrix code.

49. A computer program product, storable on a non-transitory computer readable memory, comprising software instructions that when executed by at least one processor cause the at least one processor to:

digitally capture images of a scene via an image sensor on a mobile device;

identify an object using image characteristics from the digitally captured images;

acquire an image of the at least part of the identified object in the scene;

identify symbols in the image based on locations within the image;

process the symbols according to their symbol type;

obtain an address related to the identified object and the processed symbols; and

initiate a transaction associated with the identified object via a server, wherein the transaction is related to the address.

50. The computer program product of claim 49, wherein the operation to acquire the image includes an operation to automatically acquire the image.

51. The computer program product of claim 50, wherein the operation to acquire the image includes an operation to automatically acquire the image upon identification of the object.

52. The computer program product of claim 49, further including an operation to automatically recognize that the image is to be acquired.

53. The computer program product of claim 49, wherein the transaction is with a bank account.

54. The computer program product of claim 49, wherein the transaction is with at least one of the following types of accounts:

an account liked to a user, an account linked to the mobile device, or a credit card account.

55. The system of claim 49, wherein the symbols include at least one of the following symbol types:

text, numeric characters, a barcode, or a matrix code.