On-demand artificial intelligence and roadway stewardship system

• US 10,296,794 B2
• Filed: 03/12/2018
• Issued: 05/21/2019
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Jayant Rtti
• Original Assignee: Jayant Rtti
• Inventors: Ratti, Jayant
• Primary Examiner(s): Lau, Hoi C
• Application Number:
  US15/919,033
• Publication Number:
  US 20180211117A1
• Time in Patent Office: 435 Days
• CPC Class Codes: G06F 15/76 Architectures of general purpose stored program computers with program plugboard G06F15/08; multicomputers G06F15/16
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06Q 50/26 Government or public services business processes related to the transportation industry G06Q50/40
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 20/54 of traffic, e.g. cars on the road, trains or boats
• CPC Class Codes: G08G 1/0175 by photographing vehicles, e.g. when violating traffic rules
• CPC Class Codes: G08G 1/054 photographing overspeeding vehicles
• CPC Class Codes: G08G 1/141 with means giving the indication of available parking spaces
• CPC Class Codes: G08G 1/143 inside the vehicles
• CPC Class Codes: G08G 1/147 where the parking area is within an open public zone, e.g. city centre

Abstract

The present disclosure relates to artificial intelligence based systems and method for determination of traffic violations. The present disclosure provides systems and methods that use deep convolutional neural networks and machine vision based algorithms to perform a task of detection and recognition to provide complete solution to safe, legal and comfortable parking, driving and riding for commuters on the roadways. Roadway stewardship systems, Parking management systems when made on-demand and crowdsourced, can play a very strong role in regulating driving conditions in cities and highways. By allowing the on-demand, crowdsourced, roadway stewardship system to be automated, through the use of Artificial Intelligence (AI) sub-systems, users can be trained to recognize and be educated as well in the laws & regulations around the use of roadways; can help the process through an interactive console/game-play, which can also be used for monetization for individuals to earn money for their contribution. The AI assisted with Human Intelligence (HI) together called HAI in particular, can play a valuable role in reducing traffic density, traffic movement restrictions and fuel and time waste in large cities. Also proper driving on the roads can lead to faster and safer commute. In Addition, multiple other objects of interest can also be identified and trained to be recognized using the Stewardship System disclosed herein.

Claims

1. An artificial intelligence (AI) based system, part of a Roadway Stewardship Network, comprising:

a non-transitory storage device having embodied therein one or more routines operable to detect objects in images using Artificial Neural Networks; and

one or more processors coupled to the non-transitory storage device and operable to execute the one or more routines, wherein the one or more routines include;

a receiver module, which when executed by the one or more processors, receives at least an object detection signal from one or more vision sensing systems, said object detection signal comprises or is accompanied by said images or series of images or a video associated with said objects;

a detector module, which when executed by the one or more processors, determines, for said objects, a region of interest (ROI) selected from the received images or a series of images or a video;

a training module which takes manually classified objects, obtained from the images or a series of images or a video and trains the Neural Network to improve the detector performance;

a logic module which takes as input detected objects of interest in a series of one or more images or videos and determines various actions or events of interest, wherein the various actions or events of interest are a location, a position, a movement, and a category associated with said objects contained in the one or more images or videos based on comparison with accumulated training data.

2. The AI based system of claim 1, wherein said system is a module added to the Stewardship network to act as a Specialist or an Auditor or a Data Collector.

3. The AI based system of claim 1, wherein the detector module, also called a detection module, which when executed by the one or more processors, recognizes a region of interest in an image and classifies it as a vehicle or a part of a vehicle or a person or static/dynamic markings or a policeman.

4. The AI based system of claim 3, wherein the region of interest (ROI) is a vehicle or at least a part of the vehicle, in a series of images along with at least one another ROIs selected from of road signs, road markings, policeman, person, in the same series of images is sufficient to determine a violation of motor vehicle laws/guidelines.

5. The AI based system of claim 4, wherein the region of interest (ROI) is a vehicle or at least a part of the vehicle in a single image along with at least one another ROIs selected from road signs, road markings, policeman, person, in the same image is sufficient to determine a violation of motor vehicle laws/guidelines.

6. The AI based system of claim 5, wherein the region of interest (ROI) is at least a vehicle or at least a part of the vehicle or at least the driver of the vehicle in a single image is sufficient to determine a violation of motor vehicle laws/guidelines.

7. The AI based system of claim 6, wherein the AI based system further comprising an annotation module, which when executed by the one or more processors, configured to read license plates detected by the detector module.

8. The AI based system of claim 7, wherein the AI based system further comprising an occlusion module, which when executed by the one or more processors, hides objects ROI or hides non-ROI information in the image or series of images or video.

9. The AI based system of claim 3, wherein the detection module, which when executed by the one or more processors, is further configured to detect traffic violations based on the orientation and position of the vehicle or wherein the detection module further takes into account the position, orientation and location of other vehicles, relative to the vehicle of interest to determine traffic violations.

10. The AI based system of claim 9, wherein the AI based system further comprising:

a calculation module, which when executed by the one or more processors, configured to calculate a speed of said object utilizing object detection from the location of the said object in different frames of the video or series of images and the relative velocity of the vehicle.

11. The AI based system of claim 10, wherein the AI based system further comprising:

an allocation module, which when executed by the one or more processors, configured to allocate, one or more parking slot options, based on detection of said parking slots or by matching a host of a parking spot to the driver in need of a parking spot.

12. A method for detecting objects in images and videos using at least one or more artificial neural networks, the method comprising:

receiving, by a system, at least an object detection signal from one or more vision sensing systems, said object detection signal comprises said images associated with said objects;

determining by the system, a Region Of Interest (ROI) selected from the received images for said objects;

detecting by the system, a location, a position, a movement, and a category associated with said objects contained in the received images, using the artificial neural networks which are trained using accumulated training data, wherein the accumulated training data is obtained from specialists by having them perform among activities involving;

identifying and annotating license plates of vehicles, identifying the 2D boundary around vehicles, identifying the types, make, model and other information about the vehicles, identifying various roadway violations committed by the vehicles, identifying roadway markings and signs, identifying and marking ROI for objects of interest.

13. The method of claim 12, wherein a data collector, configured to obtain the images and videos, has at least one sensor, visually capturing the direction(s) of gaze of person(s) inside the vehicle and at least one sensor capturing the direction(s) of visual marking(s) outside the vehicle;

wherein when, the general direction(s) of gaze and visual marking(s) match, provides a general confirmation of the visual markings having been witnessed.

14. The method of claim 13, wherein the visual markings collected by the data collector are given a score based on number of persons who witness the markings, and wherein the scores are used to qualify the efficacy of placement of the visual marking(s) and the efficacy of the content(s) of the visual marking(s).

15. The method of claim 14, wherein a speed of the vehicle inside the ROI is measured from objects in images and videos collected by the data collector by checking the centroid or median or mean location of a license plates relative to the data collector.

16. The method of claim 15, wherein a Stewardship network, in which the specialists and auditors who review a video of series of images collected by the data collectors, earn points based on a count and accuracy of the detection of traffic violations, wherein, the collection of points emulates a game.

17. The method of claim 16, wherein the detection and identification of the system is supported by re-training the neural networks by adding the reviewed data to the accumulated training data.