PARKING OBJECTS DETECTION SYSTEM
1. A parking vehicle detection system, comprising:
- a source beacon configured to broadcast its identity by emitting a short-range wireless radio signal from a location of apparatus associated with an access region of a vehicle parking location, and the source beacon configured to establish on a wireless communication link connectivity with a smart device implemented with radio frequency wireless technology to scan for a source beacon in a vicinity of the smart device and to provide location-based information content, and, when in proximity to the source beacon and upon receiving the short-range wireless radio signal emitted by and identifying the source beacon, the smart device forming a connection with the source beacon on the wireless communication link;
a mobile application provided to and operating on the smart device to initiate and conclude a vehicle parking session; and
a backend server in communication with the smart device through a wireless communication link, the backend server storing for delivery to the mobile application identities of vehicle parking locations in the vicinity of the smart device, and the mobile application determining whether a vehicle carrying a user accompanied by the smart device and located near the access region that is in operative association with a detected source beacon associated with the vehicle parking locations is in an ingress or egress scenario, performing a ranging procedure to determine whether a distance between the smart device and the detected source beacon or the access region with which the detected source beacon is operatively associated is within a specified amount, and, upon determination that the distance is within the specified amount, forming a state of wireless connectivity between the smart device and the smart beacon to enable activation of a vehicle parking transaction session.
A parking objects or vehicles detection system (60) uses BLE proximity sensing to identify a vehicle (520) with a beacon ID to within about one vehicle length from an access gate (516) of a vehicle parking surface lot or garage. The system performs vision-based parking inventory management (514) by monitoring live vehicle ingress and egress traffic and communicating to a backend server (70) changes in instances of vehicle parking events.
- 1. A parking vehicle detection system, comprising:
a source beacon configured to broadcast its identity by emitting a short-range wireless radio signal from a location of apparatus associated with an access region of a vehicle parking location, and the source beacon configured to establish on a wireless communication link connectivity with a smart device implemented with radio frequency wireless technology to scan for a source beacon in a vicinity of the smart device and to provide location-based information content, and, when in proximity to the source beacon and upon receiving the short-range wireless radio signal emitted by and identifying the source beacon, the smart device forming a connection with the source beacon on the wireless communication link; a mobile application provided to and operating on the smart device to initiate and conclude a vehicle parking session; and a backend server in communication with the smart device through a wireless communication link, the backend server storing for delivery to the mobile application identities of vehicle parking locations in the vicinity of the smart device, and the mobile application determining whether a vehicle carrying a user accompanied by the smart device and located near the access region that is in operative association with a detected source beacon associated with the vehicle parking locations is in an ingress or egress scenario, performing a ranging procedure to determine whether a distance between the smart device and the detected source beacon or the access region with which the detected source beacon is operatively associated is within a specified amount, and, upon determination that the distance is within the specified amount, forming a state of wireless connectivity between the smart device and the smart beacon to enable activation of a vehicle parking transaction session.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14)
© 2018 Citifyd, Inc. A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. 37 CFR § 1.71(d).
This disclosure relates to use of Bluetooth Low-Energy (BLE) wireless personal area network proximity sensing to identify an object having a beacon identifier (ID) and, in particular, to detection of presence of a vehicle transporting an individual carrying a BLE-enabled smart device a short distance from an access gate to a vehicle parking location.
The disclosed parking objects or vehicles detection system uses BLE proximity sensing to identify a vehicle with a beacon ID to within about one vehicle length from an access gate of a vehicle parking surface lot or garage. The system performs vision-based parking inventory management by monitoring live vehicle ingress and egress traffic and communicating to backend servers changes in instances of vehicle parking events.
Additional aspects and advantages will be apparent from the following detailed description of preferred embodiments, which proceeds with reference to the accompanying drawings.
The disclosed parking objects detection system includes an application program in which an algorithm operates to implement proximity detection using BLE to identify an object with a beacon ID to within one vehicle-length from an access gate. In the description below, Citifyd App (sometimes referred to as App) is the name given to a mobile application program operating with a user beacon on a user smart device, and SmartBeacon device is the name given a source beacon 10 (
With reference to
With reference to
Beacon 10 can be attached to many surfaces of and locations in a vehicle parking or mass transit service facility. Beacon 10 can be mounted on top of a pole supported on a floor base, allowing outdoor usage. Beacon 10 can be mounted also on a wall or other fixed structure.
In all device-to-device interactions taking place in the preferred embodiments described, smartphone 36 acts as the primary or Master device informing all secondary or Slave devices of the customer'"'"'s intentions. When smartphone 36 is within range (e.g., 1 m) of beacon 10, the Citifyd App operating on smartphone 36 informs the customer of actions the customer can take. If the customer has no active vehicle parking session, the Citifyd App prompts with messaging asking whether the customer would like to start a session. If the customer is currently in an active session, the Citifyd App prompts the customer to end the session.
Beacon 10, when broadcasting, can be identified from 45 ft (13.7 m) to 230 ft (70 m) away with fair accuracy. The closer smartphone 36 is to beacon 10, the greater the accuracy. Once smartphone 36 detects beacon 10, functional data are automatically sent to and from beacon 10 without the customer'"'"'s permission. The Citifyd App tracks and calculates its distance from beacon 10 and prompts smartphone 36 when a range of 1.0 m or less has been reached. The Citifyd App begins broadcasting its own advertisement data until the distance between beacon 10 and smartphone 36 exceeds 1.0 m. For a case in which beacon 10 is the sole device with Internet connectivity, beacon 10 can be configured to send from a backend server over a wireless communication link through cellular communication network protocols (
With reference to
The following describes the operation of vehicle parking beacon system 60. With reference to
As a customer moves toward the transportation vehicle or parking access gate or attendant, an entry beacon 10e detects customer smartphone 36 at about 30-45 ft. (10.7-13.7 m) and prepares for a connection handshake 104. With reference to
The following diagram outlines the logic state flow of the beacon and customer smartphone for parking ingress, in which there is Internet capability on the customer smartphone.
The following diagram outlines the logic state flow of the beacon and customer smartphone for parking ingress, in which there is an absence of Internet capability on the customer smartphone.
With reference to
Alternatively, as soon as GPS navigation system 74 detects the separation of beacon 10x from customer smartphone 36 with a “stop session” identification code (
The following is a diagram outlining the logic state flow of the beacon and customer smartphone for parking egress.
The process can be repeated multiple times during the day, week, or other set period, and the customer account keeps the tally. At the end of that period, the customer credit card is charged only once.
The system reliance on the cellular or Wi-Fi communication connection at the moment of authorization is eliminated or reduced, and the problem of cellular or Wi-Fi connection delays is removed by (1) performing pre-authorization and account verification before embarking or approaching a parking access gate/attendant and within the Citifyd App creating one or both of an identification code and an authorization screen hidden from the customer and (2) verifying the customer/authorization at the moment of entrance or embarkation through the connection handshake with beacon 10 only.
In the embodiments described below, the Citifyd App and SmartBeacon device are implemented with iBeacon wireless personal area network technology standard equipment, with the source beacon broadcasting regularly.
The Citifyd App operating on a customer'"'"'s or user'"'"'s smart device 36 (preferably, and hereafter sometimes referred to as, smartphone) scans for and monitors a SmartBeacon device. The scanning process is based on the GPS coordinates of the user'"'"'s smartphone 36 and starts scanning for SmartBeacon devices in the vicinity of the user'"'"'s location. The App presents to the user vehicle parking locations available in the vicinity of the user, the actual live inventory of parking spots for each parking location presented through a Live Inventory Management Engine (LIME), and identifies pricing through a Pricing Engine for each parking location. (The term parking location refers generally to a site of a surface or an open parking lot, a parking garage, or other vehicle parking facility.) The LIME and Pricing Engine operate on the backend servers. Once the user selects a parking location by one or both of advance purchasing a parking spot and landing in a parking lot that is Citifyd system-enabled with a SmartBeacon device, the App connects to the SmartBeacon device for that parking lot location. At this point, the App switches from Monitoring to a Ranging Mode for access control.
A SmartBeacon device applies a system of unique beacon ID filtering and a four-fold proximity test (i.e., ranging) to determine the appropriate SmartBeacon device to which smartphone 36 is to connect.
Backend servers 70 first provide to the App a list of unique SmartBeacon device IDs of and information associated with the nearby entrances and exits of the parking garage or lot the user driving a vehicle is entering or exiting. This information includes latitude and longitude of each of the entrances and exits, an indication whether the SmartBeacon device is dedicated for ingress or egress, a unique identification number, and indications of the hardware sensing abilities of the SmartBeacon device (specifically, is a configuration capable of sensing when a vehicle is located over an arming loop embedded in the ground (i.e., in front of an access gate), and is the configuration capable of detecting whether the access gate wired to the SmartBeacon device is physically open).
The App distinguishes whether the user driving the vehicle is in an ingress or egress scenario, and knowing the user'"'"'s location, filters out any unwanted SmartBeacon devices for the current situation and scans for only the SmartBeacon devices it desired. Furthermore, if fewer than the expected number of beacons are detected, the App assumes that another Citifyd system user is currently connected to one of the SmartBeacons and will wait until it determines that the other user has disconnected before completing the scanning process.
The four-fold ranging test procedure after this filtering process includes Tests A, B, C, and D and is as follows.
Tests A and B use iBeacon proximity value measurements, which come in as “Immediate,” “Near,” “Far,” or “Unknown” distances from the SmartBeacon device, in conjunction with the iBeacon identifying minor value, which is an integer between 0-65535 that broadcasts the unique identification number of that iBeacon.
Test A: Does iBeacon report the SmartBeacon device as “Immediate” (<˜0.5 m) or “Near” (<˜3.0 m) at the instantaneous moment of measurement?
Test B: Over an interval of N seconds, were over M number of “Immediate” or “Near” proximities measured? The values of N and M are adjustable (by manual or machine learning) to fine-tune Test B.
Once tests A and B pass, the iBeacon continues ranging to maintain the validity of its results, but also begins BLE communication-enabled device discovery, which scans for connectable BLE devices. Specifically, the iBeacon scans for devices whose broadcast manufacturer data and BLE device name provide additional information as to how to connect to the device. The device-naming convention of a SmartBeacon device looks like “C2<00300”, and this name is parsed for the useful data it contains. The first character identifies the device as a Citifyd system SmartBeacon device and can be used for additional identification information. The second character represents the firmware version number of the SmartBeacon, which can be used to help decide the intricacies of further communication. The third character is the ASCII code representation of the received signal strength indicator (RSSI) signal strength threshold that is required to pass Tests C and D. This character can be further adjusted by backend servers 70-side provided information received in the filtering process in order to facilitate adjustments when not in the field. The final 5 digits are again the unique identification number of the SmartBeacon device.
Test C: Over an interval of N seconds, have there been measured over M number of RSSI values that fall within the RSSI threshold defined in the device name? These N and M values can be independent from those in Test B and are adjustable to fine-tune Test C.
Test D: Over an interval of N seconds, has the Mode taken from the averaged values of all RSSI measurements been within the RSSI threshold on the device? (To take a Mode, the highest and lowest RSSI measurements in the array of RSSI measurements within the N second window are ignored.) This N value is also independently adjustable from the previous tests.
These tests, taken together, smooth out some of the moment-to-moment fluctuations in the RSSI radio signal strength of the BLE devices, thereby adding a greater degree of certainty as to the proximity of the user being within 0-3 meters of the SmartBeacon device and in all likelihood in front of the SmartBeacon device rather than to its side or behind it (the antenna on the SmartBeacon device is slightly directional).
If all four Tests A, B, C, and D are passed, the App assumes that the proximity of the user'"'"'s vehicle is within a few feet of the access gate or SmartBeacon device. At this time, a connection is made between the user'"'"'s smartphone and the SmartBeacon device, and a security handshake (password or shared private key exchange) takes place. Commands can then be sent between the App and the SmartBeacon device.
The SmartBeacon device also has the ability to look for Citifyd system App-enabled mobile devices for detection of the signal strength of the user'"'"'s mobile device. Such ability enables the SmartBeacon devices to broadcast to which mobile device the SmartBeacon device believes is closest, thereby adding, effectively, a Test E.
Once user'"'"'s smartphone 36 and the SmartBeacon device are connected, if the information provided by a SmartBeacon/Gateway indicates that the parking access gate and SmartBeacon device are able to determine whether the user'"'"'s vehicle is located over a magnetic loop detector 490 (
The App also has the ability to customize the SmartBeacon devices to the installed spaces. Because each parking garage inherently has variable signal-to-noise characteristics, the SmartBeacon device can be set to match the environment of installation through a set of proximity calibration set points that compensate for the variabilities for each site of installation.
The SmartBeacon device operating in coordination with integrated vision devices is capable of assigning detected objects to object tables in an object detection module at the point of entry and allow a pricing range for the sizes and types of the objects (such as, for example, trucks, sedans, motorcycles). The objects are reviewed in the vehicle travel lane'"'"'s “field of vision” of the SmartBeacon device, and the pictorial proximities of the objects with respect to one another, as well as with respect to the SmartBeacon device (in multiple lanes of mixed ingress and egress), are determined.
SmartBeacon device 510 with augmented vision system sensors 518 identifies an object 520 (shown and referred to as a vehicle) and the direction of travel of the object 520 requesting access to a parking structure. Vision system sensors 518 placed around detection area 502 also identify the type of object 520. Vision system sensors 518, depending on the site application, may include, for example, one or more of an IR sensor, a camera, a thermal sensor, a magnetometer, a pressure sensitive bar, a sound sensor, a light detection and ranging (LIDAR) sensor, a laser-based depth sensor, a motion sensor, and a beam break technology-based sensor. Vision system sensors 518 use an object detection module 522 as part of a LIME Agent 514 vision system software component to assign a proper identification to the current object 520 in detection area 502.
Vision processing entails the use of the Open Source Computer Vision Library (OpenCV Library) with its deep neural network (DNN) execution environment. OpenCV Library is a cross-platform library of programming functions mainly directed to real-time computer vision. LIME Agent 514 uses DNN associated with Mobile Net-SSD, with site-specific training. Mobile nets are for mobile and embedded vision applications and are based on architecture that uses depthwise separable convolutions to build light weight deep neural networks. OpenCV 3.4.1 is a deep learning module with Mobile Net-SDD for object detection. The source code can be downloaded from https://opencv.org.
This comparison is strengthened over time through an AI Engine 524 operating in backend servers 70 to compensate for environmental variations. AI Engine 524 recognizes changes in ambient environment over time. Such changes include, for example, effects of weather on background illumination, daylight and nighttime conditions, and an oversize vehicle straddling two parking spaces. AI Engine 524 enables performing parametric changes in vision system sensors 518 to compensate for such environmental changes.
If the user has access to software that can interact with SmartBeacon device 510, it in turn can activate an NFC/RFID communicator 526 to cause an access gate controller 528 to open access gate 516, allowing vehicle entry into the parking structure. Regardless of whether SmartBeacon device 510 and NFC/RFID communicator 526 granted access to the user, LIME Agent 514 operating in backend servers 70 and executing instructions on SmartBeacon/Gateway 512 is updated with the type of object 520, inventory count, and other metrics collected in detection area 502 once object 520 passes through access gate 516 and into the parking structure.
The vision-based SmartBeacon device 510 utilizes raster/vector-based algorithms that delineate and distinguish between object 520 and the direction of travel. Sensors 518 and objects 520 are correlated through object detection module 522. For an open surface lot environment, such as that shown in
LIME Agent 514 has the ability to process video streams from vision system sensors 518 and detect vehicles of interest in each video frame it processes. LIME Agent 514 implements AI methodologies such as deep neural network systems with a pretrained detection library for objects of interest such as, for example, cars, trucks, sport utility vehicles, buses, trailers, motorcycles, delivery trucks, and bicycles. It can use video frame subtraction methodologies to identify moving objects. The detection system of LIME Agent 514 has the capability to decipher vehicle type, size, location, license plate, and other unique identifiers, such as make and model, color, and vehicle beacon IDs. LIME Agent 514 software includes a kernelized correlation filter (KCF) tracking framework, a core component of which is a discriminative classifier that distinguishes between a target and surrounding environment. The source code can be downloaded from https://github.com/joaofaro/KCF/cpp. LIME agent 514 implemented with a KCF tracking architecture has the ability to track multiple detected vehicles within predefined localized region(s) of interest and calculate relative motion vectors for each vehicle. The tracking module of LIME Agent 514 uses a combination of movement prediction and actual image characteristics of each vehicle for tracking. This system has the ability to resolve for stagnant cars within the region of interest and track partly occluded objects. The system activates vehicle processing by locking on to each vehicle within the region of interest and thereafter computes net movement to ingress or egress after the vehicle is no longer within the region of interest. LIME Agent 514 communicates information relating to egress and ingress vehicle movements through a secure representational state transfer (REST) application-programming interface (API) to backend servers 70. LIME Agent 514 has an ability to change vision system sensor settings to compensate for environmental variations. LIME Agent 514 is also implemented with redundant processing in cooperation with a thermal sensor to provide an ability to compensate for environmental conditions in which a typical RGB camera is impaired (such as snow, extremely low light, or fog) by switching to different sensors as appropriate for the given environment (for example, switching to LIDAR, a thermal sensor, or an IR sensor). This would entail switching to, for example, a thermal sensor and loading an associated neural network training file for thermal images.
LIME Agent 514 provides public and private connectivity for identifying live inventory data for all parking lots in the Citifyd system, as well as an independent channel for LIME Agent 514 for entities outside of the Citifyd system. The algorithm compensates for weather variation, along with other environmental factors, by recalibration of reference points through AI Engine 524 with various live feed and sensor inputs. Background reference points are compared from time to time as a result of sensors input data, as well as other live feeds (e.g., weather pattern). Random reference comparison points and times are determined through a self-learning AI Engine 524 (cluster of vision system sensors 518). These vision system sensors 518 are used to activate and unleash self-learning environmental parameters and patterns that can improve the calibration of reference points. Vision system sensors 518 that collect the input for AI Engine 524 may include RF heat detectors in combination with a depth sensing vision system, stereo camera clusters with variable depth of field adjustment per installation, along with an RGB camera system. The RGB camera system is connected to the Citifyd SmartBeacon system, which is edge processor-enabled and extrapolates and interpolates one or both of object movements and stationary objects in larger areas.
The combination of multiple SmartBeacon sensors data and RGB remote camera systems distinguishes between objects, the heat pattern from objects, as well as the depth of an object and thereby continuously enhances AI Engine 524 for the specific installation and site specific data. AI Engine 524 has the ability to coordinate and feed reference geographic based data for other nearby (zone) locations.
A stereo camera system operating as a sensor can detect patterns of movements of objects in entry and exit vehicle travel lanes. Stereo camera sensors can be a combination of RGB and RF, along with other depth sensing signals.
RFID (active) communicators are enabled at SmartBeacon device 510 for enabling access gate control without direct wiring to the access gate mechanism. This bypasses the direct connection and, as an option, coordinates inventory counts through the vision system and LIME Agent 514 for gated parking garages.
A Lot Navigator application operating as part of LIME Agent 514 in backend servers 70 is a localized navigation option for larger parking garages that need to direct incoming users to an available open parking space in the parking garage. The Lot Navigator option directs a user in a vehicle to a specific available spot through a map overlay of the lot in the Lot Navigator. The communication for the Lot Navigator is coordinated between multiple clusters of SmartBeacon devices 510 in conjunction with GPS coordinates that connect through thread communication protocol to SmartBeacon/Gateway 512 and backend servers 70. An example of the protocol is a WiFi network or a clustered beacon network using IEEE 802.15.4 protocols. The use of multiple pole-mounted vision system sensors facilitates an ability to communicate a full lot composite image to backend servers 70 for the purpose of open spot discovery. Lot Navigator uses and references comparative images from the pole-mounted vision system sensors 518 to the reference background image to identify an available spot through LIME Agent 514. LIME Agent 514 has the ability to locally log to backend servers 70 all communications, including false positive images and negative images, locally or to backend servers 70 as necessary. False positive images are objects that are identified erroneously as vehicles. Negative images are vehicles that were missed by object detection module 522. The vectorization computation and comparison is improved through AI Engine 524 on a regular basis to compensate for environment variations.
The Lot Navigator uses the following equipment to direct incoming users to an available parking space. Pole-mounted vision system sensors 518 report changes in inventory by reporting objects that result in change against a reference background image. This comparison is strengthened over time through AI Engine 524 to compensate for environmental variations. Communication between vision system sensors 518 and object detection module 522 happens through SmartBeacon devices 510 clustered around the parking structure. SmartBeacon devices 510 act as a communication conduit between vision system sensors 518 and SmartBeacon/Gateway 512 and backend servers 70 through thread communication protocol (e.g., WiFi, IEEE 802.15.4 protocols, and LTE). Changes in parking space state—occupied or empty—are communicated through LIME Agent 514 to the Lot Navigator by updating a map overlay of the parking structure. This information is passed along to the user through an application that will guide the user to an empty, assigned parking space.
To facilitate description of an optional triangulation technique that is especially useful for vehicle detection at an ingress or egress access gate of a system not implemented with vision sensors,
A SmartBeacon device 510 (referred to as a gate beacon, GB) positioned near access gate 516 is separated by a known distance from a standard BLE signal-emitting source beacon (referred to as a lane beacon, LB) positioned along the travel lane. Gate beacon GB and lane beacon LB are preferably separated by 14 ft. (4.3 m), which is about two vehicle lengths. Gate beacon GB and lane beacon LB emit radio signals of substantially the same strength. Gate beacon GB and lane beacon LB cooperate to eliminate, irrespective of the number of passengers occupying a vehicle in the queue, the effect of signal absorption, which affects the RSSI radio signal strength measured by the smart device.
The RSSI ratios indicate the order of vehicles 5701, 5702, and 5703 in the queue, with RSSI ratio=1 representing a tipping point at which the vehicles closest to and farthest from access gate 516 exhibit the largest fraction >1 and the smallest fraction <1, respectively. The App operating on the smart device transmits the RSSI ratio information to gate beacon GB, which either causes the App carried by an occupant in vehicle 5701 to produce an on-screen button for access gate control or causes access gate controller 528 to open access gate 516 for vehicle 5701 to pass through.
The triangulation technique for vehicle detection is configured for use in multi-lane vehicle parking facilities, which often install access gate structures on a common lane divider. This technique enables opening the correct access gate associated with a gate beacon for a vehicle traveling in the lane associated with and positioned next to the access gate.
Upon completion of the communication link setup, vehicle detection is made ready to be performed in accordance with the following procedure. Process block 710 represents continual reading of all configured vision system sensors 518, and process block 712 represents cropping, filtering, and scaling of their inputs. Process block 714 represents detection of vehicles within the defined regions of interest. Stream video is received and processed for the regions of interest. Restricting the segment of the video to be analyzed is defined in the setup represented by process block 702. Use of variable selectable frame rates allows for efficient processing. Once a vehicle is detected, object detection module 522 verifies that it meets initial criteria for objects of interest. The resulting vehicles are verified and classified through object detection module 522 using a deep neural network (DNN). The deep neural network uses an initial training file and is augmented with site-specific training parameters. Process block 716 represents adding newly detected vehicles to a tracker list. Once verified, the vehicle is passed on to a tracking module in LIME Agent 514 and verified to be the same in predefined or random intervals through the deep neural network. The vehicles are tracked with the customized areas of interest in a manner that, if occluded, will continue to be tracked through a predicted path within a region of interest and re-verified through the deep neural network. Decision block 718 represents an inquiry whether any of the vehicles have crossed out of the region of interest. If the answer is NO, decision block 718 represents a return to the vehicle detection processing operation of process block 710, where all of the configured vision system sensors 518 are read. Travel movement within the region of interest of vehicles is vectorized to determine the distance and direction of travel with respect to the boundaries of the region of interest. If the answer is YES, decision block 718 directs processing to process block 720, where the distance and direction of travel is computed for each vehicle that crossed out of the region of interest. Process block 722 represents a determination of movement of each vehicle as either an ingress or egress event. Process block 724 represents transmission of ingress or egress event data to LIME Agent 514 operating in backend servers 70. Once it exits an area of interest, a vehicle is removed from the tracking module and an image of the local vehicle is stored for scheduled use to update the training file for the parking location (e.g., parking lot). Process block 724 represents a return to the vehicle detection processing operation of process block 710, where all of the configured vision system sensors 518 are read.
It will be obvious to those having skill in the art that many changes may be made to the details of the above-described embodiments without departing from the underlying principles of the disclosure. The scope of the invention, should, therefore, be determined only by the following claims.