Freeside:5:Documentation:Tower mapping

This document describes the tower coverage features on the 5.x branch, which has not yet been released. The tower coverage system works, but further development is planned. Red text is for notes on future improvements.

Tower coverage calculation

The goals of tower coverage calculation are:

• Mapping: estimating a tower's coverage area, for marketing, deployment planning, and regulatory reporting.
• Prequalification: estimating the quality of signal available at a customer site, prior to doing an on-site survey.

These require the same calculation: evaluating a signal strength function at a geographic point, using the tower's transmitter characteristics and a terrain map, and comparing to a minimum specified signal strength:

<math>{P}_{\mathit{RX},\mathit{\min }}\le {P}_{\mathit{TX}}+{G}_{\mathit{TX}}-{L}_{\mathit{path}}+{G}_{\mathit{RX}}</math>

where P_TX is the transmitter power, G_TX and G_RX are the transmitter and receiver antenna gains, and L_path is the path loss. For an unobstructed path length d and a frequency f , the path loss is:

<math>{L}_{\mathit{path}}=20{\log }_{10}4\mathrm{\pi }df/c\approx 36.6\mathrm{dB}+20{\log }_{10}f-20{\log }_{10}d</math>

where f is in MHz and d is in miles. For a partially obstructed path, this is more complicated; note that the path is significantly wider than the physical antenna due to diffraction. Consult your radio engineer for details.

Tower configuration is in Configuration / Services / Wireless broadband / Towers.

Tower/sector schema

A record in “tower” represents a site where broadband equipment is installed, including:

• Latitude and longitude
• Tower elevation

A record in “tower_sector” represents a transmitter and antenna installed at a tower. Normally, each wireless broadband service is linked to a sector. For each sector, Freeside tracks:

• IP address
• Height above ground level in feet
• Antenna direction (azimuth and “downtilt”, i.e. elevation)
• Beam width (horizontal and vertical)
• Minimum signal margin
• Power output, in dBm
• Line loss (in cables and connectors), in dB
• Antenna gain, in dB
• “Low-quality” and “high-quality” signal margins. Note that this represents maximum path loss in dB, not signal strength as such. For example, if db_low = 100, then for “low quality” service to be available, P_TX + G_TX + G_RX − P_RX,min must be at least 100 dB. It is normal for db_low to be larger than db_high. Possibly these should both be export parameters rather than being set per sector. Allowing more than two levels might also be useful.
• “image”, a PNG graphic of the coverage map
• The geographic bounds of the image: west, south, east, and north.
• hardware_typenum, a foreign key to the hardware_type table.
• title, a string field for storing an external ID.

Coverage maps are stored in “sector_coverage”. A separate record is created for each minimum signal level; the intent is that they can be used as map layers. Each one contains:

• sectornum (tower_sector foreign key)
• db_loss, the level of signal loss represented by this map layer
• geometry, a GeoJSON polygon for the region where the signal loss is at most db_loss.

If you move the database into PostGIS, with tower_sector.image becoming a raster and sector_coverage.geometry becoming a polygon layer, then a simple prequalification looks like:

dbh->selectall_arrayref('SELECT sectornum, db_loss FROM sector_coverage WHERE ST_Contains(sector_coverage.geometry, ST_Point(?, ?)) AND db_loss > ? ORDER BY db_loss DESC', {}, $longitude, $latitude, $max_loss)

Tower mapping with Splat: process_generate_coverage

Inserting a new tower_sector will queue a process_generate_coverage job for the sector. Editing the height, antenna position or beamwidth, or signal loss levels will also queue this job (from the replace() method). Changing a tower's coordinates will queue process_generate_coverage() for all sectors on the tower.

CloudRF replaces most of this; supposedly they will provide an ESRI shapefile, which we can pull into OGR and convert into a stack of sector_coverage records.

process_generate_coverage makes a temporary directory and creates a Map::Splat object, passing the tower coordinates, antenna height, azimuth and downtilt, vertical and horizontal beamwidth, and the two signal levels to calculate (db_low and db_high). It calls the calculate() method to start generating the map.

Splat is a standalone program that operates on a set of text files. Map::Splat, our wrapper library for it, writes the input files into the temp directory:

• The tower location file (QTH). This only contains the tower coordinates and antenna height.
• The model parameter file (LRP). This contains the frequency, and several static parameters (in particular, the model's safety margin may need to be adjustable).
• The loss contour file (LCF). This maps signal loss levels (dB) to 24-bit colors. Splat always uses white for areas with “no signal” (that is, above the highest specified loss level) and won't accept a color list containing black, or with more than 32 colors. Map::Splat accepts a list of desired signal levels, and spreads them out in the red channel from #04 to #FC. Here, we only have two signal levels, so they'll be at the endpoints.
• The antenna profiles in the azimuth plane (AZ) and the elevation plane (EL). These start with the antenna orientation (which is applied as a rotation to the entire data set) and then list several angular positions and field strengths (relative to the on-axis field strength). Currently we use a simple approximation where the power is 1.0 within the beamwidth, falls off by 3 dB at 1.7 * the beamwidth, and is effectively zero at twice the beamwidth. It would not be hard to modify Map::Splat to accept an .ANT file, since that's just a list of field strength at each one-degree increment.

After that, Map::Splat ensures that we have the map data for the region of interest. It uses the data from the NASA SRTM survey, which is available from the U.S. Geological Survey. Each elevation file covers a 1 degree tile; the downloader fetches maps covering a 5 × 5 square centered on the transmitter position. (There are no map files for ocean tiles, so if the tower is near the coast, the downloader will simply fail to download some maps. Presumably there are no customers in that area, and no obstacles either.) The maps get unpacked (from zip files), converted with the “srtm2sdf” tool, and stored in /home/freeside/.splat; future Splat calculations will reuse them rather than re-downloading everything.

There is a higher-resolution version of the SRTM data available, but it's organized on the USGS servers in a way that makes it a little more complicated to download, so we don't use it yet. If you do use it at some point, remember to change the “srtm2sdf” and “splat” command lines to “srtm2sdf-hd” and “splat-hd”.

Map::Splat runs Splat with a command line that tells it where to find all those files, the maximum loss level to consider (equal to db_low), the maximum coverage radius, and an assumed height for each receiver (both of which should be export parameters). Splat's output includes:

• A PPM file containing the image, which will be three colors (white and two shades of red). We pull this into ImageMagick and store it in the “image” property of the Map::Splat object.
• A KML file containing a “LatLonBox” element, which gives the bounding box represented by the PPM. We read this with LibXML, read the coordinates of the bounding box, and store them in the “box” property.

Back in process_generate_coverage, the bounding box fields in tower_sector (south, west, east, north) get filled in from the Map::Splat box, and the png() method is called to convert the image to a usable form (it turns the white area transparent so that the image could be used as a ground overlay, though we don't currently do that).

Then we call Map::Splat::polygonize_json(). The first thing this does is open the image as a GDAL raster data set. It sets the spatial reference (WGS84, GPS coordinates) and the transform matrix (so that the image pixels map onto the correct area of real geography). The output of Splat often contains a lot of noise around the edges (where the signal is close to threshold) so it applies a sieve filter that removes blobs smaller than 200 pixels, then calls GDAL's Polygonize() function to turn the red channel into an OGR in-memory polygon layer. Each area of uniform color is a polygon, tagged with the shade of red that it was.

polygonize_json() goes through the list of known colors (we currently have two, db_low and db_high), calls SetAttributeFilter to select all polygons that color, and makes a single polygon that's the union of them. It outputs a GeoJSON feature collection, containing a feature for each signal level, marked with a “level” property. process_generate_coverage takes each of those features and writes its level and geometry (still as GeoJSON) to a sector_coverage record.

The tower map

This exists in Freeside 4 but is greatly improved in version 5.

search/tower-map.html is a Google Map displaying three types of features: towers, services, and coverages. Towers and services are point markers, and use a styling function that merges a base style (“baseMarkerStyle”) with the “style” property of each feature, if it has one. This allows style properties (such as marker color) to be overridden per-feature.

Point markers have click and double-click events. “clickHandler” pops up an info window connected to the feature location. If the feature has a “content” property, then that's the HTML to put in the info window. If it has a “url” property, then we make a jQuery.ajax request for that URL, and load the result into the info window. “dblClickHandler” does exactly the same thing, plus it centers the map on the marker and zooms in.

Towers are all on a single data layer (“tower_data”) which is created within tower-map.html as the @features array. They're styled as markers with a custom “antenna” icon. The info window content is the tower's name, a link to edit/tower.html, and a list of sectors, with the number of up and down services on each sector. If there's a [#Tower exports tower export] with the export_links hook, the links will also be shown here.

The tower info window also has checkboxes to toggle the tower's services and coverages.

Services are in a separate data layer for each tower (“tower_svc_data[towernum]”). By default all of these layers are hidden; they're turned on from the checkbox by calling setMap(). Each layer is populated by calling loadGeoJson() on search/svc_broadband-json.cgi, passing the towernum. Each broadband service is shown as a point marker with color according to its up/down status (red, green, or gray). The info window loads dynamically from view/svc_broadband-popup.html, and shows the customer name and status (small_custview), service label, and the last known service status and latency.

Coverages are also in a separate layer for each tower (“tower_coverage_data[towernum]”) and hidden by default. They're polygon features. These are populated, in the same way as service layers, from misc/sector_coverage-json.cgi, which pulls their geometry from sector_coverage. The coverage styling function works like the one for point markers, but sets the fill opacity based on whether the coverage feature has the “low” or “high” property. Coverage layers have no click handlers.

Finally, there's a search box. This uses the Google Places API, is attached at the top right as a map control, and lets the user search for addresses or locations near the current display area, in the usual way of Google Maps. What should happen is that clicking on a point that's not a tower or service location should query the sector_coverage table for coverage polygons containing that point, and pop up a window showing which tower/sector has the best estimated signal strength there.

Tower exports

Currently only one of these exists (the one for TowerCoverage.com). The code from tower_sector to deal with Splat should be moved into an export also. CloudRF support should be done as yet another export.

These aren't “exports” in the conventional sense of service-provisioning interfaces; the export hooks are invoked from FS::tower_sector's insert(), replace(), and delete() methods. All part_exports that have “tower_sector” in their 'svc' array will be invoked. The export_insert/replace/delete hooks take the tower_sector as an argument. The insert and replace hooks are called after inserting/replacing the record (like for service exports), so if the export needs to modify the tower_sector record, it should set “local $FS::tower_sector::noexport_hack = 1” to avoid recursion.

There's also an export_links hook, which returns optional links to be included in the tower info popup (see above). This takes the tower_sector as an argument.

The get_antenna_types() hook was created specifically for TowerCoverage, and is used by the Edit Tower UI (specifically in elements/tr-tower_sectors.html). It should return an ordered hashref (Tie::IxHash) of hardware_type.typenum values and labels allowed in the “antenna type” property of a sector. (This is the tower antenna, not the customer's antenna.)

Finally, there are “qual” and “qual_result” hooks for prequalifying service at a location. These both take the FS::qual object as an argument, which gives access to the proposed service location (via qual.locationnum) and sector it's trying to link to (qual.sectornum). qual() is invoked from FS::qual::insert, on whichever export was selected for qualifying service. It needs to return a hashref containing:

• status: 'Q' for qualified, 'D' for declined. This is the most important.
• vendor_qual_id: A transaction number for this qualification, if there is one.
• options: A hashref of additional data describing the qualification. This will be inserted into FS::qual_option.

qual_result() is called to interpret the qualification for display (probably by looking at its options). It returns a hashref containing “pkglist”, a hash of pkgparts and labels for the package definitions that could be sold at that location.

This feature is incomplete: the UI for creating quals doesn't have a way to choose a sector, and the TowerCoverage link path implementation doesn't yet do anything useful with the results.

The TowerCoverage export

TowerCoverage.com provides services for both coverage mapping and link path calculation. FS::part_export::tower_towercoverage is an interface to this service.

TowerCoverage requires the sector antenna to use a known configuration. A list of these configurations is hardcoded in the tower_towercoverage module; the first time an export of this type is created, we create a hardware class named “TowerCoverage.com antenna”, and insert the entire list into hardware_type under this class. hardware_type.title contains the TowerCoverage ID of the antenna type. The export has a get_antenna_types() method to list the allowed types.

Certain coverage parameters need to be configured on the export rather than per sector: the client antenna properties (height, gain, and cable loss), maximum range, signal strength thresholds, and frequency band. Splat and/or CloudRF coverage mapping should store these things as export options also.

The export_insert hook does most of the work. It converts the tower_sector properties to the form expected by the API (including metric units on everything, coercing some data types, and naming everything correctly) and creates an insert_coverage queue job. (export_replace does exactly the same thing.)

insert_coverage runs from the queue and sends all its arguments in a POST request to the CoverageAPI endpoint. What it gets back is an XML document containing the path to the rendered map. We don't do anything with the map itself (since we're not logged into the TowerCoverage website). What we do is parse the map file name to get TowerCoverage's ID for this coverage map, and store that in tower_sector.title. export_links() then uses that ID to return a link to their map.

qual() uses the Link Path API. It builds an argument list similar to the one for coverage mapping (where “Site1” is the tower and “Site2” is the client location), sends it to the LinkPathAPI endpoint, and stores everything that comes back in “options”. This is incomplete.

Station monitoring

The tower map shows connected customers as green and unreachable customers as red. How does it know?

“freeside-pingd” is a daemon that attempts to scan all IP addresses listed in svc_broadband and tower_sector records. The pingd-interval config enables this daemon and sets the polling frequency (or you can set it at the command line with “-i”.)

The addr_status table stores the last known ping status of an address. During each scanning pass, the daemon fetches a list of all addresses that haven't been scanned in (interval) seconds (or at all). It then passes each address to a separate process via FS::Daemon::daemon_fork(), which rate-limits to a sensible number (defined here as “ten”) of child processes. Each child process runs the scan() function.

scan() finds the addr_status record for the address, if there is one, then uses Net::Ping to do a TCP probe of the address, with a fixed 5-second timeout. It updates (or inserts) the addr_status with the ping result (up or down), latency, and timestamp. This could be improved by sending multiple pings to estimate packet loss, sending pings with payload data, keeping more than the most recent result, etc.

After recording the ping result, the child process exits. If there are more addresses in the queue, daemon_fork() will start scanning the next one. After all addresses have been scanned, the daemon determines how long to wait before doing another scan. It calculates the expiration time (timestamp + interval) for each addr_status record and finds the lowest expiration time that's in the future. The daemon sleeps until that time, or at most (interval) seconds (since all scan records will be expired by then).

FS::svc_IP_Mixin (services with IP addresses assigned) implements addr_status() (returns the service's addr_status record) and addr_status_color() (returns the CSS color name that goes with the status). search/svc_broadband-json.cgi uses this to set up the marker style. view/svc_broadband-popup.html additionally shows the latency and timestamp from the status record.