Category Archives: Linux

UniTrunker Automated File Merge

I’ve put together some tools to analyze UniTrunker data.  The details can be found here: UniTrunker file merge.

These tools are great, but fairly labor-intensive. So in an effort to automate the process and spend more time viewing the files and less time processing them, I recompiled the merging program and wrote a Linux script to do the work.

This processing could be done on a Windows machine using scripts and tools there, but I decided that I’d rather work in Linux for file viewing and processing.  This leaves my Windows machines to run  UniTrunker and create the logfiles, and my Linux machine to copy that logfiles and do the processing.

There are a few manual tasks that I had to perform initially.  These include:

1. Export the system.xml file from UniTrunker and run either msxsl.exe or xslproc to extract group and user information into a .txt file.
2. Set up the folders and file hierarchy in advance
3. Mount the shared file location of the UniTrunker-logfile on the Windows computer

Once these tasks were done, I created a script for each system I was going to export and a cron job to run them each night shortly after midnight after UniTrunker had created and zipped the daily logfiles.

A link to the Linux version of UTC to merge logfiles with user and group data can be found here: UTC file

An example of the script I created:

datestring=$(date +%Y%m%d)
yesterday=$(date --date="yesterday" +%Y%m%d)
yearmonth=$(date --date="yesterday" +%Y%m)
filestring=UniTrunker-$yesterday.log.Z #Name of logfile created by UniTrunker
filestring2=UT$yesterday.log.Z #Shorten filename for ease
#Copy logfile from Windows machine to Linux machine for unzipping:
cp /mnt/nameofwindows/computerRunning/unitrunker/S00000031/$filestring ~/Documents/UTSort/raw/$filestring2
gunzip /home/parallels/Documents/UTSort/raw/UT$yesterday.log.Z #unzip logfile
#Move unzipped file to log storage location:
mv /home/parallels/Documents/UTSort/raw/UT$yesterday.log /home/parallels/Documents/UTSort/$yearmonth/Davis/UT$yesterday.log
#Run UTC sort/merging program to add user names and group names to logfile in addition to UID and GID numbers
#Note: Prior to running this script, it is necessary to export the System.xml file in UT and then run MSXSL.exe in Windows
#or xsltproc in Linux to create users.txt and a groups.txt files to be used by the UTC sort/merging program
UTC /your/path/here/$yearmonth/SystemName/UT$yesterday.log /your/path/here/users.txt /your/path/here/groups.txt $processfile
#Below are examples of searches/sorts that I do on the logfiles to sort via talkgroups or users or find unknowns.
#Place your talkgroup IDs or search terms in the field after grep and create folders and change filenames to be relevant
#to your local groups and users:
more $processfile | grep ,X | grep -v XP >${processpath}/WXPD/WXPD$yesterday.txt #WX PD
more $processfile | grep 10816 >${processpath}/Ops5/Ops5$yesterday.txt #Davis County Ops 5
more $processfile | grep 10848 >${processpath}/Ops6/Ops6$yesterday.txt #Davis County Ops 6
more $processfile | grep 8544 >${processpath}/BountPD1/BountPD1$yesterday.txt #Bountiful PD 1
more $processfile | grep '9440\|9408' >${processpath}/DavisLaw/DavisLaw$yesterday.txt #Davis Law 1 and 2
more $processfile | grep 9952 >${processpath}/DavisService/DavisService$yesterday.txt #Davis Service
more $processfile | grep 17600 >${processpath}/NRegional/NRegional$yesterday.txt #Northern Regional
more $processfile | grep 18464 >${processpath}/Event2/Event2$yesterday.txt #Event 2
more $processfile | grep  ', ,G' >>${processpath}/Unknown/DavisUnknown$yearmonth.txt  #unknown users
more $processfile | grep 'G,.*, ,'>>${processpath}/Unknown/DavisUnknownTGID$yearmonth.txt #unknown talkgroups
sort -u -t, -k8,8 ${processpath}/Unknown/SLCUnknownTGID$yearmonth.txt | cut -d, -f8 | sort -n>${processpath}/Unknown/SLCUnknownTGIDSorted$yearmonth.txt #List unknown TGID numbers
sort -u -t, -k5,5 ${processpath}/Unknown/SLCUnknownUID$yearmonth.txt | cut -d, -f5 |sort -n>${processpath}/Unknown/SLCUnknownUIDSorted$yearmonth.txt #List unknown UID numbers

If you’re interested in compiling the UTC program yourself, email me and I’ll get you the source code.

UniTrunker Logfile Merging

The program UniTrunker uses XML files to output data pertaining to Systems, Groups, Users, Sites, etc.  It stores call log files in Unicode format with just User numbers and Group numbers, but not text labels associated with them. I wrote a C program to merge the User and Group data with their respective numbers in the call log file.  This is a multiple step process to extract information from the System.xml file into text files for Users, Groups, and Sites; followed by converting those Unicode files to ascii files for handling with the C code; and then running them to created a merged logfile with both User and Group lables appended.  For this I created a batch file to complete the process with 2 inputs from the user during the process.

Prior to running the batch file, all of the needed files need to be placed in the same directory.  These include:

msxsl.exe (which can be downloaded from Microsoft’s website)
UTC.exe (the program file which I wrote to perform the merge)
UTMerge.bat (the batch file which calls all the programs and converts files to correct filetypes)
UTxsl.bat(the batch file which calls the msxsl.exe to apply the xsl stylesheets)
UTlog.bat(the batch file which calls UTC.exe program to create a merged log file)
System.xml (this file is exported by going to the systems tab in UniTrunker and exporting the desired system)
users.xsl, groups.xsl, sites.xsl which I have created/modified based on the example from the UT website
UniTrunker-********.LOG file (created daily by UniTrunker and archived to a gzip file)
      -Note this file needs to be unzipped in order for the batch file to work

Once all the files are in the same directory, simply change directories to the location of the files, and then type “utmerge” to call the batch file.

The user will be prompted to enter the name of the .xml file.  Case doesn’t matter, but be sure to type it correctly.  After a few seconds of processing the user will be prompted for the date of the logfile to be merged.  After a few more seconds a text file with the merged data will be created in the same directory.

A zip file of the required files can be found here:

UniTrunker Merge Program files

Download and extract the file to an accessible location.  Then extract a system.xml file from UniTrunker.  Copy or move a .LOG file from UniTrunker’s logfiles to this same directory.  You can go to Run…and type CMD to open a DOS window.  Then change directories to the location of your UTM files and type utm to run the batch file. 

An easier way is to use windows explorer to browse to the directory and double click on UTM.bat. The program will run in a popped-up DOS window.  Carefully type in the information required when prompted, and VOILA, a merged log file will be created!


I created 2 additional .bat files:UTxsl and UTlog. The UTmerge file can still be used successfully.  However, each time it runs it creates the same users.txt, group.txt, and sites.txt files.  This does no harm, but is redundant and takes time.  I spilt the processes into 2 batch files.  The first to be run should be UTxsl.  This will apply the xsl stylesheet once and create the necessary .txt files.  After these are created, there is no need to run this batch file again unless changes are made in UniTrunker to your system files and subsequently exported.

The second batch file to be run is UTlog.  This simply prompts for the date of the log file you would like to merge with the users and groups files.  It can be run multiple times on logs for different dates without recreating the users.txt and groups.txt files.

UPDATE 2 (February 23, 2013)

After looking for a way to further process the merged log files, I realized the Linux grep command was perfect for looking through logs to isolate individual users or groups or both.  This means I either have to run a grep program on my Windows computer, or move the files to my Linux computer to further process them.  It is a little cumbersome to have to move files to a different platform, but the power to sort the data makes it worth doing.

The power of the the grep command in dissecting the large amount of information in one days’ log file is impressive.  For example, after a recent fire department response to a building fire, I sorted the log files based on the fire departments operations channel to see just the calls for the fire separate from the 1000’s of other calls on the TRS for the day.

For a better explanation of the grep command see:

A sample output from a TG sort using the command:

~$ more UT20130220merged.txt | grep 10848 


20130220210159,2,Call,I,32092,Engine 85,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),488, - 
20130220210242,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),525, - 
20130220210247,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),497, - 
20130220210256,2,Call,I,32031,Engine 85,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),498, - 
20130220210300,2,Call,I,32014,Ambulance 83,G,10848,Davis Ops 6 - South Davis Metro Fire (Bountiful fire response),498, -

MMDVM DMR Repeater

After a year of assembling, programming, troubleshooting and tweaking, we recently put up a MMDVM (Multi-mode Digital Voice Modem) repeater for the south end of Davis County, UT. Last year I entered the world of digital voice with a purchase of an MD-380 digital DMR handheld radio. Since our area lacked good repeater coverage, I decided to try to build a repeater to help fill in some of the coverage gaps.

One of the hardest parts of putting a repeater on the air was simply getting a frequency coordination for the 70 cm band. The key pieces of the repeater are the actual MMDVM board that sits on an Arduino Due board. The Due is connected to a Raspberry Pi which runs G4KLX’s MMDVMHost software that interfaces with the MMDVM software on the DUE  and connects the repeater to the Brandmeister DMR network. The radios are two Motorola CDM-1550LS+ radios connected via duplexer to a Diamond X-50A antenna.

Below are some pictures of the repeater:

Python Script to Update Icecast Metadata With METAR and TAF Data

Python Script to Update Icecast Aviation Audio Feeds with METAR and TAF Data

Here are two scripts to retrieve airport METAR and TAF data from the National Weather Service website and then send this data to an Icecast server to update stream metadata.

You can get the scripts here (use wget from a Linux command line):

Save them with whatever names you’d like (probably imetar[AirportName].py). In each script enter the server/stream information and modify the script to download your desired station name. Then run a cron job every hour (or whatever interval you choose) to update your stream metadata with the airport METAR and TAF data.

Bash Script to Download TAF Data From NWS

Linux Bash Script to Download TAF Data from the NWS to Display in the Console


Example: $taf kslc

Save the following in a file called taf:

station=$(echo "$1" | tr '[:lower:]' '[:upper:]')
wget -q -O /home/$USER/taftmp$station.TXT
echo " "
more /home/$USER/taftmp
echo " "
rm /home/$USER/taftmp