It has been a long road starting in 2017 when we purchased the new Kenwood Gen II NEXEDGE system to upgrade our radio system. We completed the last of the changeovers with customers and the tower sites on Wednesday, June 9, 2021. We decided to purchase the Gen II system because it offered significantly enhanced features that would provide numerous benefits to our customers who use the system and provide us as system owners with additional flexibility and management benefits.
We ordered the Gen II system at the IWCE show in Las Vegas. We were the first company to place an order for the new system in the world. When we purchased the Gen II system, we did not expect the extended delays obtaining delivery which were caused by Kenwood finding numerous “bugs” in the operating system which delayed the ultimate delivery of the system. They also did not anticipate Dell discontinuing the server which they used to develop the system and needed time to certify that the system would operate properly with another server. Once they resolved those issues, they delivered the first system to another company to which we objected. Kenwood claimed that they were able to deliver the other system first because it was a simpler configuration, but it did not sit well with MRA. About 4-8 weeks later, MRA finally received delivery of our new Gen II system.
The system was very complex and required time to study it, learn it and install it. At each of the tower sites where we had 10 or more channels, we had to install a site server. The server is a one rack unit computer server which was 19” rack mount if you had a computer rack. The hardware server was 48” deep which created mounting challenges at our tower sites. Although we own several of our tower sites, the ones that we do not own contain lease requirements and restrictions. Some of these restrictions prevented us from installing a server cabinet. The cost of a server cabinet is also an issue as they cost a considerable amount of money and the physical size of a server cabinet creates problems with finding space in many buildings as well as the additional monthly cost to have the huge cabinet in the building. Therefore, it became necessary to become inventive to come up with ways to install the servers at the different tower sites.
It all started with an upgrade to our existing hardware at all of our tower sites. We had identified several areas of weakness in our system design, so we set out to upgrade the hardware to resolve those issues which included:
- Power distribution did not allow for remote control or remote monitoring. We standardized on using a power distribution product from ICT which required us to change out a significant amount of cabling at every site. The panel cost under $1000, but there was a significant cost to make the cabling that connects to the power supplies, battery systems, repeaters, routers, amplifiers, etc. This allows us to remotely reboot the equipment from our office as well as remote trouble shooting of system problems.
- The system requires battery backup at all sites and had it at most locations to prevent power transients from causing a system reboot and to provide for continued network operation during times of power outages. We needed to insure that we had batteries at every location and we needed to increase the capacity of the battery system to allow for a longer runtime when power outages would occur.
- We had UPS systems at many of our sites to power our routers, microwave links and other equipment that did not run from 12v directly, but the UPS would stop working long before the main batteries were depleted, so the site would cease working once the UPS quit. This required a redesign so that the site would keep working as long as there was any battery capacity.
- We believed that the Adtran routers that we originally purchased were capable of handing the advanced features that we needed when we upgraded the network. We found out that we were wrong because the routers did not work properly with those features which were essential for our network upgrade. Therefore, we had to replace the routers at all of our tower sites.
- We installed new smart battery chargers to keep from “cooking” the batteries at the tower sites. The older battery chargers were “dumb” chargers that would overcharge and ruin the batteries. The new battery chargers had smart sensing which would stop charging the batteries when they were fully charged.
This hardware upgrade program took about 2 years to implement. Upgrading one tower site is typically not that complicated a task, but upgrading over 80 tower sites requires a lot of planning. We would have meetings about once per month to discuss the issues that were at hand during the process and what we could do to minimize the impact to the customers. This caused us to constantly adjust our plans to achieve greater efficiency and lower impact to the customers.
When we would receive the Gen II upgrade equipment at the office, it could take 3-5 days depending upon our other workload taking care of customer’s needs to get the equipment ready to go to the tower site to install. Three of our tower sites are less than a 45 minute drive. Many of our tower sites are 90-150 minute drives to the site. Some of our sites are 3-8 hours of driving each way to the tower site which means that you need a very long day to drive there, perform all the upgrades and return home the same day. Some of our sites are so far away that we have to stay over night in a hotel to work at the site to perform routine maintenance at the location. Scheduling a crew to go to each site to perform the upgrades required a lot of coordination to keep from violating agreements with customers who have specific requirement for system performance and outages. Some of our site upgrades were performed at night or weekends to avoid irritating the customers. Some upgrades were performed by one person while other upgrades required a crew. Most of the upgrades had to be coordinated with the office to make certain that there was someone available to test the upgrades along with considering the customer requirements.
While the upgrades were being completed, we embarked on installing new tower sites that were part of the Gen II system, thus providing the customer with a “reward” of better coverage for making the change to Gen II. Once we had many of these new sites on line, we embarked on creating Gen II capacity at our existing tower sites. At some sites, we had sufficient capacity to take some channels away from the original system and create Gen II system at the site. At other sites, we had to install additional capacity in addition to taking some capacity from the original system to create the capacity for Gen II at the site. At still other sites, we had to install a duplicate system at the site to have both the original NEXEDGE system and the Gen II system. At other sites, we flipped the site from the original NEXEDGE to Gen II in a matter of an hour. Each of these approaches has it good points and drawbacks, creating extra work for our personnel to facilitate the most seamless transition possible.
While the system upgrades were proceeding, the reprogramming of the radios for Gen II radios was in progress. The work that was needed would vary depending upon the model of radio and the version of the firmware installed in the radio. The most popular model of radio on our Diga-Talk network is the NX300 / NX800 portable & mobile radios. These radios required upgrade of the firmware to a version that would work with all the Gen II features. These radios also required a Gen II key to be installed in the radio which we had to purchase and we provided to the user for free. The radio then required the programming file (which is custom for each customer and different for every radio on the network) to be replaced with a new programming file. Other model radios such as the NX320 / NX-820 radios had the same requirements while the NX-5000 series radios would operate on a Gen II system out of the box. The firmware for the NX-5000 series radios may have needed an upgrade if newer firmware was available that resolved an issue with a feature that was of importance to the customer or added a new feature the customer needed, but not all users needed the upgrade. A similar situation occurred with the NX-3000 series radios. However, both of these radios required a new custom programming file that had to be burned into the radio.
Some users received new Gen II programming and were now finished with the conversion because we took out the original NEXEDGE programming. Other users were programmed with both the original NEXEDGE and the new Gen II system because we did not have enough sites converted over to Gen II at the time. Some customers allowed us access to all or substantially all of their radios at one time while others made us program one radio at a time when the radio came in for routine service. We catered to the needs of each customer who was treated differently for reprogramming of their radios to the Gen II system depending upon their individual requirements and needs. Some customers brought all their radios to the office and we reprogrammed them to Gen II at which time they were done. Most customers had us come to their location to reprogram the radios. Smaller companies were easier to get all the radios together while larger fleets required multiple trips to the customer’s location to catch all the radios for the reprogramming.
The process of reprogramming radios was complicated and took a lot of time. The customers who had their radios reprogrammed in the early stages of the process typically had both the original NEXEDGE system and the Gen II system programmed into the radios. We would invalidate the radios on the Gen II system until we were ready to have the customer transition to the Gen II system based upon the coverage and capacity that we had available on the Gen II system at the time. Once we felt it was time for them to move, we would invalidate them on the original system and validate them on the Gen II system. The radios were programmed with auto system select which would look for the other system whenever it would lose coverage from the existing system. The drawback of this feature is that when you were in a marginal signal area and the radio lost coverage, the radio would look at the other system to find coverage before returning to the correct system. This means that the radio would spend a considerable amount of time looking for radio coverage where there was none. This irritated some users, so we had to remove the “dual” programming from the radios after we changed them over to the Gen II system.
Work proceeded on both fronts. The customers were having their radios upgraded to the Gen II system while our crews were working at the tower sites converting the system, installing more capacity and adding new coverage. This created a high quality problem as we were reaching the capacity of the server hardware we had purchased. Each set of servers can handle up to 48 sites. However, we were about to exceed that number, so it became necessary to order a second set of servers to expand our capacity to 96 sites. We were informed by Kenwood that we were the first company in the world to order a 2nd block of 48 sites which gave us confidence that we were on the leading (or bleeding) edge of this technology and therefore had the largest system in the country. This caused us some delays proceeding with the Gen II conversion while we installed the additional servers and software to increase our capacity to 96 sites. In the middle of adding the additional block of 48 sites, Kenwood announced a significant upgrade to the Gen II system to make it work more efficiently, faster and resolve some issues from the original configuration. They wanted us to send our existing servers to their office for the upgrade. We said ‘NO BUT HELL NO!” to their request because we could not shut down our existing customers while Kenwood upgraded our servers. So we suggested that they have the new servers for the next block of 48 sites configured for our first block of 48 sites, ship the servers to us and then we would swap our the servers. We could then send the original servers back to them for the upgrade and configuration to our second block of 48 sites. This proved to be a viable and successful strategy to resolve the issue.
As time went on, we discovered that having computer servers at tower sites which are prone to dust, dirt, power outages, power surges, lightning and vandalism has its challenges. Professional computer servers are hardened hardware devices, but nothing is impervious to all attacks from all sources. The servers have redundant power supplies to prevent service outages, but we have had several power supplies fail. Fortunately, we have not had both power supplies in a server fail at the same time. Whenever we are notified of a failed power supply through our remote diagnostics, we schedule the replacement of the power supply. The same can be said for the hard disk in the server. All servers have redundant hard disks which we have had to change when they fail.
To improve overall reliability, MRA built and installed an SNMP monitoring system. We have a 55” TV screen that displays all network elements at the same time. When something happens, the system detects a failure and gives us an audible alert that there is a problem with the system. This gives us the opportunity to resolve the problem often before the customer detects the problem. Some problems we can fix remotely while other problems we can mitigate so that the customer is not affected by the problem while other issues require a trip to the tower site. The SNMP monitoring program allows us to provide the best and fastest resolution to the issue at hand.
As the project has proceeded, we finished reprogramming all the customers to the Gen II system by April 10th. However, we still had to go back to many tower sites to convert the remainder of the channels to the Gen II system. We encountered a problem with the final conversion which caused us to halt the final cutover for about 6 weeks while the issue was resolved. Once we were able to proceed, we finished going back to each of the tower sites to perform the final cutovers to make the system 100% Gen II.
We now can breathe a sigh of relief. The project which has taken years is complete. Now we can concentrate on adding coverage from new tower sites and adding user capacity instead of having to spend time working on system upgrades and the conversion of the tower sites to Gen II. We still have a few customers who want the original NEXEDGE programming removed from their radios which is being completed as this article is being written.
What benefit does this offer to our customers? The big answer is coverage, coverage and more coverage. The old system had a hard limitation of 48 sites and a soft limitation of 30 sites which we had reached a long time ago. The new system has a hard limitation of 1152 tower sites and a soft limitation of 284 sites. This gives us room for significant coverage expansion. Other features include DFA (direct frequency assignments) which eliminates the need to update the radio programming as we add more frequencies. The Gen II system give the radio the ability to learn about new tower sites over the air so that it is unnecessary to reprogram the radio under many circumstances as the system is upgraded and expanded.
It is a new day at MRA. Join in and enjoy the benefits of the largest and most feature rich NEXEDGE system in the country and it is only getting larger……………..