Telecommunications Program provides growing range of communications

January 26, 2016

Communication is required to get information to the electronic message and variable speed limit signs that help keep motorists safe on the highways, and get information from snowplows and weather sensors to the Transportation Management Center (TMC) in Cheyenne.

WYDOT’s Telecommunications Program gets that information through, along with the radio transmissions between Highway Patrol vehicles and dispatch, and telephone calls to department offices around the state.

Telecommunications Manager Robert Wilson oversees the program that includes Field, Microwave, Project Development, Telephone and WyoLink sections.

In addition to the statewide telephone and radio networks, the program provides communications to 368 Priority 1 devices, including dynamic message and variable speed limit signs, and 143 Priority 2 devices, which create roadside Wi-Fi (wireless networking) hotspots.

The Field Section under Larry Sheridan includes five district offices and 11 field shops around the state, and those technicians maintain and support all aspects of the department’s varied communication systems. They serve as the point of contact for most WYDOT employees with communications problems, questions or needs.

“We have expert telecommunications technicians deployed across the state for quick response to provide for public safety and transportation efficiency,” Wilson said.
“The field guys really need to be proficient in a lot of areas. They are a lot more than radio techs, they are telecommunications techs.”

The Wi-Fi communications the program provides is for public safety and transportation operations around the state. Wi-Fi for office computer use is provided by the Information Technology Program.

The program’s level of response to problems is determined by the importance of the devices affected.

“Some are more important than others, and of course it varies somewhat between winter and summer, because obviously during winter you really need to know what the road conditions are and communicate that to the traveling public,” Wilson said. “An overhead digital message sign that the TMC wants to read that there is an accident ahead, road closed, that becomes very mission critical and is a Priority 1 device.”

The Priority 1 devices are monitored around the clock and any problems are reported to an on-call technician in Cheyenne who coordinates the response to repair the equipment.

“Within half an hour our guys are rolling on any mission critical outage,” Wilson said. “It used to be you responded at that level just for radio, and for the mountaintop radio systems that most of the law enforcement, fire, EMS and WYDOT agencies in the state count on.”

At least three technicians are on call at any given time, and all the field technicians are subject to call, which means they could be required to respond within a couple hours.

Most of the microwave radio tower sites are on mountaintops, and in some cases the roads to those sites are closed in the winter.

“We use snow cats to access them and we run snow cat and winter survival skills training for all of our field people,” Wilson said. “We offer training on how to effectively operate that equipment and tow it, because you’ll be driving it in the worst possible weather. It’s a tribute to the dedication of the technicians in the field that they are willing to go out in those conditions if needed.”

To minimize the chances such a trip will be required, all the tower sites are inspected in the spring to see if there was any damage over the winter, and again in the fall to make sure fuel tanks are full, the generator is serviced and everything is prepared for winter.

While a service call to a mountaintop site can be difficult in winter, when the weather is favorable it can provide a perk few other jobs provide.

“The field technicians going to the mountains in the summertime get to see some of the most beautiful landscape in Wyoming,” Sheridan said. “They get to places that the general public wouldn’t even know how to get to.”

The broad range of skills needed for the technician positions can make it difficult to fill vacancies, but when fully staffed, the program has the personnel it needs to keep all of WYDOT’s communications systems operating smoothly.

“We ramped up a few years ago and added additional staff to the districts, because we knew the roadside devices were coming,” Sheridan said. “The initial roadside devices layout was done between here and the Summit in the spring of 2003, and the rollout of the WyoLink system came at about the same time. We saw the handwriting on the wall, executive staff agreed, and they did staff up.”

Field Section

The Field Section’s district offices and 11 field shops around the state are each manned by one to three telecommunications technicians.Telcom-climber.jpg

Scott Prettyman is the District 1 supervisor in Cheyenne, where technicians Jon Miles, Chris Salmon and David Shepard also work. Murray Hill in Laramie and Brian Wyza in Rawlins round out the District 1 technicians.

Ken Link is the District 2 supervisor, and is joined by Joe Compston in Casper and Nick Palmer in Douglas. The District 3 team is led by Tony Richno, and includes Dylan Jensen in Rock Springs and Jack Herman in Jackson.

Nate Smolinski is the District 4 supervisor, and his technicians are Ken Kraft in Sheridan and Bill Torrance in Gillette. The District 5 team includes supervisor Curtis Clark and technicians Randy Savage in Sheridan and Travis Gallo in Gillette.

The technicians do much of the installation, maintenance and troubleshooting of the mobile radios in department vehicles, the microwave radio sites that transmit messages from those radios around the state, and the Wi-Fi devices that provide communications to the dynamic message and variable speed limit signs along the highways. They also are responsible for 26 permanent and 10 portable highway advisory radios around the state.

A big part of the Cheyenne technicians’ job is making sure all the communications equipment in the Highway Patrol’s 240 vehicles works together.

“Anytime a new car comes in or a trooper gets a new car, it is done through the Cheyenne shop,” Prettyman said. “Troopers have certain equipment issued to them, so we take that out of the old car and put it into the new car, and make sure everything works.”

Electronic equipment in Patrol vehicles includes the two-way radio, front- and rear-reading radar, tablets and printers connected to an electronic ticketing system by cellular modems, dashboard video recorders and light bars.

“All of the electronics in the vehicle have to work together,” Wilson said. “If they interfere with each other, nobody’s happy. The same thing applies to the snowplows. Making all the electronics work together in a vehicle is a WYDOT Telecom specialty.”

Once a Patrol vehicle is out on the road, the local telecommunications technicians address any communications problems that arise.

Work on radios in vehicles and at the mountaintop sites is the technicians’ primary focus. They have a fixed schedule of routine maintenance in the spring, summer and fall to make sure everything is working well before winter.

The radios in the vehicles require regular maintenance to make sure they are on  frequency, have the correct power output, and the brackets and connections are tight.

“We use a communication analyzer to make sure the internals on the radio are correct so it’s working at its best on the WyoLink system,” Prettyman said. “We do the same thing for the radios at the mountaintop sites. We try our best to find problems before our customers do.”

The Cheyenne technicians also work on radio equipment at the TMC and Patrol Dispatch Center, where there are a combined 19 radio positions.

Maintaining communications to roadside devices is a growing part of the their jobs.

“The communications we’re using to the roadside locations now are wireless broadband,” Shepard said. “Wireless Internet providers often use these to supply Internet to residential homes. We use them to supply communications to DMS signs, variable speed limit signs and various devices out on the side of the road.”

Wi-Fi connectivity is now available at sand piles along the highway, so snowplow operators can to use tablets to report road conditions and view weather maps.

Hill, who came to WYDOT in 2008 after 20 years as an avionics technician in the Navy, said he works closely with the GIS/ITS Program and others to make sure the department’s communications devices are working smoothly.

“We have a really good working relationship with ITS,” he said. “They own the roadside cabinets and the stuff in there. Our primary function is to make sure that there is a signal that they can use to transmit that information back to Cheyenne. But sometimes they’re doing something else and not available, so we step in to help out.”

The installation of 11 new dynamic message signs on I-80 between Cheyenne and Rawlins this summer will increase area technicians’ workloads only slightly, Hill said. Working on mobile radios remains the mainstay of his job, whether installing them in new vehicles or tuning 187 radios a year.

“I have to work with the plow drivers, engineers, mechanics, the guys in the head shed and figure out their schedules so I can take their radios out and tune them up,” Hill said. “I connect them to an IFR, which is a spectrum analyzer, a signal generator, a receiver, and it runs through a series of test patterns to test certain applications within the radio. You make sure the radio is within an acceptable range.”
 
He also does preventive maintenance on 13 radio sites in the area, inspecting the towers, buildings and equipment inside them. Those inspections reduce the chances of being called out at night or during a storm to address outages.

“We have a great group of guys and they’re very conscientious and hardworking folks, so we don’t have a lot of those, ‘Oh my goodness, we’ve got to go fix this moments,’” Hill said. “We’re doing a lot more preventive maintenance than just-in-time stuff.”

Working with the other technicians, keeping up with technological advances and providing a valuable service are the aspects of the job he enjoys most.

“My customers make it really worthwhile when they say, ‘Thank you for doing such a good job,’” Hill said. “The fact that technology never stops is another key. If you enjoy the thought behind technology, then this is the place for you. Technology moves at such a pace that, if you get bored with your job, you’re not really doing your job well.”

Microwave Section

Microwave-tower-and-falling-tower.jpgWYDOT’s microwave system is the backbone for department communications, carrying the circuits for WyoLink and conventional radio services, some telephone switches and communications for message signs, web cameras and weather sensors.

Microwave System and Radio Site Manager Chuck Kakalecik, and Kevin Gerlitz,  telecommunications technician 2, work closely with the field technicians around the state to monitor the system and keep it operating smoothly.

Joe Compston is the newest member of the team, adding part-time microwave technician trainee to his normal duties at the District 2 radio shop in Casper.

“We do a little bit of everything,” Kakalecik said. “Generally, if we’re fully staffed, there are three people in our department: me and two techs over at the microwave shop, and we’re responsible for the whole statewide system. We rotate being on call, and address the issues as they come up. Plus, we perform preventive maintenance on every radio every summer, do a lot of new construction, and any repairs needed in the process.”

An alarm computer in Cheyenne alerts personnel to problems in the system, and a network manager allows Kakalecik to build circuits, route and configure the network from his office.

“Generally I can drill down here and usually see what’s going on,” he said. “But if it’s lost a link and I can’t see it, I’ll try to get the guys in the field out to give me some boots on the ground and eyes to see what’s going on.”

The Microwave Section personnel make repairs and changes to the microwave system, multiplex, power supplies and the antennae systems. The field technicians are being trained to do some of the preventive maintenance work.

“We build and maintain the entire site, and technicians have to be kind of a jack-of-all-trades,” Kakalecik said. “There’s a lot of civil work, and you have to know microwave radio, two-way radio, telemetry, multiplex, datacomms, network and user interfaces, emergency generators, AC power systems and transfer equipment, building systems and some HVAC. There’s a lot that goes on at these remote sites.”

During summer construction season the team spends at least 50 percent of the time on the road, sometimes 75 percent, depending on work planned and service calls.
“It helps to like the outdoors and working outside, and not be afraid of the winter,” said Kakalecik, who has been with WYDOT 35 years, 25 of them in his current position. “We work pretty hard year-round, regardless of weather or closed roads.”

The team’s biggest problem is Wyoming’s harsh weather, with lightning, ice and wind causing damage above the normal wear and tear.

“A mountaintop during wintertime isn’t the best place to be, and lightning during the summer can take you off the air pretty quick,” Kakalecik said. “We’ll get tower and power line strikes a lot, and it’ll wipe out a lot of equipment. There are just so many different things that you get with a direct hit, regardless of how aggressively we ground and bond the system. We’ve got sacrificial electrostatic discharge protection on everything, and it works pretty well most of the time, but we believe Murphy was an optimist and we stay ready to respond”

Mountaintops are where the microwave towers have to be in order to provide a statewide communication system, because microwaves transmit gigahertz radio frequency in narrow beams limited to line-of-sight transmission.

“If the microwave paths are engineered right, it’s a good solution for building a statewide interconnected system,” Kakalecik said. “That’s the easiest and best way to do it. We suffer less backhoe fade with microwave.”

Because of the line-of-sight limitation, the best way to assure that a problem at one of the towers doesn’t prevent transmissions to a particular area is to configure the towers in rings, so if an outage blocks communications on the most direct line, it’s still possible to communicate by routing the transmission around the ring in the other direction.

“With so many mountains, wilderness areas and Yellowstone, it would be hard to get a full ring around the state, so we’re trying to go with four smaller rings for alternate routes,” Kakalecik said. “We’re looking at two years to complete each ring, but that’s with a full crew. I’ve been so short staffed that’ll be hard to do.”

The section likes to install new equipment itself, so the technicians who will be monitoring, maintaining and repairing the equipment know how the system was put together and works.

The current system is operating well and is in good condition, Kakalecik said.

“Our core is built out pretty good, and since the new digital system was installed we’ve never gone under 99.99 percent availability, which is not bad,” he said. “But, technology is always migrating forward, and we’re working toward the future.”

Project Development

Telecommunications Project Development Supervisor Mark Kelly’s job is to figure out the best way to get communication to any location where WYDOT needs it.

“When a project comes along and they want another dynamic message sign or whatever, we get the input from the districts where they want the device, and then I figure out how to get the communications from there,” Kelly said. “If I can’t get it through the backhaul, then I work with Mike Kurz in Telephone to get fiber, a DSL circuit or cellular modem.”

Backhaul is the term used to describe WYDOT’s own communications infrastructure built over the years to provide links to areas not served by commercial systems.

It uses Internet Protocol (IP) wireless communications to serve roadside locations and field shops, and then piggybacks on the department’s microwave radio towers and dishes to get the signal to its destination.

Often WYDOT needs communication to sites where nobody else needs it, so there is no commercially available service, or the service is not cost effective or as reliable as the department needs to meet its transportation efficiency and public safety responsibilities.

“So we’re utilizing our own sites that we already have and not adding more infrastructure to accomplish this,” Kelly said. “We just keep following the microwave path out and once we get to a site, then we can break it out and go to the roadside with another radio from the tower site to the roadside site.”

Path analysis is needed to make sure it will all work, and, although terrain analysis software helps with the job, Kelly and field technicians still have to go to the sites themselves to make sure trees or buildings won’t block the transmissions.

In addition to being able to customize the system to WYDOT’s needs, other advantages of WYDOT’s backhaul are there are no recurring monthly fees for the service and when problems arise the department’s technicians can restore service quickly.

“We buy the equipment, install it and maintain it, and we can normally restore communication faster than a commercial provider, because we have techs throughout the state, and they’re fairly close and they have spares in hand,” Kelly said. “If something goes down, they can analyze the problem, and replace or fix what’s needed.”

The department’s public safety communications needs are expected to continue to grow, with more Wi-Fi hotspots needed along the highways to support the tablets going into Patrol and Maintenance vehicles.

One of the most exciting new technologies on the horizon is the use of dedicated short-range communications to connect vehicles on the road to each other and to WYDOT’s communications infrastructure.

The U.S. Department of Transportation announced last month that Wyoming is one of three locations nationwide that will get grant money to explore the use of the technology to improve traffic flow and safety.

“We would provide communications to the connected vehicle equipment at the roadside so the TMC could put a message on that unit that would broadcast it in a loop,” Kelly said. “When a vehicle connected, it would get a message on their connected vehicle display saying something like “Crash 20 miles ahead, slow down and be prepared to stop,” or “I-80 closed, exit at Laramie.”

Connected vehicles also could communicate with each other, so, if a vehicle started slowing down, a vehicle behind could get a message such as, “Vehicle slowing ahead, be prepared to slow down,” or “Snowplow ahead traveling at 30 mph.”

“So, if you think about the Summit pileups in fog or the number of times we’ve had vehicles hit the back of our snowplows, it would seem like this technology could prevent that,” Telecommunications Manager Rob Wilson said. “This technology is coming, and it depends on the special wireless communication that is our forte. Mark is our point man on that.”

The connected vehicle devices are already available as an option on some vehicle models, and the National Transportation Safety Board is urging Congress to require them as standard equipment in the future.

One Kelly’s jobs is to stay abreast of new developments for providing communication to roadsides.

“It can be a challenge,” he said. “I’m subscribed to several different forums and magazines and I get weekly updates about what’s going on in the world. It’s always changing, but people share the information easily. They want the world to know.”

He’s worked in the telecommunications industry for 21 years, eight of them with WYDOT, and he enjoys the challenge of learning new technologies and helping the department use them to make travel safer.

“It excites me to think, ‘What am I going to learn today?’” Kelly said. “I’m always letting the guys in the field know, this is coming. Keeping the guys informed in the field has made my job a whole lot easier. They’ve taken on that challenge, and it’s been a great working relationship.”

Telephone Section

Cheyenne-Telecom.jpgWYDOT’s telephone network includes more than 20 systems around the state and all the data circuits needed for the Information Technology Program’s computer network, and it’s all overseen from Cheyenne by Mike Kurz and Mike Pukash.

“Basically all the telephone circuits and the whole IT network come downstairs to our telephone room where all the main circuits go, so everybody gets the telephone and Internet service they need,” Kurz said.

Small offices around the state, such as a leased space for Driver Services or Highway Patrol, are not directly connected to WYDOT’s network, so the Telephone Section contracts with local telephone companies to provide the needed service.

The section also oversees the department’s cellular program, which includes not only cell phones, but approximately 500 modems used for data connectivity.

Every Highway Patrol vehicle now has a cellular modem to provide Wi-Fi connectivity for troopers’ tablets. Cellular modems also provide connectivity for many of the electronic message signs along the highways, along with remote weather information sensors and some of the department’s traffic counters.

Remote shops, including Granger, Pole Creek and Medicine Bow, rely on cellular modems to provide Internet connectivity for their computers.

When Kurz came to WYDOT 10 years ago, he estimates about 60 department personnel had cell phones and there were no cellular modems. Now WYDOT has about 1,300 cellular devices, about 600 of them being smart phones and the remainder being modems and other devices.

WYDOT’s telephone system also has undergone upgrades in recent years, going from independent systems throughout the state to a gateway network controlled by a computer in Cheyenne.

“Everyone of the telephone systems out there in Sheridan and Basin, Casper and Rock Springs are all connected via some kind of a data line, DSL or maybe the IT network or the microwave network,” Kurz said. “We bring them all back to Cheyenne and they connect to the main computer.”

The advantage of that system is it allows WYDOT personnel to call many of the department offices around the state by dialing only their seven-digit numbers, and others by dialing nine and then the seven-digit number. That saves the department money, and when it’s time to upgrade a local system, that also is less expensive.

“By having the main brain here, I was able to upgrade Casper for something like $35,000 instead of $200,000,” Kurz said. “It also allows us to program it, so if someone in Sheridan needs a new phone installed, we actually program it from here, we don’t have to go there to do it.”

Because WYDOT’s has public safety responsibilities, the telephone system is considered mission critical, so if anything goes wrong with the network, it is set up to allow calls to be made on the local system by dialing 1-307.

The department spends millions of dollars a year for telecommunications, but has seen a steady drop in the costs over the past four years, due to reduced long distance costs, and negotiating better deals for leased lines and equipment.

“Just for WyoLink alone, it used to cost about $1.3 million a year for their leased circuits,” Kurz said. “We’ve gotten it down over the last four or five years to about $600,000 by renegotiating contracts and using metro Ethernet instead of T1s.”

Kurz works closely with the state’s Department of Enterprise Technology Services (ETS) as well as WYDOT IT.

“If a customer wants something but is not quite sure what it is they want, I figure out a solution,” Kurz said. “ETS verifies that it is a good solution. Right now IT is upgrading around the state. When the bids come through, I look at them for WYDOT and for IT to make sure they are good bids.”

The Wyoming Unified Network (WUN) that ETS is working to build will make it possible for WYDOT to cut telephone costs further through the use of voice over IP technology know as VOIP.

“As we are upgrading, we know that the future technology to make things even less expensive is VOIP,” Kurz said. “You have VOIP that actually goes out to the provider, like Vonage, and we are moving that direction. All of the phones at the new complex in Sundance will be VOIP.”

WYDOT personnel using the phone system won’t notice any difference in their service, but in addition to the cost reductions, the VOIP system offers advantages for department technicians.

“You could take your phone and move to another office or go to another city and it will still work and it will be the same phone number you had,” he said. “Nothing changes, and that’s going to save a lot of money on maintenance − on me sending a technician out to do the installation and move things around.”

VOIP and the WUN also have the potential to allow the department to combine its voice and IT networks. Currently, if the two were combined, it would mean an outage would knock out both telephone and Internet communications. But the WUN is designed to provide the reliability, redundancy and alternative routing capabilities needed to avoid that.

“That’s one of the biggest things that’s going to be changing in the future,” Kurz said. The phones will work the same, it’s just that the way we route it will be different so it’ll save money.”

WyoLink Section

WyoLink is the state’s digital trunked public safety communications system, which is operated by WYDOT, but used by 264 state, county, local and federal agencies.WyoLink.jpg

WYDOT is the biggest single user of the system, but accounts for only about 17 percent of the total use, WyoLink Section Supervisor Marty McCoy said.

His section monitors the system, troubleshoots any problems that arise and handles administration and planning.

“Troy Berg and Scott Harris prepare all the radio program files to go in each radio on the system,” McCoy said. “We retain the sole ability to do that, with the exception of two certified commercial shops, because the parameters and programming is very complex and interrelated.”

It’s a big job because there are more than 16,000 radios capable of using the system. So far this year they have completed programming for 2,044 radios.

“We do not actually touch the radios,” McCoy said. “Users and shops in the field send the radio program file to us, we add WyoLink or make the WyoLink changes they want, send it back to them and they program it in the radio.”

Through the end of August, the system had handled 13.7 million calls this year, with the system being available 99.99 percent of the time that users have attempted to make a call. During 2014 WyoLink handled more than 20.8 million calls.

“The system is very reliable,” McCoy said. “We track every outage, when it starts, how many towers it affects, what the cause was and when it ended. We report that and graph it every year in our balanced scorecard, and we use that to look for what we can do to reduce the frequency and the time of outages.”

The section has two radio consoles in its office, so when agencies report issues with their consoles, WyoLink technicians can try to duplicate the problem and find a solution before working on the agency’s console. They also have mobile equipment they use to test the system’s coverage around the state.

In most areas WyoLink provides the same or better coverage than the analog system it replaced, but offers the advantage of allowing all the agencies on the system to talk to each other when needed.

Interoperable talk groups are built into the system for communications between agencies in a particular county, and multiple agency talk groups allow communication between any agencies on the system.

“How it’s set up, the technical part of it, is insanely complicated,” McCoy said. “We try to move the line between insanely complicated and easy to use as close to the easy to use side as we can, and keep the user on the easy side of it.”

Because the system is vital to public safety and used by so many agencies around the state, reliability and security are top priorities for the WyoLink Section.

“All of our sites have battery backup, and they all have emergency propane-fired generators,” McCoy said. “If the battery chargers fail, we get an alarm. If the power goes off, we get an alarm. If the generator doesn’t start we get an alarm. Generators exercise every Monday morning at 6 a.m.”

The equipment room in Cheyenne that operates the system has a 100,000-watt uninterruptable power supply that can power all the equipment during the 10 seconds needed for the emergency generator to take over.

The system also has redundant servers and controllers, so if there is an equipment failure, often the users never know it.

“You have to have passed a fingerprint-based background check through the Department of Criminal Investigation before you are allowed unaccompanied access to the equipment room,” McCoy said. “We also have a firewall that locks this thing down 42 ways to Sunday, and it is not connected to the Web.”

If one of the radios on the system is stolen, the technicians in Cheyenne can disable it.

The section’s monitoring software provides extensive real-time and archived data on the system’s operation. At any given time it can tell how many radios are connected to the system, who is talking at the time, how many busy signals have been given and the history of the system over the past 24 hours.

“It can produce graphs and reports on system trends, what sites and talk groups are being used, how much a certain agency uses the radios and what time of day sees more use on the system,” McCoy said. “It really helps us drill down into the system and get an idea on what is going on both now and in the past.”

The system also shows whether a door is open at any of the equipment buildings at the remote sites, and whether an auxiliary cooler is operating. The door alarms were installed after some of the buildings were broke into, and the cooler alarm indicates the temperature in the building has exceeded 80 degrees and the air conditioner is not working, meaning a tech visit is needed.

“Everything that happens on this system, we get an alarm,” McCoy said. ““It’s about public safety and first responders across the state, which is one of the reasons that we have been accused of being rabid about responding, about upkeep, about maintenance and security.”
Telecommunications Program provides growing range of communications

Telecomm-tower.jpg