From time to time, events happen that can cause large or even total listener drops. In this article, we will explore some of the more common reasons that listener drops happen and how to get an idea what might have caused the drops to happen.
Noticing the Drop
The Triton Digital Webcast Metrics Realtime Listening statistics referenced below are used to provide an indicator of how many listeners are active on your streams at a given time. The data displayed is not fully filtered and processed, as these efforts take time, which would defeat the purpose of showing a "realtime snapshot." Because of this, Realtime Listening should not be used in place of other Triton measurement tools to determine precise listenership on your streams.
Significant listener drops are often noticed first in Webcast Metrics under the Realtime Listener reports. When looking at the "Listeners for Last 24 hours" graph, you can see when there is a sharp decrease in listeners. The graph shows either a drop of a portion of listeners (partial drop) or a drop to zero listeners (full drop). The type of drop (partial or full) is our first clue in determining a possible cause.
Partial Listener Drop
A partial listener drop could be due to:
A show or program just ended.
A single mount had an issue.
Planned maintenance of our infrastructure.
A regionalized internet outage.
An issue with Triton’s Real Time Stats mechanism.
Full Listener Drop
A full listener drop could be due to:
Loss of connectivity at the source of the stream.
A major issue happened in the Triton infrastructure.
An issue with Triton’s Realtime Listening statistics mechanism.
Possible Causes of Listener Drops
Possible causes of listener drops are described below.
A show or program just ended
We won't go into much detail here as we assume that you are aware of your station's programming and listener habits enough to recognize this type of listener behaviour. It is worth mentioning here, however, as sometimes these events coincide with others to give the impression of a larger issue.
Region-wide Internet Service Provider issues
The internet is an enormous network of interconnected servers serviced by a large number of backbone and secondary Internet Service Providers (ISPs). Sometimes, large ISP providers have issues which affect certain regions of listeners. Major ISP issues can mean that large groups of listeners lose connectivity to the internet and therefore cannot download the stream from the Triton servers. When these issues happen, ISPs usually communicate via their web sites.
Loss of connectivity with the source of the stream
This is the most common of the problems, but unfortunately the hardest to diagnose and prevent.
From Triton's perspective, the "source" disappeared and became unavailable for a period of time. Short disconnections happen more often than you might expect, sometimes happening several times a week in locations where the internet isn't very stable. Most of the time, those disconnections do not translate into a full loss of the source of the stream as our system has mitigation measures (such as stream buffers and server redundancies) in place which allow the stream to continue without interruption. However, those mitigation mechanisms have their limits and when an issue lasts for a long period of time (upwards of 30 seconds), it can happen that the source is fully disconnected from our network. When this happens, all listeners will disconnect as there simply is no more stream for us to send to them.
Several things can happen to disconnect the Station Manager encoder from our Media Relays:
"Something" happened and the connection between the streaming computer (Station Manager encoder software) and the Triton network (Media Relay servers) was severed. The encoder was okay, your ISP was okay, the Triton MR servers were okay yet there was something between them that prevented the stream from being sent reliably. Even after the stream moves out of your ISPs network, there are ofter several "hops" (links / relays) the information travels through to get to our network. Any one of these hops is a potential failure point that can cause issues. A "ping" to our servers might not show the issue, as it does not replicate the sustained flow of information that a stream requires. The Triton Support team has a monitoring tool that can be installed upon request to attempt to give some visibility on where the issue happened. This tool (which is essentially a tracer outing tool) does not help to explain past events but may help in the case of recurring problems. Even then, the issue in this case is almost never within Triton's power to fix.
There was an issue with your Internet Service Provider (ISP). Your ISP should be able to confirm if this was the case. Please verify with them first if there was an issue in the area where your streaming encoder is located.
There's an issue within your local network. Perhaps a switch or router failed, a network cable was disconnected, or a network card experienced a hardware failure. Please engage with your IT team to verify these critical components.
The streaming computer had an issue or was shut down. Perhaps it restarted because it was performing Windows Updates, perhaps there was a memory leak and it froze, perhaps someone tripped over the power cord (it has happened!). It is very hard for Triton to know exactly what happened in such cases but we hope that your engineering team will be able to inform you of any such occurrences.
Station Manager had an issue or stopped working properly. If you notice such an issue on your encoder, please open a ticket with Triton Support. The Station Manager logs should give the Triton Support Team some insight on the cause and possible fix.
Someone stopped the stream. While extremely rare, it can happen that a person opted to stop the encoding of the station on the Station Manager software. Please verify within your organization if anyone stopped the stream on purpose. The Triton team will only be able to see that the station was stopped manually in Station Manager but not who performed this action.
A major Media Relay configuration was changed at Triton. While troubleshooting an issue, Triton Support agents sometimes change Media Relay locations to improve connectivity issues. Such changes are always coordinated with our clients and scheduled through a Support ticket. Performing these changes resets the connection with Station Manager so a temporary loss of the source will occur.
An issue with Triton’s Realtime Listening statistics mechanism
It is possible that Triton is having an issue with the mechanism displaying the Realtime Listening statistics. While we are usually quick to correct such issues, please feel free to contact Triton’s Support team and alert us to the issue you are seeing.
Note that in some cases, such an issue can also be seen as a plateau in the statistics where several columns display the exact same numbers.
Because of how the Realtime mechanism works, we currently cannot correct past display issues. However, any such issues will not affect the data in other reporting tools, as Realtime Listening is an independent mechanism, separate from other Webcast Metrics reporting tools.
Triton is actively working on improving this mechanism to make it more robust.
A major issue in the Triton infrastructure happened
Despite our best efforts, major problems sometimes happen. Key hardware can fail, a data center can become unreachable, a major fibre connecting our infrastructure to the rest of the world can be cut. When these problems happen, we post notifications on our Statuspage site. We recommend you subscribe to alerts from this page to stay updated on system and product issues as well as planned maintenance.
Still need help?
If, after reading all this, you encounter an issue that does not fit the above situation, or if you still have questions, please contact our Support Team and we will help get to the bottom of the issue as best we can.