UPDATE: Approximately at 10:50am CST our monitoring systems confirmed that 99% of the servers were online. As of this update (12:50pm CST), 100% of the servers are online. Once more, we sincerely apologize for this inconvenience and appreciate your belief in Blazing SEO’s desire to provide the best service possible.
At 3:38am CST on May 14, 2019, our monitoring systems notified us of a large, widespread, outage on one of our upstream ISP providers. We immediately contacted their support to expedite the resolution of the problem to no avail. There has been intermittent uptime throughout the morning, but they are still working on restoring some power issue in their facility. The estimated resolution time at this time is around 12:30pm CST May 14, 2019. This blog post will be updated at the top of the blog post with any updates we receive from them.
As a reminder, our proxy infrastructure is a composition of owned IP addresses that we own ourselves, as well as leased IP addresses that we lease from various ISPs in order to provide as great of a ASN and subnet diversity as possible to our customers. By utilizing multiple ISPs, we mitigate a 100% outage and are able to provide much better diversity. However, on the downside, we cannot ensure 100% uptime across all these ISPs as each one of them has different levels of accepted downtime (we have zero tolerance of downtime on our owned networked). Since these are leased IPs which run on our proprietary proxy software, we are at the mercy of this particular ISP to resolve the issue as soon as possible.
Please accept our apologies in this matter. We have slowly reduced our footprint on this provider about 6 months ago when we started noticing individual servers going offline for a few minutes at a time which indicated to us that they have a loose tolerance on uptime. Unfortunately, the diversity of subnets this provider offers is too great to entirely remove ourselves from their network.
We would like to take this moment to provide some notes:
- If your use case allows it, go into your proxy dashboard to set your proxies to be replaced if one of them is discovered offline. Our proxy system is purposely built so that if we detect any outage, they will be replaced with a working proxy. By default this option is turned Off because many use cases (such as social media users) would not want this to happen and would prefer to wait until the proxy is online again.
- Our engineers are watching our monitoring system actively to await for power restoration from the ISP – we are doing everything we can on our side to have these online as soon as possible. Furthermore, our tech department is calling their phone service each hour for an update to ensure things are progressing positively.