On Friday 22nd and Saturday 23rd November 2019 GreenNet's website and most of our online services, including most of our customers' websites, went offline for several hours on two occasions. Our technicians became aware of the outage on Friday lunchtime and worked remotely and at our data centre until the small hours of Saturday morning to identify the fault and restore access to websites and emails.
By about 2AM on Saturday things were coming back online, but they went down again later that day. By then we had revised our initial diagnosis of a hardware problem, and as we checked though our logs and discussed with our upstream provider LDEX, we became aware that the outage was caused by a deliberate attack on our infrastructure. We tweeted updates and kept in touch with users through the weekend as we worked to bring back our services.
GreenNet hosts websites for many campaigners, activists, NGOs and charities, some of which are politically "challenging", so we are prepared for issues of this nature to occur - sometimes customers will let us know that they have received threats or suspect that their website may be or has been attacked, and there are a variety of ways that hackers can try and access or disrupt our web services that we deal with routinely. We had a well-documented attack in 2013, which we believe was aimed at a website run by Andrew Jennings exposing corruption at FIFA (the campaign culminated in Sepp Blatter's investigation and suspension in 2015).
Our immediate priority was to bring our services back online. The attack was a Distributed Denial of Service (DDoS) which basically hammers away at our servers with millions of requests until they run out of capacity and become unavailable. With that much traffic it was hard to identify the actual target of the attack initially. The volume of traffic was at one stage equivalent to our data centre's entire traffic with Google, which gives an idea of the scale of the DDoS - the impact on the data centre was significant enough that they also had to block access or "blackhole" a large amount of traffic.The attack was repeated the following weekend, with less disruptive impact though some websites were unavailable for a short time.
Once we had got everything back up the and dust had settled we were able to carry out some forensic examinations of our servers and logs. The DDOS did not involve any actual access to our servers, so there was no risk of a data breach or actual hacking or intrusion of email or websites. We spotted that the attacks lasted almost exactly an hour each time, so which indicates that it was likely to have been a "DDoS for hire". We were able to glean strong circumstantial evidence that we believe shows that the target was an organisation that was publishing a new research report that weekend Papuans behind Bars. We noted that PBB work in a region where similar attacks had been carried out, and that suspicious checks on the website's availability were made immediately following the start of the DDOS attacks.
In the past few days we have been in contact with PBB (as well as a few other customers that we think may be at risk of similar attacks) and have worked with our friends at Equalit.ie to set up a "proxy" hosting arrangement that will prevent future attacks from taking down their websites, or other sites on our servers from suffering collateral damage, using the Deflect CDN.
Here is the report that the attack was aimed at blocking, so now we'd like to ask that you please share as far and wide as possible!
Here's the PDF version of the report itself: https://www.papuansbehindbars.org/wp-content/uploads/2019/11/PBB-Jan-2018-Oct-2019-Nov2019.EN_.web_.pdf
and the web page all about it: https://www.papuansbehindbars.org/?p=3809
We've been in contact with TAPOL https://www.tapol.org/ and Safenet https://safenet.or.id/2019/08/keepiton-in-papua-and-west-papua, who work closely with PBB and in the region to promote human rights and to highlight the treatment of political prisoners, as well as to challenge internet shutdowns. Please promote their important work! #keepiton
We will be tweeting about this over the next few days, please feel retweet, boost and amplify the message! If you tweet, please use the #keepiton hashtag and tag us and @PapuanPrisoners, @TapolUK and @Safenetvoice
Thanks from all of us at GreenNet! We'd also really like to say a huge thank you to our customers for all your patience and support during and after the attack.
(This is a longer version of our blog post at APC which was linked by BoingBoing)
Add new comment