2019-05-10 InsightsCenter Malware
Post-Mortem Owner: Brendan O'Connor
Meeting Scheduled For: No meeting; the fixes have been prioritized and will be executed.
Overview#
Early on the morning of May 10, an attacker caused a security breach at Pagely which affected our Insights Center blog as collateral damage. This caused the blog to redirect some visitors (to a particular page) to a malware host for between 95 and 193 minutes (the exact length is unknown), followed by public-facing intentional downtime for 155 minutes while we investigated and remediated the issue.
What Happened#
Pagely, who we use as a blog host for the Insights Center B2B blog (https://insightscenter.realself.com), suffered an attack against one customer blog (not ours). Due to improper separation between their customers, an attack on this one blog caused malicious code to be inserted on a large subset of their customers' blogs (anything hosted on the same cluster), which included our blog. The attack included placing a redirect on the top of a wide variety of PHP pages in Wordpress, but the only place on our blog at which it was apparently effective was at https://insightscenter.realself.com/all-posts/ . Visitors to the affected page were redirected to a standard malware hosting site; the attack did not appear to be targeted.
Contributing Causes#
- Pagely had (by their own statement) insufficient separation between customers in their blog hosting platform. Had a sufficient chroot jail (or equivalent) been in place, the attack would not have affected multiple customers.
- Unlike the Treatment blog (hosted by Pantheon), Pagely does not enforce a code vs. files separation; that is, a blog hosted at Pagely can self-modify its code in production (which is a normal part of Wordpress, but which is disabled for blogs hosted at Pantheon). Had this separation been in place, the attack would not have affected even one customer.
- Pagely did not go through the SOFTSERV process, meaning that its security was never evaluated. Pagely was not put through SOFTSERV because RealSelf's use of Pagely predates SOFTSERV (first invoice: May 26, 2014), and it has never been re-evaluated. (This is also true of Siteground, which until recently hosted The Treatment, and still hosts AtWork.)
Things that Were Not Causes (but Which Are of Concern)#
- During our investigation, we found that Insights Center had 14 plugins for which patches had not been applied. This, while an issue that commonly causes security breaches for Wordpress blogs, did not contribute to this particular incident.
- During our investigation, we found that Insights Center had a significant number of users who routinely write content for the blog who had Administrator privileges. This, while an issue that commonly causes security breaches for Wordpress blogs, did not contribute to this particular incident.
Resolution#
Bernard put a temporary fix into place (blocking public access to the blog) while we diagnosed the problem, and Pagely put a "full" fix into place by removing the inserted malicious code. However, a full resolution to this issue will involve moving our blog from Pagely to a vetted host (Pantheon), and doing the same for our blog hosted at Siteground. This will ensure that these public-facing RealSelf resources are hosted on a platform for which we have some evidence of security.
Impact#
For at least 1 hour, 35 minutes, and potentially for as long as 3 hours, 13 minutes, all visitors to https://insightscenter.realself.com/all-posts/ were redirected to malware-bearing sites. This was intended by the attacker to affect (and apparently did not) all pages on that blog.
Responders#
- Brendan O'Connor
- Bernard Gardner
- Andrew Mills
Timeline#
The date for all items below was May 10.
| Time (PDT) | Event | Data Link |
|---|---|---|
| 6:27 AM | The attack on Pagely starts (stated in their email to us after the fact). | |
| 7:30 AM | Pagely starts investigating the issue (stated in their email to us after the fact). | |
| 8:15 AM | Eva Sheie posts that there's a browser redirect being published on the "See All" page of https://insightscenter.realself.com/ . | https://realself.slack.com/archives/C0291FNR3/p1557501350351200 |
| 9:05 AM | Bernard Gardner pings Brendan O'Connor to examine the issue. | https://realself.slack.com/archives/C083EGUJD/p1557504344051000 |
| 9:08 AM | After investigating with Andrew Mills, Bernard Gardner files PIT-1857 as a P1 issue. | https://realself.atlassian.net/browse/PIT-1857 |
| 9:16 AM | Brendan successfully gets online with a laptop and begins diagnosing the issue. | https://realself.slack.com/archives/C083EGUJD/p1557504965054300 |
| 9:40 AM | Bernard cuts the site over to running through Fastly, and takes it offline (with a user-visible "Be back soon" message) for everyone other than Bernard and Brendan. At this point, there is no-longer a public-facing redirect to malware. | https://realself.slack.com/archives/C083EGUJD/p1557506435066700 |
| 10:11 AM | The site is now accessible to Brendan once more. (In the interim, there was no public-facing malware, but Brendan couldn't work on the site.) | |
| 10:58 AM | Brendan confirms that the May 10 backup (reported by Pagely as having been taken at 06:23, but no clear time zone) does not contain the malware redirect, and that the current site contains the redirect code in many (several hundred) pages, and therefore that a rollback is the most appropriate remedy to get the site back up and redirect-free in a reasonable time (since we have SFTP, but not SSH, access to the site host), and therefore files a ticket with Pagely to roll the site back to a backup (Pagely does not allow users to roll back). | https://atomic-beta.pagely.com/account/2845/support/152174 |
| 11:12 AM | Pagely acknowledges the issue and alerts their security team. | |
| 11:29 AM | Brendan requests an updated status from Pagely. | |
| 11:56 AM | Pagely states that the site has been cleaned up without a rollback, and that they will shortly send additional information via email. | |
| 12:01 PM | Brendan confirms that from his laptop, the malware code has indeed been excised by Pagely. | |
| 12:01 PM | Brendan requests that Pagely send any information to security@realself.com as well. | |
| 12:03 PM | Bernard confirms that from his laptop, the malware code has been excised by Pagely, and begins to put the site back online (removing Fastly and the block). | |
| 12:15 PM | Bernard confirms that the site is now online for all users. | https://realself.slack.com/archives/C083EGUJD/p1557515733095000 |
| 1:00 PM | Pagely finishes fixing all their customers' sites (stated in their email to us after the fact). | |
| 2:04 PM | Pagely sends their email explaining the issue to security@realself.com. |
How'd We Do?#
What Went Well?#
- Bernard invented, on the fly, a way to block public access to the blog without compromising our ability to work on the blog; he then proceeded to document it in case of future need (at https://github.com/RealSelf/devops/wiki/Gone-fishing-for-alternative-sites).
- Pagely had backups, to which we had intended to roll back. This provided an excellent candidate for a way to fix the issue, though it isn't what we ended up doing (though it's what we intended to do).
What Didn't Go So Well?#
- The lack of a clear technical owner for blogs other than the Treatment (now owned by ACE) meant that there wasn't a clear action to take to begin the response. Accordingly, Bernard made an excellent decision to summon Brendan; however, reliance on individuals, rather than on job functions, should be avoided.
- Pagely's lack of rapid utilities to understand and modify the blog in production contributed to a significant delay in understanding how widespread the issue was; in fact, we never achieved a full understanding of how many files were modified by the attacker, due to there being no efficient way to download the current state of the blog (only previous backups). Note that this is likely also to be true at Siteground.
Action Items#
Messaging#
Internal Slack (#On-Call)#
This is a follow-up for employees. It should be sent out right after the post-mortem meeting is over. It only needs a short paragraph summarizing the incident and a link to this wiki page.
Briefly summarize what happened and where the post-mortem page (this page) can be found.
Appendix#
Email From Pagely (Sent to security@realself.com)#
Hello ,
We are reaching out to inform you of a security-related matter that affected one or more of the sites we host for you. Just to let you know, we've already cleaned things up and taken additional measures to prevent it from happening again.
What happened?#
Our Security Team detected suspicious content was being injected into your web pages. This was causing spam content to be displayed on your site(s) for a brief period of time. We found this to be residing in the files at rest and not manifesting through any kind of web traffic interception.
When did it happen?#
The attackers began modifying files at 13:27 UTC on May 10th, 2019. Our staff identified this was not an isolated event and began investigating the matter by 14:30 UTC, and completed the investigation and cleanup process by 18:00 UTC.
What was affected?#
We found the following site(s) to be affected:
- insightscenter.realself.com
How did it happen?#
Our investigation concluded that attackers compromised a single site on a small segment of the shared web hosting cluster your sites are on and were able to work around security controls designed to isolate each hosted site from other sites on the same segment. Initially attackers only had access to one site, but were able to work around our isolation methods and gained read and write access to files outside of the initially compromised website's document root; this allowed them access to your files but not provide them knowledge of what website’s files they were accessing. The attackers used this access to write a malicious javascript snippet to all files they could write to, which included some of your site’s files.
We were able to identify the attack vector and how the attackers worked around our isolation to gain access other sites' files and have taken measures to prevent it from happening again.
There is no evidence that this attack was caused by any code you uploaded to our platform, but we would recommend you double check that your site shows no other signs of tampering to ensure our cleanup of the site was complete, and did not cause any adverse effects.
Is my site safe/secure?#
We have already removed all malicious content that was added to affected sites, and have cleaned up the mess the attackers made. This attack was not directed at your website specifically, but your website was affected due to the access level the attackers were able to gain. The method the attackers used to get around our site isolation was identified during the investigation and steps have already been taken to prevent any further abuses of this same hole. Sites are functioning as they did before, and our isolation method is working as expected.
What was done in reaction to the problem?#
Pagely has performed a deep inspection of the affected apps' codebase and removed the malicious HTML string from any PHP scripts. All server-side caches were cleared after this cleanup was performed to ensure sites are no longer serving spam content.
We have also reset the database password for your site(s) and updated the credentials in the wp-config-hosting.php file as an additional precaution.
Internally we have blocked the method the attackers used to subvert our isolation controls, and are implementing tests and hardening methods to ensure that our site isolation security works as expected in the future.
If you have any questions, please do not hesitate to write back!