March 1, 2017
This article is part I of a Managed Services Security Series. Part II discusses how unfinished projects create the best holes in security. Part III is the story of solving a security hack at a Fortune 500 size customer.
As everyone has heard in the news over the last few years, ransomware is a big deal. Companies are losing money and getting egg on their face. CIOs are very publicly losing their jobs. OST Managed Services sees the effects of ransomware first hand. We know how to effectively battle an infection and prevent them in the future. Anyone who watched Mr. Robot season 2 and (spoiler alert) saw a bank hit with ransomware just as grandma was trying to withdraw some money, we experienced some of the same things in real life at several customers the same week the episode aired. Minus taking $5,900,000 in duffle bags and lighting it on fire (I said spoiler alert). In our cases, getting the data back. The intent of this article is to give you advice on how to survive ransomware, without any plugs or vendor bias.
Where it comes from
Ransomware can come from a number of sources, but the three most prevalent sources are email attachments, emailed links, and website popups. Yes, email attachments are still spreading viruses. Back in the day, we would put up a “wall of shame” and post the latest user to open an email that trashed the Novell server (yeah, I went back there). In 2016, email attachments accounted for 31% of new infections, email links 28%, and websites (via popups, auto downloads, or java/flash vulnerabilities) 24%. In total, infections were up 500% over 2015. (1)
2016 also saw an increase in targeted attacks, using ransomware as one tool in the toolbox. This is a situation where a hacker or group, usually aiming to steal money, target a specific organization. They obtain publicly available information about the company’s reporting structure, determine a number of email accounts via phishing and publicly available lists (or databases available for purchase on the dark web), and use that information to begin an attack to make an entry into the organization. Spamming people with emails and attachments, sending emails crafted to look like they came from other legitimate parties in the organization, even hijacking DNS entries to put up convincing fake websites with malware. There are several hacking “kits” available on the dark web used in some of these cases. These are kits that contain ransomware (custom compiled to fool AV software initially), rootkits to allow you backdoors onto PC’s, and in some cases, even a full online store to process the ransomware payment and transfer you the funds. These are actual companies, and they make their money by charging a fee on all ransoms collected. In 2016, OST Managed Services battled these attacks at several customers, and they are quite sophisticated. (2)
Did you know… I can purchase the entire employee list of an organization, with phone numbers, and email addresses, about 75% accurate circa 2015 for about $5 on the dark web. That’s less than 100th of a bitcoin (or half an ETH if your still riding that wave). So remember, your email address, name, and phone number are PUBLIC INFORMATION. It is not possible to hide in this day and age by obscurity, or simply not giving out your information. You must assume, if you plan to interact with anyone on the interwebs, that everyone will eventually have your contact information, and attackers will inevitably try to target you, and your end users.
Where it hits
Fileservers. Sure, some poor shmucks PC also got encrypted in the attack, but since that is the person that clicked the link that destroyed the fileserver, they shouldn’t have a PC anyway. File shares and NAS storage are by far the most hit in ransomware scenarios. Someday, ransomware will get more sophisticated and attack databases, email servers, etc., but for now, its file servers. This is because it is successful, and there is little being done to protect against it, so hackers really have no reason to go further. Fortunately, file servers and NAS systems also can be the most protected, if done right.
So we know it hits PC’s with people who made the mistake to click the latest cat-pizza-taco meme (Did you click it? Shame…), and those PC’s then encrypt all the file server data they have access to. That means we just need to educate the users, protect the PC’s, and backup the data. Each one by itself has holes, but on a whole, it’s a solid plan.
Here is what we’ve found works best.
Common sense protection
Security awareness trainings: Ransomware has a higher likelihood of infecting non-technical end users. Ransomware, and most malware, is typically dependent on the user making a mistake, and clicking a link or email allowing for an infection to take hold. The best defense against this, quite simply, is educating everyone on the essentials of being secure.
What to do:
- Conduct security awareness trainings that cover at a minimum: security is everyone’s responsibility, what to watch out for (phishing, clever browser popups, emails with fake senders, attachments that infect on opening, etc), and how to practice common sense security habits (how to choose a password, never give out credentials, always treat callers and emails with suspicion when asking for something, etc.). In order to be effective, these have to be short, and slightly interesting. Maybe 30 minutes, with 30 minutes for questions.
An antivirus that works, and keeping it working: 60% of the ransomware infections we see take hold after the virus signature was released from an antivirus vendor in use at that customer. This means that, had the antivirus protection been healthy in the environment, 60% of the incidents we have seen would not have happened. The truth is, antivirus is no longer sexy and cool. Back in the day of the blaster worm, everyone was racing to get a good enterprise AV deployed. Now, the focus is on fancier high tech tools, like SIEM systems and intelligent firewalls, but antivirus is just as important today as it ever has been. We routinely see environments where AV has been forgotten about, and the protection is just not effective. In addition, some antivirus engines are better than others, specifically in what is called “zero day” detection. This means detecting a piece of malware before there is an antivirus signature for it released. Many organizations have replaced antivirus engines over the past decade to get something cheaper, or with a better management frontend, and now have a less effective tool. I wont go into vendors, since this is a vendor neutral post, but a couple of very popular and cheap AV’s we see at several customers, are nearly pointless in fighting ransomware. It’s an important thing to pay attention to if you want the protection to be effective.
What to do:
- Select a highly rated antivirus system, not just what’s cheap or easy. A good resource: https://www.av-test.org/en/compare-manufacturer-results/
- Establish a process where the environment is scanned for all network connected devices, and that is evaluated against known protected devices, with frequent (maybe weekly) reporting. You should know what is on your network that is not in your AV console, and it should be one of the most important reports for any CIO.
- Monitor antivirus health and definition deployment, as a tier 1 service. If your main AV definition server hasn’t downloaded updates, or some clients have not, that needs attention right away. Sometimes the window is short if you want the protection to work.
Scan shares for inconsistencies: Certain ransomware change filenames, or leave ransom notes behind as they encrypt data. This means it can sometimes be possible to tell when a ransomware encryption of a network share has taken place. There are tools developed as open-source on the web that you can setup to scan shares, watching for peculiar behavior. OST Managed Services has developed tools internally to scan our customer file servers, and generates tickets when suspicious activity is seen. For instance, if known ransomware extensions are found, or a significant percentage of files are changed in a share. Why is this important? After all, users will let you know when data is encrypted, you will get calls, people will yell. Well, what if they don’t. What if data is encrypted, and its not noticed for several months. What if that’s beyond your backup retention, and what if the data is critical? We have seen this scenario play out a number of times, with potentially disastrous consequences.
What to do:
- Find and implement an encrypted file detection script, as an example: https://gallery.technet.microsoft.com/scriptcenter/Scan-for-Ransomware-and-cb075ccb
Snapshots, snapshot, snapshots: Most NAS systems, and windows file servers support storage snapshots. It is critical that every important share is housed on something that supports snapshots, and that you verify they are on and working. For those that aren’t storage admins, a snapshot is essentially a copy of the file system at a point in time, that you can access and restore data. Technically behind the scenes, it actually works the opposite way, but that’s a different article. Snapshots are critical, because when there is a ransomware infection, restoring from backup isn’t always feasible. Only certain file types are usually encrypted, so you don’t want to restore everything. Only files in folders that user had rights to were effected, so you don’t want to roll back everyone’s changes to the whole server. If you have access to snapshots, once you have prevented encryption from happening again (find that PC and shut it down), you can write a simple script or xcopy command line to only copy back the files that are of the right type, that have changed during that day. Plus, its nearly instant, just copying files. It’s not mounting a backup tape, so you are up and running far quicker with less work.
What to do:
- Use only systems that support snapshots, make sure they are turned on, and audit all shares/volumes for snapshots to make sure they are working.
File access audit permissions: You can use audit file permissions to track who writes to files, and get a hit list of people to check for ransomware during an attack. Fortunately you can put things in place, without turning on the functionality, so that you only take a performance hit during a ransomware situation.
What to do:
- On a key file share, possibly one that houses personal user folders, apply “write access” audit permissions that track every write to a file, but then disable the audit logging. Most NAS appliances have a setting that turns on/off this audit tracking, as well as windows file servers, so you can have it deployed and ready in the “off” setting. When an issue happens, turn it on, and see who has the highest hit count on file accesses. Then check those PCs for ransomware.
Backups: This should go without saying at this point, since backups have been a core to IT for decades. However, many IT organizations do not monitor daily backups, follow up and correct any failure, and test that they work. In OST Managed Services, we do this religiously. You never want to have a call from a customer to restore something that’s lost, and then see nothing there when you go to do the restore.
What to do:
- Audit your backup configuration to ensure every critical system is backed up, don’t just trust the tool.
- Monitor your backups, even if it is simply having someone sit in front of a screen all day checking emails, or get a backup monitoring tool like BackupRadar.
- Test backups, do a very simple quarterly restore test, maybe just one simple folder.
Every organization will get hit in some way by ransomware. It is going to get more and more sophisticated as time goes on. However, hackers are concentrating on convincing people to open things they shouldn’t, and are reliant on us forgetting to do the basics like antivirus, backups, and common sense protection. Hackers know these things are just not that well maintained, and 2016 was the first billion dollar year for it. Get those tedious basics in order, educate people, and you wont be a statistic.