Windows 2012 R2 BAD_POOL_CALLER Blue Screen of Death and iSCSI LUN on Synology NAS

I was called out today to look at a Windows 2012 R2 server that has been running happily since it was installed back in June of this year.  When it was installed, we also installed a Synology DS415+ NAS and 2 x 4TB drives which were mirrored to provide data protection.  The available space on the NAS was split into 2 volumes.  One volume was used for files that were needed locally / remotely and the other volume was used to present an iSCSI volume to the server as a local Hard Disk so that the backups were written onto the NAS.

Earlier in the month (7th October) I received several Improper Shutdown emails from the NAS as the power supply had fallen out of the back of the NAS because someone had knocked the power adapter that comes with it and that had caused the plug (4-pin DIN plug) to come out of the back of the NAS.

After the Improper Shutdowns, everything had been fine, backups were working to the NAS and all was well.

Around 3:00pm on the 15th October, our office received a call advising us that the server had Blue-Screened with a BAD_POOL_CALLER blue screen of death and could we help to get it working again.  Numerous options were tried by our support staff to get the server back up and running again but all proved fruitless and so I was asked to attend site 1st thing in the morning.

When I arrived, I noticed that the NAS was not working properly (the power light was on but no LAN lights), so I switched it off.  I then went to look at the server which had been left in Safe Mode overnight.  Digging around I couldn’t see anything screamingly obvious as to why the server had suddenly died.  There had been no Windows Updates, no system changes, no power issues, nothing that I found using Google seemed to fit the issue I was having, so I gave the server a reboot and randomly it rebooted cleanly.  I logged in to the server and it seemed totally happy.

After leaving it running for a few minutes, I decided that if that server was fine, the NAS should be switched back on again so the users could access the files on the NAS.  After switching on the NAS, I sat down to drink a cup of tea and discuss how weird it was that the server booted happily after failing to do so numerous times the night before.  A few minutes later, the server fans went into turbo mode and I got that sinking feeling that the server had crashed again and it indeed had.

So – I switched off the NAS, rebooted the server, which booted happily, removed the iSCSI connections to the NAS, powered on the NAS and everything was working happily again.  At some point in the very near future, I will zap the volume, zap the iSCSI LUN / Target on the NAS, re-create them and then re-connect the server back up and hope that the problem goes away.

So – if you receive a BAD_POOL_CALLER Blue Screen of Death error on a Windows Server using an iSCSI volume on a NAS, then it may well be the iSCSI volume that caused the problem and you will hopefully know how to fix it.

 

Gameover Zeus – What Is It and What Can You Do About It?

What is all the fuss about?

Recently, the FBI together with authorities in several other countries, took down some key computer systems that were used to control infected computers around the globe and the infections were designed to steal usernames and passwords on the infected computers.

Those key computer systems are no doubt in the process of being rapidly replaced somewhere else in the world and as soon as they are up and running again they will resume communications with the infected computers and unleash an attack on as many computers that they can infect with the aim of stealing yet more usernames and passwords so that these can be used to steal your money!

What could happen to me and my computer?

Not much if you have an Apple Mac computer!  This nasty will only affect Windows-Based computers (because the largest proportion of the computers in the world are running Windows).  So us Apple Mac users can sit back with a smug grin on our faces😀

What will the virus do?

If you are already infected or become infected in the future, initially the virus will (apparently) sit quietly and wait for you to login to your bank account online and then steal the login credentials (username and password) you use to access your account, which once it knows all the details, can then be used to empty your bank account into the criminals bank accounts.

If this first attempt to steal money from you fails (maybe you don’t use online banking, you don’t have a bank account or you don’t have much money in your bank account), or isn’t rewarding enough for the criminals behind this (who knows what constitutes enough money), then the second phase of the virus attack will kick in (CryptoLocker virus).

This second phase will encrypt the interesting user data on your computer (spreadsheets, documents, databases, pictures, email files etc) and then throw up a Ransom Demand screen asking you to pay around $300 in order to obtain the key to decrypt your data.

If you don’t pay the ransom demand within the time indicated on the Ransom Demand screen (showing an ever reducing count-down clock), then the key that can be used to decrypt your data will be deleted and you won’t be able to recover your data unless you have a backup of your files somewhere (if you use services such as DropBox or SugarSync or any other service that syncs your files into the Cloud, then this DOESN’T constitute a proper backup).

Could I already be infected and not know it?

Yes – in the UK it is estimated that around 15,000 computers will already be infected, worldwide, this is thought to be in the millions.

The infected computers will no doubt try to harvest email address from the local Windows address book / Outlook contacts and then send out an infected email to those locally harvested addresses.  Those recipients, unless they have their wits about them, may think the email is a genuine email because it comes from someone they know and of course open it, open the attachment and then they will be infected and then the process starts again.

If you are already infected, then your Internet Service Provider (ISP) may contact you (if the rumours are true) and tell you that you are infected.  IF YOU HEAR FROM YOUR ISP – DO NOT IGNORE THE WARNING!

What can I do about it?

McAfee have kindly produced a tool to scan for and remove the infection from an already infected computer and this can be downloaded here.  There is no harm in downloading the tool right now and checking your machine even if your ISP doesn’t contact you, so why not err on the side of caution and check your computer anyway?  This should make sure you aren’t currently infected.

Once you know you are clean, the best advice is to buy an external hard disk drive or a large capacity memory stick and backup ALL your critical personal data to the disk / memory stick and then unplug the disk / memory stick and keep it somewhere very safe.

If the disk / memory stick is kept connected to your computer, then the data on that will also become encrypted if you subsequently become infected, so keep your backed up data completely isolated from your computer and you should be fine.

Worst case, if you do get infected after you have taken your backup, then the virus can be stopped and you can recover your data from your external disk.

If you don’t backup your data and you do become infected, then there is still a small chance of recovering your files if you have a feature called Shadow Copies enabled on your computer (see the link to the left to find out how to enable them).

If you aren’t already infected, well done.  You should still backup your files and remain ever vigilant when opening new emails, even from people you already know that contain attachments or links to sites.

What’s in it for the criminals?

Well – the Cryptolocker virus that reared it’s head around October last year has supposedly netted the criminals around £60m from their ransom demands and even some Police forces have had to pay the ransom to get their data back, so clearly it’s well worth their while writing the virus and setting it loose into the world and no one is immune from attack.

If 1% of the supposed million + computers that are infected pay the ransom demand, then that’s about $3m in the bank.  Add to that the amount from bank accounts that get emptied, which presumably will have more than $300 in them, then if 5% of an infected 1 million computers who have $500 in their account get emptied, then we are talking about $125m in income alone.

You have been warned!

2013 in review

The WordPress.com stats helper monkeys prepared a 2013 annual report for this blog.

Here’s an excerpt:

The Louvre Museum has 8.5 million visitors per year. This blog was viewed about 160,000 times in 2013. If it were an exhibit at the Louvre Museum, it would take about 7 days for that many people to see it.

Click here to see the complete report.

CryptoLocker Ransom Virus Cleanup

In case you haven’t discovered the CryptoLocker virus, it is a particularly nasty virus that sits unannounced on your computer and basically encrypts a whole variety of useful files that you would save with the following file extensions:

*.odt, *.ods, *.odp, *.odm, *.odc, *.odb, *.doc, *.docx, *.docm, *.wps, *.xls, *.xlsx, *.xlsm, *.xlsb, *.xlk, *.ppt, *.pptx, *.pptm, *.mdb, *.accdb, *.pst, *.dwg, *.dxf, *.dxg, *.wpd, *.rtf, *.wb2, *.mdf, *.dbf, *.psd, *.pdd, *.pdf, *.eps, *.ai, *.indd, *.cdr, *.jpg, *.jpe, img_*.jpg, *.dng, *.3fr, *.arw, *.srf, *.sr2, *.bay, *.crw, *.cr2, *.dcr, *.kdc, *.erf, *.mef, *.mrw, *.nef, *.nrw, *.orf, *.raf, *.raw, *.rwl, *.rw2, *.r3d, *.ptx, *.pef, *.srw, *.x3f, *.der, *.cer, *.crt, *.pem, *.pfx, *.p12, *.p7b, *.p7c

The virus encrypts the files using a Public AND a Private key.  The only way to decrypt the files is to pay the $300 ransom asked of you which basically provides you with the Private Key to decrypt your files.  The virus gives you a countdown warning that if you don’t pay the ransom within the ever diminishing time, then the Private key will be destroyed and your data will be toast!

Having been called to deal with another CryptoLocker virus discovered on a new customer’s computer recently, the damage to the server was overcome by restoring all data from a recent backup but I was now looking at cleaning up the infected client’s computer (which had been removed from the LAN and switched off).

To stop the warning message appearing, I used Rkill (the iexplore.exe version – I find this works more often that using the rkill.exe version) to highlight the random .exe file that is running, but as there are usually two copies of the file running, once Rkill identifies the name of the file, it will kill one of the processes, but the 2nd will spawn another version of itself and so you find that the processes are still running.

A quick DOS TASKKILL command will kill both processes off at the same time (taskkill /f /im randomexefilename.exe)

Once the processes have been stopped, I then use Roguekiller to identify and clean up the computer, noting the name of the two .dll files that are created in c:\users\USERNAME\appdata\roaming and then scour the registry for those .dll files and the original randomexefilename, removing any traces that are found.  I also check to make sure the .dll and .exe files are removed too – you can’t be too careful.  The .exe file usually resides in c:\users\USERNAME\appdata\roaming\randomfoldername.

Once the registry is clean, you can then look at recovering local files from the Shadow Copies.  There is a good write up on BleepingComputer on how to do this, so I won’t expand on what’s written there.

During my cleaning of the computer today, I noticed that the folder that contained the randomexefilename.exe file was created back in July 2012 and the two .dll files in appdata\roaming were created on the 1st October 2013.  As I had a vague idea the infection came in via Email, I looked for odd emails on the 1st October in the users mailbox and noticed that there were several emails with the Subject: “Your Amazon.co.uk order #” which all contained attachments and were all .ZIP files.  Having deleted all those emails and checked the Anti-Spam logs on the server, the emails appeared to come from Hotmail.com accounts, so I tweaked the filters on the Anti-Spam Software on the server (Vamsoft ORF Fusion) to block all .ZIP file attachments (except from trusted sources).

So it would appear that this particular virus, or at least the origins of it may have been hiding dormant in the customer’s computer since July 2012 and then the opening of the .ZIP file attachment on the 1st October added more to the virus and then it finally completed its encryption of files a week or two later at which point it popped up its ransom demand.

This particular customer was lucky because they backup their files nightly and Shadow Copies were enabled on the client computer, so encrypted files could be recovered completely, but if you are reading this now and you have the warning and don’t have a backup, then you will need to pay the $300 ransom to get your files back.  If you kill the virus off and tidy up after it and don’t have a backup, you can kiss your files goodbye permanently.

Alan

How we recovered a stolen laptop with help from GFI Remote Monitoring, Prey Project, a dash of cunning and a little bit of luck!

A customer of ours recently called us up to request a quote for a replacement laptop because the one we had bought them only a few months earlier had been stolen in a burglary and they didn’t expect to ever see it again. The laptop (running Windows 7) was configured with Office 365 and the password was cached, so we reset the password for the account to prevent anyone abusing the account and at that point we thought we couldn’t do anything else to help.

The thought of getting the laptop back was the furthest thing from our minds (and our customers) because there was no software on it that we could use to gather information that could be used to recover it (or so we thought). We therefore started to search for a suitable replacement laptop and passed on the prices to our customer accordingly.

It was only after having a discussion in the office that Mark, my business partner mentioned about PreyProject and what a shame that it wasn’t already installed on the laptop. He did a bit of digging around on their website and found that there was a batch file that could silently install the software if only we could get the software on to the laptop.

We currently use GFI Max RemoteManagement to monitor our customers servers, computers and laptops and we could see that the laptop had been connected to the Internet on a new IP Address, so we started to record the IP Address (screen-shots of the laptop in the GFI Control Panel) and passed the information on to the Police. The Police would then be able to use the IP Address to trace the user at the date/time we recorded it and from that, trace the address and hopefully the laptop. That would all take time though and it was possible that the laptop would be moved to a different location and therefore getting the laptop back would take time and might not happen at all. GFI also records a multitude of information about the hardware including the Serial Number, Make and Model and I also passed this information to the Police to identify the laptop should they eventually get the opportunity to recover it.

Thinking more about GFI and what was available to us, one of the options available is to use a Script Check to perform remote commands. If only we could somehow come up with a script to remotely download and install PreyProject then we might be able to do more than just trace the IP Address, we could possibly get some web-cam pictures of the person using the laptop, some screen-shots of what they were doing and local Wi-Fi networks that were in the vicinity of the laptop. If only……..

So I set about writing a script that could download the .exe file and the batch installation file using FTP from my Draytek router (with memory stick plugged into it) and tested this locally, which worked very well. Testing the same script at a different location unfortunately didn’t work and so I gave up on the FTP route and searched for an alternative. The alternative that I came up with was to use BITSADMIN (Background Intelligent Transfer Service), which is used by Windows to download files for things like Windows Update and was of course installed on the laptop by default. After uploading the PreyProject .exe file and their installation batch file to our website, I wrote a script to download the files to the laptop and then run the installation. I then tested the script out on my laptop and it worked successfully.

My script which I eventually used looked like this:
@echo off
md c:\temp
bitsadmin /transfer myDownloadJob /download /priority high http://www.mywebsite.com/preyinstall.bat c:\temp\preyinstall.bat
bitsadmin /transfer myDownloadJob /download /priority high http://www.mywebsite.com/prey-0.5.3-win.exe c:\temp\prey-0.5.3-win.exe
c:
cd c:\temp
preyinstall MY_Prey_API_Key

(any script writers out there please forgive the very basic nature of the script – I am no batch-script writer – but it works!)

Having uploaded the script to GFI, I then assigned the script check to the stolen laptop and waited.

After a short while, the script came back with a Timeout Error. At that point I was disappointed because I had scheduled the script with the maximum timeout value of 150 seconds and it must have taken longer to run that the 150 seconds. I then set about checking the script to make sure all was well and couldn’t see a problem. I even started timing the downloads and for me it all worked well within the 150 seconds. So presumably the current laptop user was using a slower connection.

I decided to login to the Prey Project Control Panel to make sure there was space for the stolen laptop to be installed (on a free account you get 3 free spaces to monitor devices and I knew that I had one space left). I was very surprised at that point to see the stolen laptop appear in the list of Devices I could monitor and so I reported it as stolen, configured the settings to geo-location information, grab web-cam shots, screen-shots, Wi-Fi networks and anything else it could to help me locate the laptop. There are additional options available to lock the device, have the device make a noise, display a warning message on the screen and to hide emails, delete browser cookies and stored passwords, but I decided to leave those alone for now as I didn’t want to scare off the laptop user, I wanted to get the laptop back.

I set the PreyProject monitoring interval to the smallest interval available and waited to see what came back into the 10 available reporting slots available on a free account (this can be increased for a paltry $5.00 a month to a 2 minute interval and 100 reporting slots).

What initially came back was a very dark image of the laptop user playing games and their location was reported as being in Central London (it was night-time so the lighting wasn’t brilliant). I flagged my laptop as being stolen and waited for the first report back from that and when it came back, I was reported as being right next to the stolen laptop, which clearly wasn’t the case, so I didn’t pay much heed to the geo-location information and eventually turned it off.

I sat back and waited for more reports to come in and was rewarded the next time with a slightly clearer picture of the laptop user who was busy watching porn!

After a while, it seemed that the laptop user was aware of the webcam being used and the images then came back completely black, so I turned off the web-cam setting and continued to grab screen-shots and wi-fi networks etc.

The next day (during daytime), I turned the web-cam back on and ended up getting a very clear picture of the laptop user which I then passed along to the Police.

I continued to enable / disable the web-cam option and as my available slots for reports were filling up, decided I didn’t want to lose any good images of the laptop user, so upgraded my account and then increased the monitoring interval.

After several not very exciting reports with no new information, I then received the most interesting screen-shot that was to give the laptops location away precisely. They were writing a letter and had put their address and telephone number at the top of the letter and I had a screen-shot of the letter. The location wasn’t too far away from my own location, so I decided to pay the address a visit and took my iPhone with me to verify the Wi-Fi networks that were shown in the report.

Standing outside the address in the letter I turned on my Wi-Fi on my iPhone and took some screen-shots of the available networks and 5 of the networks that were captured using PreyProject matched the networks I could see. The laptop HAD to be close and thus the address on the letter had to be the laptop users address. I excitedly passed this information on to the Police and they then arranged for a Search Warrant from the Courts the next day and then paid the address a visit the following morning.

I then received a phone call from my customer to say that they had been contacted by the Police and that they could collect their laptop from them (with some proof of their identity) the following day, which they did. We are now restoring the laptop back to a pre-stolen date to remove traces of software that was installed during its absence and we heard that the insurance company was not going to pay out for the laptop because it was a work laptop and was not therefore technically covered on the household insurance from the house it was stolen from, so it was just as well we got it back.

So – if you don’t already have PreyProject installed on your iPad, iPhone, Laptop, Computer, Android phone, Apple Mac, Linux PC or anything else that it can be installed from, then what are you waiting for. If it gets stolen without PreyProject installed, you had better be one of our customers with Monitoring software installed or you can kiss it goodbye!

Apple release iOS 6.1.1 to fix one bug but it doesn’t fix the bug with Exchange

After Apple released iOS 6.1 on the 28th January 2013, numerous people have complained of various issues with 3G connectivity, others have complained about battery life being reduced dramatically and more recently, Exchange servers around the world have been slowing down due to what appears to be a problem with the devices looping when Calendar Appointments are accepted on the iPhone / iPad.

Today Apple has released iOS 6.1.1 (only for the iPhone 4S) which seems to address the 3G issues, but it doesn’t fix the Exchange issues and Microsoft / Apple are working together on the problem to see if it is an Exchange issue or an Apple issue.

So whilst some can upgrade, not everyone can and even those that can upgrade, may well have to upgrade yet again when a new update is released that fixes the Exchange issue.

Exchange Admins all over the world are probably restricting access to their Exchange Servers for those who have upgraded to iOS 6.1 until they delete and re-create their Exchange Accounts and promise not to do anything with Exchange Calendar Appointments (in terms of Accepting / Declining etc). Once they have deleted and added their account back, the Admins may allow them back on the server as this is rumoured to ease the performance issues that the Exchange servers are suffering.

The iOS 6.1.1 release is 968Mb in size, so it isn’t a small download. If you are not suffering from battery / 3G issues, you may as well wait to see if there is a newer release and download that instead.

I for one (with my iPhone 4S), have only just upgraded to iOS 6.1 but won’t be updating to iOS 6.1.1 because I can’t face the hassle of Jailbreaking it all over again and re-install/configuring my Jailbroken apps so recently after Jailbreaking iOS 6.1, only having to do it again when 6.1.2 or 6.2 (or whatever comes next) is released to fix the problem, assuming it lies with Apple and not Microsoft.

Watch this space for more news as and when it becomes available.

Alan

Update WSUS to show Windows 8 Computers as Windows 8 not Windows XP

If you have a server running Windows Server Update Services 3.0 SP2 (SBS 2003 / SBS 2008 / SBS 2011 etc) and you also have some Windows 8 clients that you have joined to the domain, they will probably show up in WSUS as Windows XP Clients not Windows 8!

To resolve this, please install the following patch from Microsoft:

http://support.microsoft.com/kb/2734608/en-us

Once installed, you should see the Windows 8 clients reported as Windows 8.