#1 2017-05-16 15:54:11

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Phoenix (used to be Dallas) issues [RESOLVED]

The node got hit with a huge ddos attack, much to everyone's surprise it has been tracked down to yet another irc user from Italy...

The load is so high I doubt this will be recoverable, assume a reboot will occur soon.

******************************** EDIT & UPDATE ********************************************


In case you did not read the email that was sent to you, the quick and short version:

All data was lost.

reinstall via solusvm to get back up and running.

re-enable tun/tap if you need it in solusvm.

remove and re-add your ipv6 addresses in solusvm.

your redirected ipv4 ssh port ends in 21 e.g. 192.168.0.151 would use port 15121 on the external IP.

For further info please read this forum thread.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#2 2017-05-16 16:11:41

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

I was forced to do a hard reset and sadly now it will not boot.

Going in to full recovery mode now, expect about 24 hours of down time.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#3 2017-05-16 16:27:12

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Seems to have just been a dead grub, odd, booting now.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#4 2017-05-16 16:53:27

WSS
Trusted Member
Registered: 2016-12-22
Posts: 286

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Grub is so 2008.  BRING BACK LILO!


RbyeR4Nm.png

Offline

#5 2017-05-16 17:16:39

Bochi
Trusted Member
Registered: 2016-09-05
Posts: 48

Re: Phoenix (used to be Dallas) issues [RESOLVED]

As I said the last time, I still can't believe the stupid amount of issues that arise from IRC...

Offline

#6 2017-05-16 17:26:52

WSS
Trusted Member
Registered: 2016-12-22
Posts: 286

Re: Phoenix (used to be Dallas) issues [RESOLVED]

People pay pennies to use someone else's equipment to be the huge dickholes they generally are, and when the machine gets attacked, they've lost nothing- meanwhile, Ant/David/et al get a load of shit from us other cheap fucks who want their crappy services online.

Banning IRC is a great idea, but sadly, I kind of wanted to use one of mine to ping EFnet every couple years.  Guess I can do that from my $5 Kimsufi..


RbyeR4Nm.png

Offline

#7 2017-05-16 17:36:48

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Spoke too soon, throwing up a load of filesystem errors now, joy.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#8 2017-05-16 18:06:24

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

This is not looking good at all, hope everyone has backups.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#9 2017-05-16 19:16:02

Bochi
Trusted Member
Registered: 2016-09-05
Posts: 48

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Backtogeek wrote:

This is not looking good at all, hope everyone has backups.

Nope, but no complicated setup that couldn't be replicated within half an hour. wink
All the best for the recovery process, to whatever extent they make sense and thanks again for all the efforts you are putting into this project! smile

Offline

#10 2017-05-16 19:19:56

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

partition table is messed up, can't recover superblock, testdisk is detecting some files and only half the partitons (deep search done) but recovery fails, I am going to give this a few more hours and then put it on my list of things to do tomorrow, full rebuild and reprovision.

You can be sure IRC will be blocked when it comes back up either way.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#11 2017-05-16 19:24:54

Bochi
Trusted Member
Registered: 2016-09-05
Posts: 48

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Always kinda fascinating how this kind of errors come up triggered by another incident and a reboot. tongue

Offline

#12 2017-05-16 19:31:29

AlesioRFM
Trusted Member
Registered: 2017-01-21
Posts: 11

Re: Phoenix (used to be Dallas) issues [RESOLVED]

So glad I made a backup, I hope the guy who did this is banned from ever getting a new vps here

Offline

#13 2017-05-16 19:32:38

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

1 Minute : 2197.16
5 Minute : 2052.50
15 Minute : 1623.78

It was this point I had no choice but to do a hard reset, it booted after I fixed /boot & grub, from what I can tell kernel care was involved somehow in the disaster, got around a 3rd of the containers booted, it spewed out a load of inode errors and went read-only, the whole system hung resulting in another hard reset and then much to my horror all anything can see is /dev/sda with no partitions.

Looking at options now, if I have to manually recreate the containers it may take a few days to resolve.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#14 2017-05-16 19:33:49

zwv970
Trusted Member
Registered: 2016-01-30
Posts: 77

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Could someone explain how DDoS can cause severe filesystem corruption?

Offline

#15 2017-05-16 19:38:03

Bochi
Trusted Member
Registered: 2016-09-05
Posts: 48

Re: Phoenix (used to be Dallas) issues [RESOLVED]

zwv970 wrote:

Could someone explain how DDoS can cause severe filesystem corruption?

It won't by itself, but as I understand some filesystem error that already existed got then manifested on a reboot of the machine.

Offline

#16 2017-05-16 19:42:59

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

zwv970 wrote:

Could someone explain how DDoS can cause severe filesystem corruption?

I don't think the DDOS corrupted the file system, the fs issues are resulting symptoms from the actions that followed as explained already.

Load your own PC up to the point it can barely respond to anything, pull the power cord out of your running desktop PC a few times all while it is part way through performing updates, it's a coin toss at best as to its being able to boot properly again.

Don't get me wrong, I am about 95% sure with enough effort I can recover the files from the vz partition, however, the amount of time and work involved in this will take significantly longer than a reinstall and the creation fresh containers.

The reinstall and recreation would need to be done anyway as well as putting everyone's data in the right place so you would be looking at a week, to be blunt on LES the significant amount of time to do this is not justifiable.

I have not quite given up yet though.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#17 2017-05-16 19:44:57

zwv970
Trusted Member
Registered: 2016-01-30
Posts: 77

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Bochi wrote:
zwv970 wrote:

Could someone explain how DDoS can cause severe filesystem corruption?

It won't by itself, but as I understand some filesystem error that already existed got then manifested on a reboot of the machine.

Should there be an automatic periodic maintenance window to check filesystem and/or general hardware status?

Offline

#18 2017-05-16 19:56:07

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

zwv970 wrote:
Bochi wrote:

It won't by itself, but as I understand some filesystem error that already existed got then manifested on a reboot of the machine.

Should there be an automatic periodic maintenance window to check filesystem and/or general hardware status?

Let's assume I don't do regular checks and maintenance for a second, I do but let's pretend I don't, what specifically do you thnk would have changed?


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#19 2017-05-16 19:56:54

zwv970
Trusted Member
Registered: 2016-01-30
Posts: 77

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Backtogeek wrote:
zwv970 wrote:

Could someone explain how DDoS can cause severe filesystem corruption?

Load your own PC up to the point it can barely respond to anything, pull the power cord out of your running desktop PC a few times all while it is part way through performing updates, it's a coin toss at best as to its being able to boot properly again.

Not trying to suggest anything but if system is under external load then removing the external load (pull network cable instead of power cable) quickly returns system back to be responsive.

Offline

#20 2017-05-16 20:05:48

manjana
Trusted Member
Registered: 2015-11-23
Posts: 120

Re: Phoenix (used to be Dallas) issues [RESOLVED]

zwv970 wrote:

but if system is under external load then removing the external load (pull network cable instead of power cable) quickly returns system back to be responsive.

giphy.gif

Anyway, good luck with the recovery, Ant. Hope this mess doesn't eat up too much of your time sad

Offline

#21 2017-05-16 20:08:23

rewbycraft
Member
Registered: 2016-05-23
Posts: 1

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Ah. This situation explains what's been up with my ex-Dallas box these last few weeks.
(It's been dead for a while, with the control panels reporting it to be active but with "null" memory usage...)
I honestly don't particularly mind having to recreate the box. I may not have backups, but there was never anything on it that can't be rebuilt by my configuration management.

Anyway, many thanks for all your time/effort Backtogeek. I'm all too aware I don't pay you enough for the time and effort you put into LES.
I hope you get the server patched up soon.

As for IRC related DDoSes... I've got plenty of tales about those. (I have worked at/on public IRC "bouncer" hosting services before.)
So I feel your pain on that front. I understand your choice to block IRC.

Know that, although you have plenty of *******s on this service, there are still a fair few of us who appreciate your effort and your service!

Offline

#22 2017-05-16 20:12:20

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

I really wish it was that simple.

The network was nulled pretty quick (cable pulled if you like) all subsequent work was done via the console, so the load hits 1000, the vz service fails, all containers hang and processes start escaping to init which means you can't stop any containers but the processes are still running and they keep spawning new ones as a consequence.

The load continues to reach 2000 with no signs of stopping because nothing can complete and processes are getting orphaned right left and center, even a cursor move on the console takes 10 minutes.

What do you do next?

The scale is a huge factor here.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#23 2017-05-16 20:17:08

Backtogeek
Low End Boss
From: ~/
Registered: 2013-02-13
Posts: 3,547
Website

Re: Phoenix (used to be Dallas) issues [RESOLVED]

rewbycraft wrote:

Ah. This situation explains what's been up with my ex-Dallas box these last few weeks.
(It's been dead for a while, with the control panels reporting it to be active but with "null" memory usage...)
I honestly don't particularly mind having to recreate the box. I may not have backups, but there was never anything on it that can't be rebuilt by my configuration management.

The memory use was disabled in the solusvm panel after I found a bug, solusvm simply cannot cope with anything at scale and solusvm itself becomes the single biggest factor in performance degradation, it has nothing to do with this incident specifically.

I am at least hoping to get it to a point whereby all people need to do is click reinstall, if you have to wait for me to manually recreate everything in solusvm/whmcs it is going to take a while.


http://LittleHappyCloud.net KVM VPS with 1TB Bandwidth for €3.00

Offline

#24 2017-05-16 20:52:13

WSS
Trusted Member
Registered: 2016-12-22
Posts: 286

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Backtogeek wrote:

What do you do next?

The scale is a huge factor here.

init S and pray.. but if it's systemd, you just shrug and hit the kill switch.  Some shit is going to break.

Backtogeek wrote:

The memory use was disabled in the solusvm panel after I found a bug, solusvm simply cannot cope with anything at scale and solusvm itself becomes the single biggest factor in performance degradation, it has nothing to do with this incident specifically.

I am at least hoping to get it to a point whereby all people need to do is click reinstall, if you have to wait for me to manually recreate everything in solusvm/whmcs it is going to take a while.

As shitty as Solus is, I really wonder why Virtualizor is the only real contender for competition.  I know there are an endless array of needs for users, and even more so, vague configurations/etc/etc for networking, bridging, etc.. but damn- other than making it easier for people to reinstall their shit, it seems entirely worthless.

My personal stance: Anyone paying for a NON-LES, fix them.  LES? No SLA, No Reinstally.  Fire off an email to tell them they need to reinstall their shit, and ban anyone who bitches.

Last edited by WSS (2017-05-16 20:55:44)


RbyeR4Nm.png

Offline

#25 2017-05-16 21:43:24

V31
Trusted Member
Registered: 2017-05-16
Posts: 4

Re: Phoenix (used to be Dallas) issues [RESOLVED]

Backtogeek wrote:

You can be sure IRC will be blocked when it comes back up either way.

I've been using a vps on this node for over a year just for an IRC session (precisely, a personal quassel core).
How does IRC cause problems? Is it the spammers or irc hosts?
If you can, please consider at least whitelisting freenode. Thank you very much for all the effort put into this node.

Offline

Board footer