Ukarainian Way of Rotating IPs

Note: original IP addresses are changed to the fake ones according to RFC 5737.

Once upon a time one of my ex-employers texted me a call for help. A recently installed dedicated server used to go offline time after time.

Challenge accepted.


Get the credentials, try to log in. Fail! Host seems to be down. Ping? 100% lost. nmap also reports the host to be down.

Well-well-well. Let's go to the control panel. Send WOL, no reaction. So, it's not shut down. It works but refuses to respond. Send Ctrl+Alt+Del, the host awakes in a minutes or so. Okay. Logs are willing to tell us what? Nothing. Really, no sign of any failure, neither in syslog, nor in dmesg. So, logs suggest basically nothing. What to proceed with? Googling, sure. Success! Almost exactly the same problem is found. Even the same NIC, RTL-8169. Okay, let's mess with ACPI and system clock.

Shutting down ACPI, switching system clock to another source gives nothing. So, back to square one. Roll back those unsuccessful attempts to bring some peace to the poor server, open syslog. Let's not just search for a failure, but read all of it.

A plenty of cron executions. Okay, filter them out. Boot logs, clean and clear. ntpd... Oh, wait.

Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1986]: ntpd 4.2.6p3@1.2290-o Tue Jun  5 20:12:08 UTC 2012 (1)
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: proto: precision = 0.838 usec
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen and drop on 1 v6wildcard :: UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen normally on 3 eth0 198.51.100.14 UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen normally on 4 eth0 fe80::216:17ff:fe90:b4ae UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen normally on 5 lo ::1 UDP 123
Oct  7 16:17:04 Ubuntu-1204-precise-64-minimal ntpd[1987]: peers refreshed
<...>
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: Listen normally on 6 eth0 192.0.2.50 UDP 123
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: Deleting interface #3 eth0, 198.51.100.14#123, interface stats: received=279, sent=279, dropped=0, active_time=6538 secs
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: 78.46.223.89 interface 198.51.100.14 -> (none)
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: 134.0.27.120 interface 198.51.100.14 -> (none)
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: 213.239.239.166 interface 198.51.100.14 -> (none)
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: 213.239.239.165 interface 198.51.100.14 -> (none)
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: 213.239.239.164 interface 198.51.100.14 -> (none)
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: peers refreshed
Oct  7 18:06:02 Ubuntu-1204-precise-64-minimal ntpd[1987]: new interface(s) found: waking up resolver

Wait, what? It's address is definitely 198.51.100.14. WTF is 192.0.2.50? How did it get to eth0?

Lets take another look.

Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1992]: ntpd 4.2.6p3@1.2290-o Tue Jun  5 20:12:08 UTC 2012 (1)
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: proto: precision = 1.117 usec
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: ntp_io: estimated max descriptors: 1024, initial socket boundary: 16
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen and drop on 0 v4wildcard 0.0.0.0 UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen and drop on 1 v6wildcard :: UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen normally on 2 lo 127.0.0.1 UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen normally on 3 eth0 198.51.100.14 UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen normally on 4 eth0 fe80::216:17ff:fe90:b4ae UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen normally on 5 lo ::1 UDP 123
Oct  7 14:22:48 Ubuntu-1204-precise-64-minimal ntpd[1993]: peers refreshed
<...>
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: Listen normally on 6 eth0 192.0.2.247 UDP 123
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: Deleting interface #3 eth0, 198.51.100.14#123, interface stats: received=265, sent=265, dropped=0, active_time=6195 secs
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: 109.75.188.245 interface 198.51.100.14 -> (none)
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: 131.234.137.23 interface 198.51.100.14 -> (none)
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: 213.239.239.166 interface 198.51.100.14 -> (none)
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: 213.239.239.165 interface 198.51.100.14 -> (none)
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: 213.239.239.164 interface 198.51.100.14 -> (none)
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: peers refreshed
Oct  7 16:06:03 Ubuntu-1204-precise-64-minimal ntpd[1993]: new interface(s) found: waking up resolver

New interfaces found, ahem. There was 192.0.2.50, there is 192.0.2.247. Where did they come from? chrootkit doesn't find anything, neither does rkhunter. Wait, in both cases ntpd reports new interfaces 6 minutes past an hour. Smells like cron. Oh, it's getting hotter.

There it is!

root@Ubuntu-1204-precise-64-minimal:~# crontab -l
6 */2 * * * /usr/bin/php5 -f /var/www/system/scheduled/checkpos_ua.php >> /var/www/scheduled/logs/checkpos_ua.log

Take a look inside, wouldn't we?

<?php
if(0 == posix_getuid()){
                $ip = rand(1,254);
                `/sbin/ifconfig eth0 192.0.2.$ip netmask 255.255.255.0`;
        }
?>

It's just a part of it, but the most relevant, I suppose.
The entire script checks positions of some sites in Google using some queries. And rotates IPs in order not to get banned too fast.
Rotates IPs, my arse! By reconfiguring the only address on the only external network interface in the datacenter that knows nothing about the subnet rotated addresses are taken from.

On the one hand, digging this cost me roughly a day, but on the other hand, it made my day. Could you image a piece of code to be that not portable?