![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
its three in the fucking morning.
Dec. 16th, 2005 03:09 ami've been at the datacenter since 8:30.
first, there was the dell server with the corrupted /; and of course, its "the third most important machine in the network after the databases".. and it has no standby or backup or auxillary unit, of course.
so i get here and i start fixing it, which involves booting off a cd and then removing alot alot of detached files/inodes, then i gotta replace a whole shitload of stuff in /lib and /etc, and thank god there's another machine that happens to be the exact same build of linux or i would have had to reinstall the whole fucking thing.. then the external raid wont come up, so we reboot it again, then i find a note from my predicessor in / that says you have to do an insmod to see the external raid, then an hour of fsck.
by now i realize that one of our ATAbeasts is alarming loudly. great, one of the RAIDs went critical.. so i harvest a drive out of the other unit, so both units are not short two drives each. terrific. then oracle dumps core on the front-end database server. okay, i take it all in stride.
then the many many woes of dropping in a mysql server, and freebsd max mem size restraints.. and the fact that the fucking DBA isn't using RCS to edit the fucking my.cnf,..
now one of the index-server java thingers isn't starting so they want me to fsck the drive.. oh but the drive is /home.. ohh, linux really can't handle fscking a mounted filesystem, can it. dumb linux.
sigh.
3:45..
first, there was the dell server with the corrupted /; and of course, its "the third most important machine in the network after the databases".. and it has no standby or backup or auxillary unit, of course.
so i get here and i start fixing it, which involves booting off a cd and then removing alot alot of detached files/inodes, then i gotta replace a whole shitload of stuff in /lib and /etc, and thank god there's another machine that happens to be the exact same build of linux or i would have had to reinstall the whole fucking thing.. then the external raid wont come up, so we reboot it again, then i find a note from my predicessor in / that says you have to do an insmod to see the external raid, then an hour of fsck.
by now i realize that one of our ATAbeasts is alarming loudly. great, one of the RAIDs went critical.. so i harvest a drive out of the other unit, so both units are not short two drives each. terrific. then oracle dumps core on the front-end database server. okay, i take it all in stride.
then the many many woes of dropping in a mysql server, and freebsd max mem size restraints.. and the fact that the fucking DBA isn't using RCS to edit the fucking my.cnf,..
now one of the index-server java thingers isn't starting so they want me to fsck the drive.. oh but the drive is /home.. ohh, linux really can't handle fscking a mounted filesystem, can it. dumb linux.
sigh.
3:45..
