Note: most of this is rambling, go to the bottom for the useful technical bit.
I’m doing some work with large amounts of data (about 100 million rows), and today I tried loading it all into MySQL on my home machine, so that I could run some collaborative filtering tools over it easily. However, when I set this machine up, I did a dumb thing and gave / 4Gb and /home the rest of the space. Thinking of course, that / wouldn’t need much space. It just holds programs and stuff.
I’d forgotten that it’s also where MySQL puts its database files. It also turns out that when MySQL runs out of of disk space, it gets really unhappy and kinda belligerent. You can’t even get it working well enough to do a ‘drop database’ on the over-sized culprit until you free up some space somewhere else.
So, after a bit of playing I got that sorted, and decided to move /var somewhere else. This is a scary proposition, because it means resizing /home (with my 200Gb+ of accumulated cruft). But that’s OK, with careful use of resize_reiserfs and lvreduce (in that order! no exceptions!), I had 10Gb to put a new /var into. This is where it gets tricky. I created the new /var, copied the old stuff over, set the fstab to mount it, etc. I then rebooted to have everything clean up…and mysql wouldn’t start. I eventually tracked that down to a lack of networking.
Here start the useful technical bits, if you’re googling this or something:
However, networking either refused to start, or wouldn’t bring up lo, complaining about a lack of the /var/run/network/ifstate file. Well, true, it wasn’t there, and creating it made things come OK, until the next reboot. On bootup, I was also seeing messages like ‘special device /var/run does not exist’. After a bit of poking around the internet, I discovered that /var/run (and /var/lock) have a special filesystem mounted on them before /var is mounted. The upshot of this is that if you don’t have /var/run and /var/lock living on the /var that lies underneath of the mounted /var, things go haywire. Basically, if you don’t have these directories that are invisible 99% of the time your system is running, things break. That is something that really needs to go into a really early startup script.
A nice trick for creating them on a running system (found here) is this command:
mount --bind / /mnt && mkdir /mnt/var/run /mnt/var/lock && umount /mnt
I hope this stops someone else spending an unnecessary couple of hours trying to figure out why networking is failing mysteriously.
Why not just make /var/lib/mysql a symlink to somewhere like /home/mysql? It’s not perfect as a long-term solution but if all you’re doing is short-term experimentation it would have instantly meant that MySQL would have all the space it needed to do whatever it wanted, no mess.
I thought about that, but I figured it was worth fixing the problem properly, as it’s bitten me before (say a large system update, and the package cache fills up)
I experienced the same problem during the last days after moving my /var onto a seperate partition. This was on a Ubuntu 6.06 system. Your trick helped me out 🙂 Thanks!
Thanks for posting…
I Worry because the “special device /var/run does not exist” error appear when the system try to mount /var/run and /var/lock. I think than that are mounted in the / and not in the /var really partition.
Suggest ?
They need to exist on ‘/’, even though they will have ramdisk versions mounted over the top.