Category Archives: Enterprise level solutions

Subversion (SVN): How do you set up backup for your Subversion repositories?

Subversion is becoming more and more popular every day. To get quick installation guide, see my post on installing Subversion. Once you have Subversion set up, make sure you have backups of repository setup! This is one of those things most people forget to do. If you followed the installation guide and did source install, you already have a tool available to you. You can find tool in the tools/backup/ directory of the Subversion source distribution. One thing about this script is that it doesn’t backup all your repositories. Which might be an issue if you create more repositories but forget to create a script to backup that particular repository. Here is a very simple shell script to help you backup all of your subversion repositories. You can edit it to point to different /svn location if you didn’t install to default location. Make sure you copy to a location which is included in your path so your scripts will work.

mkdir /backups/repos -p
for repostobackup in `ls /svn/` ; do
filename=`basename ${repostobackup}`
if [ ! $filename = "" ]; then --archive-type=bz2 /svn/${filename} /backups/repos

This will create your backup files and compress them using bzip2. You can also use zip or gz if you prefer. Another thing you should do is to edit the script and change num_backups to be whatever you think is a good number. I personally use: num_backups = 10

That’s all there is to it. Just add this script to your crons and your backups will be automated. You may want to consider moving your backups to another server by adding ftp, scp or rsync at the end of the script.

MySQL: InnoDB: ERROR: the age of the last checkpoint is [number]

One of the mysql database servers I manage started to have issues with doing backups yesterday. mysqldump was running but nothing was happening on the backup side. I started to investigate to see why our full backups were failing. I opened up the mysql error log file (mine is at: /usr/local/mysql/var/hostname.err) and notice there were many instances of following error.

070815 15:31:46 InnoDB: ERROR: the age of the last checkpoint is 9433957,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

I poked around and found out our log files were set to default (5 mb) and they needed to be increased. After doing some calculations and research, I decided going to 50 mb at this point would be a good way to go. So I made the change in my.cnf file and added: innodb_log_file_size = 50M

I then stopped our mysql server and restarted it. And to my horror I saw following error messages show up in the mysql error logs:

070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'
070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'

I stopped the mysql server right away. I then remember, by help of a friend, that I have to remove the old log files if I change the size by changing innodb_log_file_size. So I issued a command: mv ib_logfile? ..

This command moves both ib_logfile1 and ib_logfile2 down one directory. I didn’t want to just remove ’em so instead I moved them. After that, I restarted mysql again and to my comfort, everything came back up without errors. So lesson learned: if you change ib_logfile size by using innodb_log_file_size setting, make sure you move existing log files before you start mysql again.

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Cron: crond will not read/honor new entries in crontab file. (CentOS, Redhat, Fedora)

I ran in to something really odd. I have been using crontab for LONG time and never seen this issue before. Basically I added new entry into my crontab by type crontab -e and as always saved/exited. For some reason that command never ran at the time I specified. I tested few more times and no luck. So I started looking on google to find out why crond will not ready my new entries from crontab. And then while I was looking, I decided to restart crond by issuing: service crond restart and to my surprise, my new schedule started working.

While I was looking around for solution for this problem on Google, I came upon few pages where they talked about if you have empty line before command, that command will not work. For example:

#this is a cron file

* * * * * /do/something

In this example, /do/something will not run since there is a blank line before it. You can fix this by moving that command up or put another comment above it. This is a very odd thing and I have never experienced it myself but I figured if you stumble upon this post looking for help, blank line before command in crontab might be your issue. Please comment if you had to delete a blank line to fix your crontab file. Also mention which OS you are using. To learn more about crond, see man crond and man crontab.

SSH: Getting error ssh_exchange_identification: Connection closed by remote host

Today I was working on setting up public key authentication over SSH on few servers I inherited recently.  I have done this many times and never ran in to following error:

ssh_exchange_identification: Connection closed by remote host

After doing further research on the server I was trying to connect to (looking at logs) I noticed the server was denying connection based on ip.  I put my servers ip in /etc/hosts.allow and tried again.  This time I was connected without any problems.  Note that you do not have to restart SSH for hosts.allow change to take affect.

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

SSH: Setting up public key authentication over SSH (CentOS, Redhat, Linux, Fedora)

Before we go in to details on how to set up public key authentication, I would like to talk about pros and cons about doing this. After you know the pros and cons, you can make an educated decision. First, let us talk about biggest benefit about setting up public key authentication over ssh. Once you set up the keys correctly, you will not need to enter password to access the other server. For example, you can just type ssh and you will be logged in. This is the method you must use if you want to set up scripts which are run from crontab. For example, you may have rsync script running which does backups from one server to another every night. With every good thing, comes a bad thing. So let us say “somebody” has access to server1 as root. This somebody could be designated user or it could be a hacker who has gain root access to server1. Now since we have public key authentication over ssh setup between server1 and server2, this “somebody” now can access server2. As you can see, this can be a problematic. Now you know number one pro and con about setting up public key authentication over ssh. Let us now go on and actually go through process to setup keys.

On server1, you type: ssh-keygen -t rsa (just press enter to keep accepting defaults). You should see something like this:

ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/
The key fingerprint is:
a2:b2:aw:w2:63:25:2a:62:fs:d5:ff:fd:11:f1:aa:60 [email protected]

Fingerprint for your server will be different. Now cat /root/.ssh/ Copy the contents it displays. You will need this for server2. Paste it into notepad or any other text editor and make sure it pastes as one line. This is very crucial you check this.

Now login to server2. Create the public key by typing: ssh-keygen -t rsa (this is same thing we did on server1). Once you are done, type vi /root/.ssh/authorized_keys2

Paste the content you copied earlier on server1. Make sure it is all on one line. Save/exit.  At this point, we should change the permissions on the file to make sure sshd likes it:  chmod 600 /root/.ssh/authorized_keys2

Go back to server1 and type ssh server2 You should be logged in with out having to put in a password. If you are prompted for password still, check server2 /root/.ssh/authorized_keys2 file and make sure everything you pasted is one line.

That is all you have to do set up public key authentication over ssh!

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.