Technology: Learn and Share » Enterprise level solutions

sshfs: How do you install sshfs and fuse? [CentOS/Linux/Redhat]

Sunny Walia — Tue, 07 Oct 2008 23:06:50 +0000

One may wonder what is sshfs and why would you want it? Well simply put, sshfs allows you to mount another server’s filesystem into a folder on your local system which in the background is doing ssh commands and transfers. As a mounted folder, you are able to move about and copy files back and forth as everything was on local server. As you can see this makes it very easy for you to work with files on multiple servers.

Note: you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz tar zxpfv fuse-*.gz cd fuse* ./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found configure: error: *** Please specify the location of the kernel source with *** the '--with-kernel=SRCDIR' option configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/ total 4.0K drwxr-xr-x 18 root root 4.0K Oct 7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64 make && make install cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz tar zxpfv sshfs* cd sshfs-fuse-* ./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old. Make sure it is in your PATH or set the PKG_CONFIG environment variable to the full path to pkg-config.

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0′ found
No package ‘gthread-2.0′ found

You need to install glib2. Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure make && make install

After installation is done, we can move on with testing the installation:

cd /mnt mkdir test sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
... sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test ...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL: How do you enable sphinxse (Sphinx Storage Engine) in your mysql installation?

Sunny Walia — Tue, 05 Aug 2008 22:21:32 +0000

As you may know mysql fulltext search is not highly scalable. One of the options to get around this scalability limitation, which I prefer, is to use Sphinx. You can use Sphinx with out having to alter your mysql installation. But, if you would like to use from within mysql and not have to worry about how to pass data between Sphinx and MySQL, you can enable sphinxse (sphinx storage engine). It is not included with mysql by default so you will have to compile it yourself.

Here are the instructions on how to get sphinxse compiled with your mysql installation on CentOS x64. I am sure same instructions will work for other flavors but I have not tested it. I will be compiling the most current version of sphinx (0.9.8) with most current stable version of mysql (5.0.51b) at the time of the writing. Let’s get the appropriate packages first:

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8.tar.gz wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/ tar zxpf sphinx* tar zxpf mysql*

You will also need “bison”, “patch”, “automake” and “libtool” installed. Let us just do a yum install for it.

yum -y install bison patch automake libtool

NOTE: if you don’t install bison, you will get the following error:
sed '/^#/ s|y\.tab\.c|sql_yacc.cc|' y.tab.c >sql_yacc.cct && mv sql_yacc.cct sql_yacc.cc sed: can't read y.tab.c: No such file or directory make[2]: *** [sql_yacc.cc] Error 2

Let us continue with patching mysql source with sphinx storage engine (sphinxse) code and compile/install our new binaries.

cd mysql* patch -p1 < ../sphinx-0.9.8/mysqlse/sphinx.5.0.37.diff #Make sure everything succeeded. BUILD/autorun.sh mkdir sql/sphinx cp ../sphinx-0.9.8/mysqlse/* sql/sphinx ./configure --prefix=/usr/local/mysql --with-sphinx-storage-engine make make install

Now start your mysql installtion and check if engine support is compiled in:

mysql> show engines\G Engine: SPHINX Support: YES Comment: Sphinx storage engine 0.9.8

To read more about how to use Sphinx storage engine, please refer to: Sphinx documentation for using sphinx storage engine

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL: How do you use symlinks with MySQL tables?

Sunny Walia — Fri, 02 May 2008 07:48:34 +0000

Creating table symlinks in MySQL is very easy and it is probably one of the features a lot of MySQL users overlook or never think about. You may ask, why would I want to use symlinks? Some of the reasons you would want symlinks is if you are running low on disk space on the partition where your data generally is and/or you want to move one (or more) table(s) on to a different disk/partition for performance reasons.

One of the things worth mentioning is that MySQL documentation states: “Symlinks are fully supported only for MyISAM tables. For files used by tables for other storage engines, you may get strange problems if you try to use symbolic links.” Keeping that in mind, if you have a innodb table and would like to create symlinks for, you should change the engine type to myisam before symlinking. Although, I have heard and personally used symlinks for innodb databases with no side affects.

First let us find out if your MySQL configuration allows symlinks:

mysql> SHOW VARIABLES LIKE 'have_symlink'\G *************************** 1. row *************************** Variable_name: have_symlink Value: YES

I am going to make following assumptions:

MySQL installation is in /usr/local/mysql.
You want to create new tables under /mnt/another_partition/mysql/var
1. you must create the dir: mkdir /mnt/another_partition/mysql/var -p
2. Make it owned by mysql user: chown mysql /mnt/another_partition/mysql/var

We will create some test data. Following are the steps:

mysql> create database db1; Query OK, 1 row affected (0.05 sec) mysql> use db1; Database changed mysql> CREATE TABLE `users` ( `id` int(11) default NULL, `name` varchar(255) default NULL ) ENGINE=myisam, data directory="/mnt/another_partition/mysql/var", index directory="/mnt/another_partition/mysql/var"; Query OK, 0 rows affected (0.09 sec) mysql> show tables; +---------------+ | Tables_in_db1 | +---------------+ | users | +---------------+

Drop back to command prompt and “ls” your db1 directory as shown below. Here you can see that users table’s data file (users.MYD) is a symbolic link to the location we specified. You can see that users index file (users.MYI) is symlinked to the location we specified as well.
# ls -al /usr/local/mysql/var/db1/ total 24K drwx------ 2 mysql mysql 4.0K May 2 00:03 ./ drwx------ 23 mysql mysql 4.0K May 1 23:59 ../ -rw-rw---- 1 mysql mysql 65 May 1 09:33 db.opt -rw-rw---- 1 mysql mysql 8.4K May 2 00:03 users.frm lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYD -> /mnt/another_partition/mysql/var/users.MYD lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYI -> /mnt/another_partition/mysql/var/users.MYI

Here are the real files which above symlinks are pointing to:

# ls -al /mnt/another_partition/mysql/var/ total 12K drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./ drwxr-xr-x 3 root root 4.0K May 1 23:39 ../ -rw-rw---- 1 mysql mysql 0 May 2 00:03 users.MYD -rw-rw---- 1 mysql mysql 1.0K May 2 00:03 users.MYI

If you go back in to mysql prompt and insert couple rows, you can see the size of users.MYD increase:

mysql> insert into db1.users values (1,"test1"); Query OK, 1 row affected (0.01 sec) mysql> insert into db1.users values (2,"test2"); Query OK, 1 row affected (0.01 sec) # ls -al /mnt/another_partition/mysql/var/ total 16K drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./ drwxr-xr-x 3 root root 4.0K May 1 23:39 ../ -rw-rw---- 1 mysql mysql 40 May 2 00:08 users.MYD -rw-rw---- 1 mysql mysql 1.0K May 2 00:08 users.MYI

If you already have data file and index file you would like to move, I recommend shutting down your MySQL server and moving the files manually and create symbolic links by using “ln -s” command. You can read more about ln by doing man ls

NOTE: Even though MySQL documentation says this is only supported 100% for myisam tables, I have read few posts where people say they have done this with innodb tables without any problems. If anybody else can comment on this, I would greatly appreciate it.

MySQL: Solution for ERROR 1442 (HY000): Can’t update table ‘t1′ in stored function/trigger because it is already used by statement which invoked this stored function/trigger.

Tracy — Tue, 04 Mar 2008 02:42:02 +0000

Here is a sample table you can create to test following problem/solution on:

CREATE TABLE `t1` ( `a` char(1) default NULL, `b` smallint(6) default NULL ); insert into t1 values ('y','1');

I have a table t1 which has column a and b, I want column a to be updated to ‘n’ when column b = 0. Here is the first version I created:

DELIMITER | CREATE TRIGGER trigger1 AFTER UPDATE ON t1 FOR EACH ROW UPDATE t1 SET a= 'n' WHERE b=0; | DELIMITER ;

The trigger created successfully but I got this error when I tried to do an update on column b on table t1:
mysql> update t1 set b=0; ERROR 1442 (HY000): Can't update table 't1' in stored function/trigger because it is already used by statement which invoked this stored function/trigger.

After searching online for a while and trying different solutions, I finally found a way to update the table which has trigger on it:
drop trigger trigger1; DELIMITER | CREATE TRIGGER trigger1 BEFORE UPDATE ON t1 FOR EACH ROW BEGIN IF NEW.b=0 THEN SET NEW.a = 'n'; END IF; END | DELIMITER ;

After the new trigger is in, I issued the same update query and “ERROR 1442 (HY000): Can’t update table ‘t1′ in stored function/trigger because it is already used by statement which invoked this stored function/trigger.” didn’t show up and it updated the col a value to “n” as it suppose to.

mysql> update t1 set b=0; Query OK, 1 row affected (0.01 sec) Rows matched: 1 Changed: 1 Warnings: 0 mysql> select * from t1\G *************************** 1. row *************************** a: n b: 0

Therefore, if you want to create a trigger on the table which will update itself, make sure you use the NEW.column_name to refer to the row after it’s updated and don’t do the full update statement!

However, if you are updating some other table, then you can use the regular update statement:

DELIMITER | CREATE TRIGGER trigger1 AFTER UPDATE ON t1 FOR EACH ROW UPDATE t2 SET a= 'n' WHERE b=0; | DELIMITER ;

MySQL: How do you set up master-master replication in MySQL? (CentOS, RHEL, Fedora)

Sunny Walia — Fri, 29 Feb 2008 21:40:02 +0000

Setting up master-master replication in MySQL is very similar to how we set up master/slave replication. You can read up about how to setup master/slave replication in my previous post: How to set up master/slave replication in MySQL. There is obviously pros and cons about using master/master replication. But this is not a post which discuses advantages and disadvantages for using master/master replication. One of the differences between master/master set up and master/slave is that in master/master set up, you have true redundancy. If one server dies, second server can take all the inserts/selects. In master/slave setup, if master dies, you will have to go through steps to make slave become the master. Master/master set up we are going to set up is essentially master/slave and slave/master. Meaning, if you had two servers, db0 and db1, your setup will be db0(master)/db1(slave) and also db0(slave)/db1(master). Here are some assumptions:

Master1 server ip: 10.0.0.1 Master2 server ip: 10.0.0.2 Slave username: slaveuser Slave pw: slavepw Your data directory is: /usr/local/mysql/var/

Let us go through the steps you must take on Master1 to enable it to act as master and slave by using following configuration which goes under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers auto_increment_increment=2 auto_increment_offset=1 # Replication Master Server # binary logging is required for replication log-bin=master1-bin binlog-ignore-db=mysql binlog-ignore-db=test # required unique id between 1 and 2^32 - 1 server-id = 1 #following is the slave settings so this server can connect to master2 master-host = 10.0.0.2 master-user = slaveuser master-password = slavepw master-port = 3306

Following is the configuration which goes on master2 under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers auto_increment_increment=2 auto_increment_offset=2 # Replication Master Server # binary logging is required for replication log-bin=master2-bin binlog-ignore-db=mysql binlog-ignore-db=test # required unique id between 1 and 2^32 - 1 server-id = 2 #following is the slave settings so this server can connect to master2 master-host = 10.0.0.1 master-user = slaveuser master-password = slavepw master-port = 3306

On master1 server, go to mysql> prompt and add the appropriate user:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

On master2 server do the same but allow right ip:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.1' identified by 'slavepw';

Restart both of the master servers and check slave status:

mysql> show slave status\G

That is all you have to do to set up the replication. Of course there are a lot more configuration options but this should get your replication going and you can tweak from here on.

MySQL: How do you set up master-slave replication in MySQL? (CentOS, RHEL, Fedora)

Sunny Walia — Wed, 30 Jan 2008 03:12:57 +0000

Before we go into how to set up master-slave replication in MySQL, let us talk about some of the reasons I have set up master-slave replication using MySQL.

1) Offload some of the queries from one server to another and spread the load: One of the biggest advantages to have master-slave set up in MySQL is to be able to use master for all of the inserts and send some, if not all, select queries to slave. This will most probably speed up your application without having to diving into optimizing all the queries or buying more hardware.

2) Do backups from slave: One of the advantages people overlook is that you can use MySQL slave to do backups from. That way site is not affected at all when doing backups. This becomes a big deal when your database has grown to multiple gigs and every time you do backups using mysqldump, site lags when table locks happen. For some sites, this could mean that site goes down for few secs to minutes. If you have slave, you just take slave out of rotation (should be built into code) and run backups off the slave. You can even stop slave MySQL instance and copy the var folder instead of doing mysqldump.

Ok let us dive into how to setup master-slave replication under MySQL. There are many configuration changes you can do to optimize your MySQL set up. I will just touch on very basic ones to get the replication to work. Here are some assumptions:

Master server ip: 10.0.0.1
Slave server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Put the following in your master my.cnf file under [mysqld] section:

# changes made to do master server-id = 1 relay-log = /usr/local/mysql/var/mysql-relay-bin relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index log-error = /usr/local/mysql/var/mysql.err master-info-file = /usr/local/mysql/var/mysql-master.info relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info datadir = /usr/local/mysql/var log-bin = /usr/local/mysql/var/mysql-bin # end master

Copy the following to slave’s my.cnf under [mysqld] section:

# changes made to do slave server-id = 2 relay-log = /usr/local/mysql/var/mysql-relay-bin relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index log-error = /usr/local/mysql/var/mysql.err master-info-file = /usr/local/mysql/var/mysql-master.info relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info datadir = /usr/local/mysql/var # end slave setup

Create user on master:
mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

Do a dump of data to move to slave
mysqldump -u root --all-databases --single-transaction --master-data=1 > masterdump.sql

import dump on slave
mysql < masterdump.sql

After dump is imported go in to mysql client by typing mysql. Let us tell the slave which master to connect to and what login/password to use:
mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.1', MASTER_USER='slaveuser', MASTER_PASSWORD='slavepw';

Let us start the slave:
mysql> start slave;

You can check the status of the slave by typing
mysql> show slave status\G

The last row tells you how many seconds its behind the master. Don’t worry if it doesn’t say 0, the number should be going down over time until it catches up with master (at that time it will show Seconds_Behind_Master: 0) If it shows NULL, it could be that slave is not started (you can start by typing: start slave) or it could be that it ran into an error (shows up in Last_errno: and Last_error under show slave status\G).

Memcached: How do you install memcached? (CentOS 64 bit, Linux, Redhat, Fedora)

Sunny Walia — Sun, 20 Jan 2008 22:52:32 +0000

[updated 5/16/2010]

Memcached is a very popular open source object caching server. It was developed to speed up livejournal.com by Danga Interactive. We use memcached for a lot of our sites. We use it for different purposes but one main purpose is to cache query results so we don’t have to keep hitting database. As most of the people who work with databases know it is costly to keep hitting database for same information over and over.

When you run the Memcached daemon, it runs and listens on a specific port. One of the things Memcached does lack is security. Memcached will let anybody who can make a connection to its port have full access to all objects. So you would have to run a firewall to block unauthorized access. It is usually wise to do put firewall on it even if you trust everybody on the same network since accidents do happen. That said, let’s get memcached installed!

Let’s get libevent which is required by Memcached:

wget http://monkey.org/~provos/libevent-1.3e.tar.gz
tar zxpfv libevent*
cd libevent*
./configure
make install

Now let’s download the newest Memcached source (at time of update to this post 1.4.5 was the latest)

wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz
tar zxpfv memcached*
cd memcached*
./configure
make install

Let’s add Memcached user to run daemon as since we don’t need it to run as root privileges.

adduser memcached

We will start the server to use 48 megs of ram (-m 48), listen on ip 10.0.0.2 (-l 10.0.0.2) and run on port 11211 (-p 11211) as user memcached (-u memcached)

./memcached -u memcached -d -m 48 -l 10.0.0.2 -p 11211

If you get the following error (which you will get if you are doing this under CentOS 64 bit):

./memcached: error while loading shared libraries: libevent-1.3e.so.1: cannot open shared object file: No such file or directory

You can fix this by simply doing this:

ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/

That is all there is to it. You can see if daemon is running by telneting to the port.

PHP: How do I install phpsh, interactive shell prompt for php under CentOS or Fedora?

Sunny Walia — Tue, 11 Dec 2007 08:26:48 +0000

phpsh requires readline support built into python. It also requires python version 2.4+. You can check which version of python you have installed by typing:

python -V

Let us download and install readline:

wget ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz tar zxf readline-5.2.tar.gz cd readline-5.2 ./configure make install

Now let us install python with readline support:
wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz tar zxf Python-2.5.1.tgz cd Python-2.5.1

I had some problems on one of the servers where it would not compile readline support in to python. I was able to compile reading support in to python by:

echo "readline readline.c -lreadline -ltermcap" >> Modules/Setup.local

Now let us continue with python installation.

./configure --prefix=/usr/local/python-2.5.1 --enable-readline make -i install

If you do not do “make -i install”, install may fail with following error (-i means ignore any errors):
Compiling /usr/lib/python2.5/zipfile.py ... make: *** [libinstall] Error 1

If you scroll up, you will find following error which seems to be the root cause:

Compiling /usr/lib/python2.5/test/test_multibytecodec.py ... Sorry: UnicodeError: ("\\N escapes not supported (can't load unicodedata module)",)

Once you do python install with make -i install, that library (unicodedata.so which is not built until later stage of build process) gets installed. If you want, you can type make install once again (without ignoring errors) and it will complete without errors.

Once you have python installed, you would want to use the new version. I like to keep a backup of old files in case I have to use older version for any reason. Run following which creates symbolic links and makes backups of current files:

for binaries in `find /usr/local/python-2.5.1/bin/*` ; do mv /usr/bin/`basename ${binaries}` /usr/bin/`basename ${binaries}`.bak ln -s ${binaries} /usr/bin/`basename ${binaries}` done

Now let us get phpsh and try it out:

wget http://www.phpsh.org/phpsh-latest.tgz tar zxf phpsh-latest.tgz cd ../phpsh chmod +x phpsh ./phpsh

At this point you should be at the shell: php>

Following is a snippet from README file which comes with phpsh. You should take a look since it has more details on how to use phpsh:

Type php commands and they will be evaluated each time you hit enter. Ex: php> $msg = "hello world" Put = at the beginning of a line as syntactic sugar for return. Ex: php> = 2 + 2 If you end a line with a backlash (\), you can enter multi-line input. For example, php> print "like \ ... this" like this php>

There we go. Now you have a great interactive php shell prompt. Note that there is interactive mode built into php as well. I personally do not like it as much but you can try it out for yourself by typing:

php -a Interactive mode enabled echo "Hello world"; Hello world

MySQL: How do I import individual table dump files in to MySQL using shell script?

Sunny Walia — Thu, 29 Nov 2007 03:35:49 +0000

After I wrote the post: How do I dump all tables in a database into separate files? I got emails from couple people asking how to import the individual table files back in to MySQL. First way to import each sql file created by the post is to import each file individually by typing:mysql db_name < table1.sql This will work as long as you are only importing few files. But if you need to import all of the files in the directory, which could be in 100’s, this method does not scale well. To achieve this I wrote a shell script which does the work for me. Of course, there are other ways to do this and I am only showing you one way of doing it. This works for me so here it is:

#!/bin/bash db=$1 if [ "$db" = "" ]; then echo "Usage: $0 db_name" exit 1 fi mkdir done clear for sql_file in *.sql; do echo "Importing $sql_file"; mysql $db< $sql_file; mv $sql_file done; done

MySQL: How do I dump all tables in a database into separate files?

Sunny Walia — Tue, 27 Nov 2007 00:22:53 +0000

There have been numerous occasions where I needed to make backups of individual tables from selected database. Usually I can achieve this by typing:

mysqldump database_name table1 > table1.sql mysqldump database_name table2 > table2.sql
This could be very painful if you have 10’s or 100’s of tables. Until today, I never ran into a situation where I had to deal with dumping more than few tables at a time. Today I had to do a dump of 181 tables. I was not going to sit there and type in that command with 181 table names. It is not just time consuming but it is also stupid. So I wrote this script to help me with this task. We still use mysqldump command as described above, except we do it programmatically to make it easier on us:

#!/bin/bash db=$1 if [ "$db" = "" ]; then echo "Usage: $0 db_name" exit 1 fi mkdir $$ cd $$ clear for table in `mysql $db -e 'show tables' | egrep -v 'Tables_in_' `; do echo "Dumping $table" mysqldump --opt -Q $db $table > $table.sql done if [ "$table" = "" ]; then echo "No tables found in db: $db" fi

You can also compress your files by adding bzip2, zip or any other compression commands after mysqldump command. Here is the same script with bzip2 command added:
#!/bin/bash db=$1 if [ "$db" = "" ]; then echo "Usage: $0 db_name" exit 1 fi mkdir $$ cd $$ clear for table in `mysql $db -e 'show tables' | egrep -v 'Tables_in_' `; do echo "Dumping $table" mysqldump --opt -Q $db $table > $table.sql bzip2 $table.sql done if [ "$table" = "" ]; then echo "No tables found in db: $db" fi

I do not recommend doing compression on a production server since most compression program put descent amount of load on the server. Also note that this will delay your dump considerably. You may also want to use different parameters for running mysqldump. Type man mysqldump in your shell to read more.

MySQL Cluster: Changing datamemory requires node restart with –initial?

Sunny Walia — Wed, 14 Nov 2007 02:52:57 +0000

I ran into something with mysql cluster today which boggles my mind. On http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-config-params-ndbd.html page, it is documented that if we you change datamemory parameter in the config.ini under mysql-cluster like below, you would have to restart nodes to reread the configuration.

[NDBD]
id=2
HostName=10.0.0.2 # the IP of the first data node
DataMemory=6G
IndexMemory=512M

But when I tried the: 2 restart in the mysql cluster managment console (ndb_mgm), it didn’t re-read the new configuration. DataMemory was initially set to 2G and it continued using that configuration. I even tried: 2 stop, and than logged on the node server and started ndbd manually, but it refused to read the new configuration. It is only when I did ndbd –initial, nodes read in the configuration and were able to use more than 2 gigs of ram.

Did anybody else run into this? Am I reading the documentation on MySQL site incorrectly?

Linux: How do I mass find and replace text in files under linux using perl?

Sunny Walia — Mon, 29 Oct 2007 07:00:08 +0000

Few friends have asked me how to do mass find and replace text in files under linux. There are quite a bit of options in linux to achieve mass replacing of text in files. If you are doing it file by file, you can achieve that in vi by opening, running replace and closing and going to next file. But sometimes that can be very tedious and you would rather do mass replacement on all files containing certain extension. We can do this by using sed or perl. Since most people are familiar with perl (at least most system admins and programmers), I will show you a perl way of doing it which you can use with sed as well if you wish. First step is to get perl to do what we want on one file

perl -w -i -p -e "s/search_text/replace_text/g" filename

-w turns warnings on
-i makes Perl operate on files in-place (if you would like to make backups, use -i.bak, this will save filename.bak files)
-p loops over the whole input
-e specifies Perl expression
filename works on one file at a time

Once we get the results we want, we can now pass it multiple files by doing something like:

perl -w -i -p -e "s/search_text/replace_text/g" *.php

This will search and replace within all php files in the directory you are in.

Now, let us say you want to go through your whole web directory and replace every place where you have “Perl is good” to “Perl is great”, you would use following command:

find /www_root -name "*.php"|xargs perl -w -i -p -e "s/Perl is good/perl is great/g"

find will start at /www_root, look for filenames which have .php extension, xargs takes that filename and passes it to perl as an arguement.

Linux virus scan: How do I check my linux installation for viruses using ClamAV? (CentOS, Linux, Redhat)

Sunny Walia — Sat, 20 Oct 2007 03:50:32 +0000

There are quite a bit of antivirus software exist for Linux nowadays. One of the popular antivirus software is ClamAV. We will install Clam AntiVirus software from source and install it to a custom directory. We will also install gmp-devel package which installs GMP library. GMP library is used to verify the digital signature of the virus database.

yum -y install gmp-devel wget http://freshmeat.net/redir/clamav/29355/url_tgz/clamav-0.91.2.tar.gz adduser -M -s /bin/false clamav tar zxf clamav-0.91.2.tar.gz cd clamav-0.91.2 ./configure --prefix=/usr/local/clamav make install for binaries in `find /usr/local/clamav/bin/*` ; do ln -s ${binaries} /usr/bin/; done

At this point Clam AntiVirus is installed and ready for use. Edit the configuration file and remove the line which says: Example It is there to ensure. If you want, you can look at other options but we don’t need to change anything else here to make ClamAV work for us.

vi /usr/local/clamav/etc/freshclam.conf #remove Example

Now let us run the freshclam which will download virus database and bring our virus database up to date. We should do this manually and make sure it didn’t give any errors. If this works, you will a lot of “downloading” messages.

/usr/bin/freshclam

If everything checks out, let us add this to our crontab to ensure our virus database is updated hourly. I chose to be updated every 9 minutes in to every hour. You can change to fit your needs or leave it as it is.
crontab -e

9 * * * * /usr/bin/freshclam –quiet

At this point our ClamAV virus database is up to date and now we can scan whichever directory we want. Go to the directory you want to scan and type:

clamscan -r -i

Once it is done scanning, it will display something similar to below.
-r parameter tells clamscan to recurse into directories
-i will print out infected filenames

----------- SCAN SUMMARY ----------- Known viruses: 159855 Engine version: 0.91.2 Scanned directories: 1437 Scanned files: 8836 Infected files: 0 Data scanned: 464.83 MB Time: 103.678 sec (1 m 43 s)

Happy scanning!
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL: Slave lag behind master and data corruption.

Sunny Walia — Tue, 16 Oct 2007 00:09:10 +0000

I am reaching out to mysql experts out there! We just inherited a site which uses MySQL master/slave replication. We set up the master and slaves per MySQL documents and did some of our own tweaking in my.cnf to get things working. Things were great when we were testing the site but than after we put it up, we started to notice missing data on slaves. At first, we thought maybe we missed something with the import of data on slaves, so we re-imported. All tables in our db, at that time, were myisam. After a day or so, we started to see data corruption on some of the slaves. We went through and repaired all the tables. Few hours after the repair, we checked row counts and the repaired tables didn’t have same count as master did. Eventually most of our slaves were at the same state so we decided to re-import data from master again.

Before we went ahead and re-imported, we made a decision to move to innodb for *most* of our crucial tables. After altering on master, we did a dump/import. Converting tables to innodb got rid of our data corruption problem but our missing rows issue is still there. Hardware on the master is 4×1.86Ghz, 8 gigs of ram; slaves hardware is: 4×1.86, 4 gigs of ram.

Here is the config running on our master:
< some of the part of the config has been cut off to keep it short/simple and to the point > skip-locking ft_min_word_len=2 long_query_time=30 max_connect_errors=10000 max_connections=1000 #skip-grant-tables skip-bdb delay-key-write=ALL key_buffer = 1G read_buffer_size = 2M read_rnd_buffer_size = 8M myisam_sort_buffer_size = 64M thread_concurrency = 8 skip-name-resolve max_allowed_packet = 32M #low-priority-updates table_cache = 1024 thread_cache = 8 query_cache_size = 64M # # added by sunny > wait_timeout = 120 tmp_table_size = 128M sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries. read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row # pointers in the result file. To optimize this, # we read in a big block of row pointers, sort them, # and use them to read the rows in sorted order into a row buffer. thread_cache_size = 8 # # Try number of CPU's*2 for thread_concurrency thread_concurrency = 16 log_slow_queries log_long_format # # INNODB Settings innodb_additional_mem_pool_size = 32M # was 16 innodb_buffer_pool_size = 4G innodb_file_per_table innodb_log_file_size = 50M innodb_log_buffer_size = 8M innodb_flush_log_at_trx_commit = 0 innodb_lock_wait_timeout = 50 innodb_flush_method=O_DIRECT

Here is the config running on our slave:

# The MySQL server [mysqld] port = 3306 socket = /tmp/mysql1.sock pid-file = /usr/local/mysql/var/mysql.pid relay-log = /usr/local/mysql/var/mysql-relay-bin relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index log-error = /usr/local/mysql/var/mysql.err master-info-file = /usr/local/mysql/var/mysql-master.info relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info datadir = /usr/local/mysql/var server-id = 31 skip-slave-start skip-locking skip-name-resolve ft_min_word_len=2 long_query_time=30 max_connect_errors=10000 max_connections=1000 #skip-grant-tables skip-bdb delay-key-write=ALL key_buffer = 512M max_allowed_packet = 1M table_cache = 512M myisam_sort_buffer_size = 64M thread_cache_size = 4 read-only #slave-skip-errors=all # # added by sunny > wait_timeout = 120 tmp_table_size = 128M read_buffer_size = 2M sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries. read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row # pointers in the result file. To optimize this, # we read in a big block of row pointers, sort them, # and use them to read the rows in sorted order into a row buffer. thread_cache_size = 8 query_cache_size = 16M # changed to 16 3/1/07 # # Try number of CPU's*2 for thread_concurrency thread_concurrency = 8 log_slow_queries log_long_format # # INNODB Settings default_table_type = INNODB innodb_additional_mem_pool_size = 32M # was 16 innodb_buffer_pool_size = 2G innodb_file_per_table innodb_log_file_size = 50M innodb_log_buffer_size = 8M innodb_flush_log_at_trx_commit = 0 innodb_lock_wait_timeout = 50 innodb_flush_method=O_DIRECT

Please feel free to comment/scrutinize any of these settings. If you think they should be different, please give quick summary about why they should be different. Some of these settings were inherited from old setup of mysql (where they had it hosted before).

Right now we are just trying to find out why we have missing rows and why our slaves lag behind constantly. We have refreshed data almost every day for last two weeks. We barely have load average above 1 or 2. What can we do to track down the problem? What are the things we should be looking out for?

MySQL: How do I reset MySQL to have only the databases when I first installed it? (CentOS, Redhat, Fedora, Linux)

Sunny Walia — Fri, 14 Sep 2007 06:44:07 +0000

One of my clients asked me today to make their MySQL installation go back to default database install. Basically they wanted me to get rid of all their databases (in this case test databases) so they can start fresh and go live with only the databases they needed. So here are the steps I followed.

Note: I have installed their MySQL from source and installed MySQL in to /usr/local/mysql folder. Your installation path might be different.

service mysql stop cd /usr/local/mysql mv var var.bak mkdir var bin/mysql_install_db chown -R mysql.mysql var chmod 700 -R var service mysql start
At this point we should have fresh MySQL db setup.

Subversion (SVN): How do you set up backup for your Subversion repositories?

Sunny Walia — Fri, 17 Aug 2007 01:00:51 +0000

Subversion is becoming more and more popular every day. To get quick installation guide, see my post on installing Subversion. Once you have Subversion set up, make sure you have backups of repository setup! This is one of those things most people forget to do. If you followed the installation guide and did source install, you already have a tool available to you. You can find hot-backup.py tool in the tools/backup/ directory of the Subversion source distribution. One thing about this script is that it doesn’t backup all your repositories. Which might be an issue if you create more repositories but forget to create a script to backup that particular repository. Here is a very simple shell script to help you backup all of your subversion repositories. You can edit it to point to different /svn location if you didn’t install to default location. Make sure you copy hot-backup.py to a location which is included in your path so your scripts will work.

mkdir /backups/repos -p for repostobackup in `ls /svn/` ; do filename=`basename ${repostobackup}` if [ ! $filename = "" ]; then hot-backup.py --archive-type=bz2 /svn/${filename} /backups/repos fi done

This will create your backup files and compress them using bzip2. You can also use zip or gz if you prefer. Another thing you should do is to edit the hot-backup.py script and change num_backups to be whatever you think is a good number. I personally use: num_backups = 10

That’s all there is to it. Just add this script to your crons and your backups will be automated. You may want to consider moving your backups to another server by adding ftp, scp or rsync at the end of the script.

MySQL: InnoDB: ERROR: the age of the last checkpoint is [number]

Sunny Walia — Thu, 16 Aug 2007 21:32:59 +0000

One of the mysql database servers I manage started to have issues with doing backups yesterday. mysqldump was running but nothing was happening on the backup side. I started to investigate to see why our full backups were failing. I opened up the mysql error log file (mine is at: /usr/local/mysql/var/hostname.err) and notice there were many instances of following error.

070815 15:31:46 InnoDB: ERROR: the age of the last checkpoint is 9433957, InnoDB: which exceeds the log group capacity 9433498. InnoDB: If you are using big BLOB or TEXT rows, you must set the InnoDB: combined size of log files at least 10 times bigger than the InnoDB: largest such row.

I poked around and found out our log files were set to default (5 mb) and they needed to be increased. After doing some calculations and research, I decided going to 50 mb at this point would be a good way to go. So I made the change in my.cnf file and added: innodb_log_file_size = 50M

I then stopped our mysql server and restarted it. And to my horror I saw following error messages show up in the mysql error logs:

070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm' 070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'

I stopped the mysql server right away. I then remember, by help of a friend, that I have to remove the old log files if I change the size by changing innodb_log_file_size. So I issued a command: mv ib_logfile? ..

This command moves both ib_logfile1 and ib_logfile2 down one directory. I didn’t want to just remove ‘em so instead I moved them. After that, I restarted mysql again and to my comfort, everything came back up without errors. So lesson learned: if you change ib_logfile size by using innodb_log_file_size setting, make sure you move existing log files before you start mysql again.

Cron: crond will not read/honor new entries in crontab file. (CentOS, Redhat, Fedora)

Sunny Walia — Thu, 16 Aug 2007 21:15:59 +0000

I ran in to something really odd. I have been using crontab for LONG time and never seen this issue before. Basically I added new entry into my crontab by type crontab -e and as always saved/exited. For some reason that command never ran at the time I specified. I tested few more times and no luck. So I started looking on google to find out why crond will not ready my new entries from crontab. And then while I was looking, I decided to restart crond by issuing: service crond restart and to my surprise, my new schedule started working.

While I was looking around for solution for this problem on Google, I came upon few pages where they talked about if you have empty line before command, that command will not work. For example:

#this is a cron file

* * * * * /do/something

In this example, /do/something will not run since there is a blank line before it. You can fix this by moving that command up or put another comment above it. This is a very odd thing and I have never experienced it myself but I figured if you stumble upon this post looking for help, blank line before command in crontab might be your issue. Please comment if you had to delete a blank line to fix your crontab file. Also mention which OS you are using. To learn more about crond, see man crond and man crontab.

SSH: Getting error ssh_exchange_identification: Connection closed by remote host

Sunny Walia — Sun, 05 Aug 2007 09:37:37 +0000

Today I was working on setting up public key authentication over SSH on few servers I inherited recently. I have done this many times and never ran in to following error:

ssh_exchange_identification: Connection closed by remote host

After doing further research on the server I was trying to connect to (looking at logs) I noticed the server was denying connection based on ip. I put my servers ip in /etc/hosts.allow and tried again. This time I was connected without any problems. Note that you do not have to restart SSH for hosts.allow change to take affect.

SSH: Setting up public key authentication over SSH (CentOS, Redhat, Linux, Fedora)

Sunny Walia — Sun, 05 Aug 2007 09:10:47 +0000

Before we go in to details on how to set up public key authentication, I would like to talk about pros and cons about doing this. After you know the pros and cons, you can make an educated decision. First, let us talk about biggest benefit about setting up public key authentication over ssh. Once you set up the keys correctly, you will not need to enter password to access the other server. For example, you can just type ssh example.com and you will be logged in. This is the method you must use if you want to set up scripts which are run from crontab. For example, you may have rsync script running which does backups from one server to another every night. With every good thing, comes a bad thing. So let us say “somebody” has access to server1 as root. This somebody could be designated user or it could be a hacker who has gain root access to server1. Now since we have public key authentication over ssh setup between server1 and server2, this “somebody” now can access server2. As you can see, this can be a problematic. Now you know number one pro and con about setting up public key authentication over ssh. Let us now go on and actually go through process to setup keys.

On server1, you type: ssh-keygen -t rsa (just press enter to keep accepting defaults). You should see something like this:

ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: a2:b2:aw:w2:63:25:2a:62:fs:d5:ff:fd:11:f1:aa:60 root@server1

Fingerprint for your server will be different. Now cat /root/.ssh/id_rsa.pub Copy the contents it displays. You will need this for server2. Paste it into notepad or any other text editor and make sure it pastes as one line. This is very crucial you check this.

Now login to server2. Create the public key by typing: ssh-keygen -t rsa (this is same thing we did on server1). Once you are done, type vi /root/.ssh/authorized_keys2

Paste the content you copied earlier on server1. Make sure it is all on one line. Save/exit. At this point, we should change the permissions on the file to make sure sshd likes it: chmod 600 /root/.ssh/authorized_keys2

Go back to server1 and type ssh server2 You should be logged in with out having to put in a password. If you are prompted for password still, check server2 /root/.ssh/authorized_keys2 file and make sure everything you pasted is one line.

That is all you have to do set up public key authentication over ssh!

Bind: Quick install guide to install and setup Bind (DNS server) in secure (chroot) environment in Linux (CentOS, Redhat Enterprise (RHEL), Fedora).

Sunny Walia — Fri, 22 Jun 2007 06:55:02 +0000

There are a lot of great articles on setting up Bind and some of them even go in great details on setting up secure DNS server using Bind. I have used one them time after time called: Linux Howto: Chroot Bind How To. One of the things I do not like about it is that I had to read over things which I really do not need to know or already know. So I have put together this quick wow To based on this great how to. Setting up secure DNS server using bind is not very complicated.

Let us dive into this. First we create user and set up directory structure.

echo "named:x:200:200:Nameserver:/chroot/named:/bin/false" >> /etc/passwd echo "named:x:200:" >> /etc/group mkdir -p /chroot/named cd /chroot/named mkdir -p dev etc/namedb/slave var/run chown -R named:named /chroot/named/etc/namedb/slave chown named:named /chroot/named/var/run mknod /chroot/named/dev/null c 1 3 mknod /chroot/named/dev/random c 1 8 chmod 666 /chroot/named/dev/{null,random} cp /etc/localtime /chroot/named/etc/

vi /etc/sysconfig/syslog
edit the line which starts with SYSLOGD_OPTIONS and edit it to look like: SYSLOGD_OPTIONS=”-m 0 -a /chroot/named/dev/log”
Let us restart syslog to use new settings and let us secure the physical files on the server.

/etc/rc.d/init.d/syslog stop /etc/rc.d/init.d/syslog start chown root /chroot chmod 700 /chroot chown named:named /chroot/named chmod 700 /chroot/named cd /chroot/named chattr +i etc/localtime var

Let us clean up previous bind install since most linux installations have named (bind) pre-installed.
rpm -qa |grep bind rpm -e --nodeps

Let us get the source files for bind and install it. At time of writing, bind 9.4.1 is the latest.

wget http://ftp.isc.org/isc/bind9/9.4.1/bind-9.4.1.tar.gz tar zxpfv bind-*.gz cd bind* ./configure make && make install

copy following content to /etc/init.d/named

#!/bin/sh # # named This shell script takes care of starting and stopping # named (BIND DNS server). # # chkconfig: 345 55 45 # description: named (BIND) is a Domain Name Server (DNS) \ # that is used to resolve host names to IP addresses. # probe: true # # Source function library. . /etc/rc.d/init.d/functions # # Source networking configuration. . /etc/sysconfig/network # # Check that networking is up. [ ${NETWORKING} = "no" ] && exit 0 # [ -f /usr/local/sbin/named ] || exit 0 [ -f /chroot/named/etc/named.conf ] || exit 0 # # See how we were called. case "$1" in start) # Start daemons. echo -n "Starting named: " daemon /usr/local/sbin/named -u named -t /chroot/named -c /etc/named.conf echo touch /var/lock/subsys/named ;; stop) # Stop daemons. echo -n "Shutting down named: " kill `pidof named` echo rm -f /var/lock/subsys/named ;; status) status named exit $? ;; restart) $0 stop $0 start exit $? ;; reload) /usr/local/sbin/rndc reload exit $? ;; probe) # named knows how to reload intelligently; we don't want linuxconf # to offer to restart every time /usr/local/sbin/rndc reload >/dev/null 2>&1 || echo start exit 0 ;; # *) echo "Usage: named {start|stop|status|restart|reload}" exit 1 esac # exit 0

Make it executable: chmod +x /etc/init.d/named
Let us set it so “named” will automatically start at boot up: chkconfig --add named

Save follow code into /chroot/named/etc/named.conf
options { recursion no; // this will disable lookups against our server recursion no; directory "/etc/namedb"; // Working directory pid-file "/var/run/named.pid"; // Put pid file in working dir forwarders { 10.10.10.10; //ip of dns server to forward requests to }; statistics-file "/var/run/named.stats"; query-source address * port 53; version "Bind 10"; }; controls { inet 127.0.0.1 port 953 allow { 127.0.0.1; } keys { "rndc-key"; }; }; // Root server hints zone "." { type hint; file "root.hint"; }; // Provide a reverse mapping for the loopback address 127.0.0.1 zone "0.0.127.in-addr.arpa" { type master; file "db.127.0.0"; notify no; };

Save following in /chroot/named/etc/namedb/root.hint

; This file holds the information on root name servers needed to ; initialize cache of Internet domain name servers ; (e.g. reference this file in the "cache . " ; configuration file of BIND domain name servers). ; ; This file is made available by InterNIC ; under anonymous FTP as ; file /domain/named.root ; on server FTP.INTERNIC.NET ; ; last update: Nov 5, 2002 ; related version of root zone: 2002110501 ; ; ; formerly NS.INTERNIC.NET ; . 3600000 IN NS A.ROOT-SERVERS.NET. A.ROOT-SERVERS.NET. 3600000 A 198.41.0.4 ; ; formerly NS1.ISI.EDU ; . 3600000 NS B.ROOT-SERVERS.NET. B.ROOT-SERVERS.NET. 3600000 A 128.9.0.107 ; ; formerly C.PSI.NET ; . 3600000 NS C.ROOT-SERVERS.NET. C.ROOT-SERVERS.NET. 3600000 A 192.33.4.12 ; ; formerly TERP.UMD.EDU ; . 3600000 NS D.ROOT-SERVERS.NET. D.ROOT-SERVERS.NET. 3600000 A 128.8.10.90 ; ; formerly NS.NASA.GOV ; . 3600000 NS E.ROOT-SERVERS.NET. E.ROOT-SERVERS.NET. 3600000 A 192.203.230.10 ; ; formerly NS.ISC.ORG ; . 3600000 NS F.ROOT-SERVERS.NET. F.ROOT-SERVERS.NET. 3600000 A 192.5.5.241 ; ; formerly NS.NIC.DDN.MIL ; . 3600000 NS G.ROOT-SERVERS.NET. G.ROOT-SERVERS.NET. 3600000 A 192.112.36.4 ; ; formerly AOS.ARL.ARMY.MIL ; . 3600000 NS H.ROOT-SERVERS.NET. H.ROOT-SERVERS.NET. 3600000 A 128.63.2.53 ; ; formerly NIC.NORDU.NET ; . 3600000 NS I.ROOT-SERVERS.NET. I.ROOT-SERVERS.NET. 3600000 A 192.36.148.17 ; ; operated by VeriSign, Inc. ; . 3600000 NS J.ROOT-SERVERS.NET. J.ROOT-SERVERS.NET. 3600000 A 192.58.128.30 ; ; housed in LINX, operated by RIPE NCC ; . 3600000 NS K.ROOT-SERVERS.NET. K.ROOT-SERVERS.NET. 3600000 A 193.0.14.129 ; ; operated by IANA ; . 3600000 NS L.ROOT-SERVERS.NET. L.ROOT-SERVERS.NET. 3600000 A 198.32.64.12 ; ; housed in Japan, operated by WIDE ; . 3600000 NS M.ROOT-SERVERS.NET. M.ROOT-SERVERS.NET. 3600000 A 202.12.27.33 ; End of File

This should get you started. Now just create appropriate zone files and you are good to go.

Ramdisk: How do you install and set up Ramdisk under Linux (CentOS, RHEL, Fedora)?

Sunny Walia — Fri, 01 Jun 2007 23:36:21 +0000

Ramdisk is very good to have if you want something to stay in memory. Files in memory makes it so you can access them with out having to access hard drive all the time. Perfect candidates would be things which do not change eg. web images or downloadable files, etc. If you have Linux Kernel 2.4 or later, you already have support of ramdisk built in. You can check if ramdisk is setup by doing:

# dmesg | grep RAMDISK RAMDISK driver initialized: 16 RAM disks of 16384K size 1024 blocksize

You should get above output on CentOS and RHEL. Other linux flavors will have similar output as well. If you would like to see how they are named and what you would need to refer to, do the following:

# ls -l /dev/ram* lrwxrwxrwx 1 root root 4 Apr 24 12:05 /dev/ram -> ram1 brw-rw---- 1 root disk 1, 0 Apr 24 12:05 /dev/ram0 brw-rw---- 1 root disk 1, 1 Apr 24 12:05 /dev/ram1 brw-rw---- 1 root disk 1, 10 Apr 24 12:05 /dev/ram10 brw-rw---- 1 root disk 1, 11 Apr 24 12:05 /dev/ram11 brw-rw---- 1 root disk 1, 12 Apr 24 12:05 /dev/ram12 brw-rw---- 1 root disk 1, 13 Apr 24 12:05 /dev/ram13 brw-rw---- 1 root disk 1, 14 Apr 24 12:05 /dev/ram14 brw-rw---- 1 root disk 1, 15 Apr 24 12:05 /dev/ram15 brw-rw---- 1 root disk 1, 2 Apr 24 12:05 /dev/ram2 brw-rw---- 1 root disk 1, 3 Apr 24 12:05 /dev/ram3 brw-rw---- 1 root disk 1, 4 Apr 24 12:05 /dev/ram4 brw-rw---- 1 root disk 1, 5 Apr 24 12:05 /dev/ram5 brw-rw---- 1 root disk 1, 6 Apr 24 12:05 /dev/ram6 brw-rw---- 1 root disk 1, 7 Apr 24 12:05 /dev/ram7 brw-rw---- 1 root disk 1, 8 Apr 24 12:05 /dev/ram8 brw-rw---- 1 root disk 1, 9 Apr 24 12:05 /dev/ram9 lrwxrwxrwx 1 root root 4 Apr 24 12:05 /dev/ramdisk -> ram0

All those ramdisks listed have same size. In above example, they are all 16MB. Let us change that so we have more space allowed. Note that I say allowed and not allocated. We allocate space in one of the later steps by formatting one of the drives above. Let us set it up so we have 128 MB. Since this has to be in multiples of 1024, we will setup Ramdisk to have 131072K.

vi /etc/grub.conf

Find first line which looks similar to following:

kernel /vmlinuz-2.6.9-42.0.10.EL ro root=/dev/VolGroup00/LogVol00

add ramdisk_size=131072 to the end of the line. Now your line should look like:

kernel /vmlinuz-2.6.9-42.0.10.EL ro root=/dev/VolGroup00/LogVol00 ramdisk_size=131072 Save and exit grub.conf. At this point you have it configured to have ramdisk with new size but it does not take effect until you reboot your system. Once you have rebooted your system, we can start doing rest of configurations.

mke2fs -m 0 /dev/ram0

This will format the ram0 ramdrive for us to use. At this point, kernel will allocate space for you. Let us setup Ramdisk mount point so we can use it. We will also have it be owned by user “sunny” so that user can read/write to that mount.

mkdir /home/ramdisk mount /dev/ram0 /home/ramdisk chown sunny.sunny /home/ramdisk

At this point you should be able to type: mount and see your new Ramdisk drive mounted on /home/ramdisk

Remember that everything you put on this drive will be gone if you reboot your server. If you unmounted the Ramdisk drive and remounted it, your files will still be there. It is because your system has that much ram set aside for your Ramdisk and will not use it for anything else. If you would like to setup Ramdisk the same next time you boot up, add these lines to your /etc/rc.local files.

mke2fs -m 0 /dev/ram0 mount /dev/ram0 /home/ramdisk chown sunny.sunny /home/ramdisk

Subversion: How do you install and set up Subversion for revision control?

Sunny Walia — Fri, 01 Jun 2007 20:32:24 +0000

Code revision control is very crucial part of development. It ensures you know who has changed which files and you are able to rollback to older versions in case new changes are breaking your website page or entire project. There are few options for you if you are looking into setting up code revision control. My preferred option is Subversion. Subversion is an open-source revision control system which is becoming more and more popular every day. It is very easy to install and setup your project under Subversion. If you want detailed instructions, please see the Subversion Book. They do a great job explaining what revision control is, what are the different types of revision control are out there, detailed instructions on installing subversion, doing administrator tasks, etc. Here are instructions on how to set up Subversion under CentOS. Same instructions apply to most linux distributions.

YUM Install

Easiest way to install subversion is via “yum” by typing:

yum install subversion

Once install is done, confirm it by typing “svn” at the prompt and you should get:

Type 'svn help' for usage.

At this point you can skip to “Setting up Subversion repository” part.

SOURCE Install

Obtain source from: Subversion. At the time of writing, latest (1.4.3) version can be obtain from this link: http://subversion.tigris.org/downloads/subversion-1.4.3.tar.gz

tar zxf subversion* cd subversion* ./configure make && make install

after everything goes well, type svn at the prompt and you should get:

Type 'svn help' for usage.

Setting up Subversion repository

adduser svnusers mkdir repos/branches -p mkdir repos/trunk -p mkdir repos/tags -p mkdir /svn svnadmin create /svn/demorepo svn import --message "Initial set up" repos file:///svn/demorepo cd /svn/demorepo chown .svnusers . -R chmod 775 * -R chmod +s db

Thats all there is to it to setup Subversion repository. At this point, you should add any users you want to have access to the repository by adding them to “svnusers” group.

Apache: How do you set up log rotation and archiving for Apache logs?

Sunny Walia — Thu, 31 May 2007 23:53:39 +0000

If you have a website and never have archived your apache logs, you may be surprised one day by running out of space due to logs using up all the free space on specific partition. Most people don’t think about archiving until it is too late. Here we set up a simple log rotation script which will work for apache 1.x and also for apache 2.x version as well. I have included comments in the script itself to help you understand what is going on throughout.

#!/bin/sh # # Created by: Sunny Walia # Date created: 5/24/03 # Date Modified: 5/29/07 # Purpose: do any type of preprocessing before log.php can do its work. # Also does log rotation and backup. # # Modification history: # 5/25/03: added comments. # Gzip and move to backup logs folder # added config info on top # # # configuration info SERVERNAME="svr1" WORKINGDIR="/admin/backups/$SERVERNAME/logs/" BACKUPLOGSDIR="/admin/backups/$SERVERNAME/logs/archived/" BACKUP_DIR="/admin/backups/$SERVERNAME/" LOGDIR="/usr/local/apache2/logs/" DATE=`date +%m%d%y%H` APACHCTL_LOC="/usr/local/apache2/bin/" # mkdir $WORKINGDIR -p mkdir $BACKUPLOGSDIR -p mkdir $BACKUP_DIR -p # # move the logs into their temp name so apache can continue to write to it mv ${LOGDIR}access_log ${LOGDIR}${SERVERNAME}_access_log.old mv ${LOGDIR}error_log ${LOGDIR}${SERVERNAME}_error_log.old # # restart apache gracefully so it will finish what its doing and start a new log file # when done serving current requests ${APACHCTL_LOC}apachectl graceful # # give some time to apache to finish serving pending requests (in secs) sleep 15 # # copy the files to our directory for processing cp ${LOGDIR}${SERVERNAME}_access_log.old ${WORKINGDIR}${SERVERNAME}_access_log cp ${LOGDIR}${SERVERNAME}_error_log.old ${WORKINGDIR}${SERVERNAME}_error_log # mv ${LOGDIR}${SERVERNAME}_access_log.old ${LOGDIR}${SERVERNAME}_access_log.$DATE mv ${LOGDIR}${SERVERNAME}_error_log.old ${LOGDIR}${SERVERNAME}_error_log.$DATE gzip ${LOGDIR}${SERVERNAME}_access_log.$DATE gzip ${LOGDIR}${SERVERNAME}_error_log.$DATE # # we move instead of copy than delete since if mv fails, we still have originals # mv can fail if target location is full mv ${LOGDIR}${SERVERNAME}_access_log.$DATE.gz $BACKUPLOGSDIR mv ${LOGDIR}${SERVERNAME}_error_log.$DATE.gz $BACKUPLOGSDIR

Let us make the script executable:
chmod +x script_name.sh

You can automate this to happen every day by putting it in your crontab:
crontab -e

1 0 * * * /path/to/this/script

This will rotate your logs at 12:01 AM every day.

Rsync: Using rsync to backup data from one server to another over SSH. Quick rsync tutorial.

Sunny Walia — Thu, 24 May 2007 22:30:05 +0000

Rsync is a great tool which can be used to do many tasks which involved copying/moving data. If privacy/security is of concern, which it always should be, you can use rsync to do all the copying/moving of data over SSH. Read through “man rsync” to get deeper understanding of rsync. Here is my attempt to a short tutorial on rsync. Let us start with most simple example of using rsync over ssh.

rsync -ae ssh server1:/home /home/backups/server1_home_backup/

This command will download all the files/directories from /home on server1 and copies them to /home/backups/server1_home_backup/
-a = archive mode. This will preserve permissions, timestamps, etc.
-e = specify which remote shell to use. In our case, we want to use ssh which follow right after “e”

Let us improve on this and add couple more parameters:

rsync -zave ssh --progress server1:/home /home/backups/server1_home_backup/
-z = adds zip compression.
-v = verbose
–progress = my favorite parameter when I am doing rsync manually, not so good when you have it in cron. This show progress (how_many_files_left/how_many_files_total) and speed along with some other useful data.

Great.. we are moving along pretty good. Let us add some security to make sure things work the way we want to.

rsync --delete-after -zave ssh --progress server1:/home /home/backups/server1_home_backup/

–delete-after = this will delete files on backup server which are missing from source after ALL syncing is done. If you don’t care of having extra files on your backup server and have plenty of disk space to spare, do not use this parameter.

Lastly, one of the VERY handy parameters,

rsync --delete-after -zave ssh --progress server1:/home /home/backups/server1_home_backup/ -n

The -n (or –dry-run) parameter is great to use for testing. It will not transfer or delete any files, rather will report to you what it would have done if it was ran with out -n parameter. This way you can test it with out destroying or transfering data just to find out that is not what you wanted.

For further reading: man rsync