libtool: link: cannot find the library `/usr/local/src/httpd-2.2.14/srclib/apr-util/xml/expat/lib/libexpat.la' or unhandled argument `/usr/local/src/httpd-2.2.14/srclib/apr-util/xml/expat/lib/libexpat.la'

I was able to fix this error by adding: --with-expat=builtin

Remember to do make clean or untar again before doing configure. Good luck!

for i in `find -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

I am sure there are many different ways to show file count for each folder in a directory and I am curious to see what people do so please do post comments with what you do.

Above command is pretty simple and can be expanded to do whatever you need.  For example, you can throw it into a bash script and be able to pass parameters.  For example:  count_files /home/  In this case your command line would look like:

for i in `find /home/ -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

only difference would be that /home/ would be argument you passed and therefore will be $1.  Here is a sample script for above example:

#!/bin/bash
for i in `find $1 -maxdepth 1 -type d`; do
echo -n $i " ";
find $i|wc -l;
done

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

./configure –prefix=/usr/local/curl –disable-ipv6

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

So without further delay, following command will telnet to local memcached server on port 11211 and issue one of the memcached commands, stats

(sleep .5;echo stats) | telnet localhost 11211
You may have to play with the sleep timer to get it to work for your environment but in our .5 was the sweet spot. Good luck and let me know if you have another shell command. Obviously we can do this from perl, php, python, etc but the beauty of this is that you do not need any other dependencies plus its a very short command.

Since “jsled” commented about nc (thanks jsled), here is the syntax to do the same thing with nc:

echo "stats" | nc localhost 11211

I needed to redirect a url:  http://www.example.com/testpage.php?foo=bar to http://www.example.com/

So I added this to .htaccess file:

RewriteRule ^testpage\.php / [R=301,L]

This did do a 301 redirect as I wanted but query string passed so my redirected url looked like:  http://www.example.com/?foo=bar

Obviously this is not what I wanted so in order for me to fix it, I had to take a rather lame approach.  My new redirect is:

RewriteRule ^testpage\.php /? [R=301,L]

The only difference is the “?” after / for target url.  I am just curious to know why query-string was passed.  I have not used rewrites for long time but I do not remember this behavior.

Note:  you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz
tar zxpfv fuse-*.gz
cd fuse*
./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found
configure: error:
*** Please specify the location of the kernel source with
*** the '--with-kernel=SRCDIR' option
configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/
total 4.0K
drwxr-xr-x 18 root root 4.0K Oct  7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64
make && make install
cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz
tar zxpfv sshfs*
cd sshfs-fuse-*
./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

OR

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0′ found
No package ‘gthread-2.0′ found

You need to install glib2.  Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure
make && make install

After installation is done, we can move on with testing the installation:

cd /mnt
mkdir test
sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse
FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt
sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
...
sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test
...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

wget http://www.whatismyip.org

Well that does the trick.  But being lazy as I am, I did not want to have to cat the output file to find out the ip (plus there is no point of creating extra files and doing extra work to remove them).  So if you are ssh’ed in, you can issue following command (I am sure there is another way of doing it, but this is the quickest way I could think of):

wget -q -O - http://www.whatismyip.org

-O tells wget to redirect output to the following file (- being the standard out ).  So it basically echo’s output to our console.

-q makes wget run in  quiet mode so you do not see all of the connection/download/etc output.

That is it!  I am curious to know what other ways people use to get the same information.  Please share your way if possible.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mkdir temp
cd temp
touch Foo FooO "Foo Bar" "FOO BAaR"
\ls | while read -r FILENAME
do
mv -v "$FILENAME" `echo $FILENAME| tr ' ' '_'| tr '[A-Z]' '[a-z]'`
done

Note:  I intentionally have slash in front of ls (\ls).  \ls means that we want to make sure there is no ls alias overwriting our command. This is needed if your system has alias setup to display ls in a different way instead of default listing.  mv -v shows us the filenames being renamed as your script goes through the whole dir.  Your output should be like:

`Foo' -> `foo'
`FOO BAaR' -> `foo_baar'
`Foo Bar' -> `foo_bar'
`FooO' -> `fooo'

One of the very powerful commands in this post is the “tr” command. This command is not as popular as sed or awk but it is very useful and simple to use (read more about tr).

If you only needed to convert spaces to underscores and you are using CentOS/Fedora/Redhat, you can use this simpler method. NOTE: this command is not available on all distributions: rename " " "_" *

Learn more about rename command
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Install perl modules:

install Term::ReadKey
install File::Tail
install Time::HiRes

Now copy paste the script below and save it in a location and set +x attribute on it so you can execute it.  On my setup, I have the script under /usr/bin/httptop:

#!/usr/bin/perl -w
use Time::HiRes qw( time );
use File::Tail (  );
use Term::ReadKey;
use Getopt::Std;
use strict;
### Defaults you might be interested in adjusting.
my $Update = 2; # update every n secs
my $Backtrack = 250; # backtrack n lines on startup
my @Paths = qw(
%
/title/%/logs/access_log
/var/log/httpd/%/access_log
/usr/local/apache/logs/%/access_log
);
my $Log_Format = "combined";
my %Log_Fields = (
combined => [qw/ Host x x Time URI Response x Referer Client /],
vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
);
### Constants & other thingies. Nothing to see here. Move along.
my $Version = "0.4.1";
sub by_hits_per (  ) { $b->{Rate} <=> $a->{Rate} }
sub by_total (  ) { $b->{Total} <=> $a->{Total} }
sub by_age (  ) { $a->{Last} <=> $b->{Last} }
my $last_field = "Client";
my $index = "Host";
my $show_help = 0;
my $order = \&by_hits_per;
my $Help = "htlwufd?q";
my %Keys = (
h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
w => [ "Show remote host" => sub { $index = "Host" } ],
u => [ "Show requested URI" => sub { $index = "URI" } ],
f => [ "Show referring URL" => sub { $index = "Referer" } ],
d => [ "Show referring domain" => sub { $index = "Domain" } ],
'?' => [ "Help (this thing here)" => sub { $show_help++ } ],
q => [ "Quit" => sub { exit } ]
);
my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
my @Record_Fields = qw/ Host URI Referer Domain /;
my $Max_Index_Width = 50;
my $Initial_TTL = 50;
my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
my %Term = (
HOME => "\033[H",
CLS => "\033[2J",
START_TITLE => "\033]0;", # for xterms etc.
END_TITLE => "\007",
START_RV => "\033[7m",
END_RV => "\033[m"
);
my ( %hist, %opt, $spec );
$SIG{INT} = sub { exit };
END { ReadMode 0 };
### Subs.
sub refresh_output
{
my ( $cols, $rows ) = GetTerminalSize;
my $show = $rows - 3;
my $count = $show;
my $now = (shift || time);
for my $type ( values %hist ) {
for my $peer ( values %$type ) {
# if ( --$peer->{_Ttl} > 0 ) {
my $delta = $now - $peer->{Start};
if ( $delta >= 1 ) {
$peer->{ Rate } = $peer->{ Total } / $delta;
} else {
$peer->{ Rate } = 0
}
$peer->{ Last } = int( $now - $peer->{ Date } );
# } else {
# delete $type->{$peer}
# }
}
}
$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values %{$hist{$index}};
my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
my $first = 0;
$first = ( $first <= $_ ? $_ + 1 : $first ) for map { $_ ? length($_->{$index}) : 0 } @list;
$first = $Max_Index_Width if $Max_Index_Width < $first;
print $Term{START_TITLE}, "Monitoring $spec at: ", scalar localtime, $Term{END_TITLE} if $ENV{TERM} eq "xterm"; # UGLY!!!
my $help = "Help/?";
my $head = sprintf( "%-${first}s %6s %4s %4s %s (%d total)",
$index, qw{ Hits/s Tot Last }, $last_field,
scalar keys %{$hist{$index}}
);
#
# Truncate status line if need be
#
$head = substr($head, 0, ($cols - length($help)));
print @Term{"HOME", "START_RV"}, $head, " " x ($cols - length($head) - length($help)), $help, $Term{END_RV}, "\n";
for ( @list ) {
# $_->{_Ttl}++;
my $line = sprintf( "%-${first}s %6.3f %4d %3d %s",
substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last }, $last_field)} );
if ( length($line) > $cols ) {
substr( $line, $cols - 1 ) = "";
} else {
$line .= " " x ($cols - length($line));
}
print $line, "\n";
}
print " " x $cols, "\n" while $count++ < $show;
}
sub process_line
{
my $line = shift;
my $now = ( shift || time );
my %hit;
chomp $line;
@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /"([^"]+)"|\[([^]]+)\]|\s/o, $line ) );
$hit{ URI } =~ s/HTTP\/1\S+//gos;
$hit{ Referer } = "<unknown>" if not $hit{Referer} or $hit{Referer} eq "-";
( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
$hit{ Client } ||= "<none>";
$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
$hit{ Client } =~ s/[^\x20-\x7f]//gos;
# if $now is negative, try to guess how old the hit is based on the time stamp.
if ( $now < 0 ) {
my @hit_t = ( split( m![:/\s]!o, $hit{ Time } ))[ 0 .. 5 ];
my @now_t = ( localtime )[ 3, 4, 5, 2, 1, 0 ];
my @mag = ( 3600, 60, 1 );
# If the hit didn't parse right, or didn't happen today, the hell with it.
return unless $hit_t[2] == ( $now_t[2] + 1900 )
and $hit_t[1] eq $Months[ $now_t[1] ]
and $hit_t[0] == $now_t[0];
splice( @hit_t, 0, 3 );
splice( @now_t, 0, 3 );
# Work backward to the UNIX time of the hit.
$now = time;
$now -= (shift( @now_t ) - shift( @hit_t )) * $_ for ( 3600, 60, 1 );
}
$hit{ Date } = $now;
for my $field ( @Record_Fields ) {
my $peer = ( $hist{$field}{$hit{$field}} ||= { Start => $now, _Ttl => $Initial_TTL } );
@$peer{ @Display_Fields } = @hit{ @Display_Fields };
$peer->{ Total }++;
}
}
sub display_help {
my $msg = "httptop v.$Version";
print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, "\n\n";
print " " x 4, $_, " " x 8, $Keys{$_}[0], "\n" for ( split "", $Help );
print "\nPress any key to continue.\n";
}
### Init.
getopt( 'frb' => \%opt );
$Backtrack = $opt{b} if $opt{b};
$Update = $opt{r} if $opt{r};
$Log_Format = $opt{f} if $opt{f};
$spec = $ARGV[0];
die <<End unless $spec and $Log_Fields{$Log_Format};
Usage: $0 [-f <format>] [-r <refresh_secs>] [-b <backtrack_lines>] <logdir | path_to_log>
Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
End
for ( @Paths ) {
last if -r $spec;
( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
}
die "No access_log $ARGV[0] found.\n" unless -r $spec;
my $file = File::Tail->new(
name => $spec,
interval => $Update / 2,
maxinterval => $Update,
tail => $Backtrack,
nowait => 1
) or die "$spec: $!";
my $last_update = time;
my ( $line, $now );
# Backtracking.
while ( $Backtrack-- > 0 ) {
last unless $line = $file->read;
process_line( $line, -1 );
}
$file->nowait( 0 );
ReadMode 4; # Echo off.
print @Term{"HOME", "CLS"}; # Home & clear.
refresh_output;
### Main loop.
while (defined( $line = $file->read )) {
$now = time;
process_line( $line, $now );
while ( $line = lc ReadKey(-1) ) {
$show_help = 0 if $show_help;
$Keys{$line}[1]->(  ) if $Keys{$line};
}
if ( $show_help == 1 ) {
display_help;
$show_help++; # Don't display help again.
} elsif ( $now - $last_update > $Update and not $show_help ) {
$last_update = $now;
refresh_output( $now );
}
}

Save/exit and make sure you make it executable by setting it to +x (chmod +x httptop)

Now you can run httptop by typing:  httptop -f combined -r 1 /usr/local/apache2/logs/access_log

NOTE:  Your access_log file might be in different location.  Point to the right location.  This sets the refresh rate to 1 sec (-r 1).  Now you can run httptop any time you want to checkout how your http traffic is doing.  Remember to press “?” to get help once you are in.

—————

DISCLAIMER: As always, if you find any inaccurate information, please comment and let me know. When you do comment, make sure you give me some references to confirm.

Here are the instructions on how to get sphinxse compiled with your mysql installation on CentOS x64.  I am sure same instructions will work for other flavors but I have not tested it.  I will be compiling the most current version of sphinx (0.9.8) with most current stable version of mysql (5.0.51b) at the time of the writing.  Let’s get the appropriate packages first:

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8.tar.gz
wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/
tar zxpf sphinx*
tar zxpf mysql*

You will also need “bison”, “patch”, “automake” and “libtool” installed.  Let us just do a yum install for it.

yum -y install bison patch automake libtool

NOTE:  if you don’t install bison, you will get the following error:
sed '/^#/ s|y\.tab\.c|sql_yacc.cc|' y.tab.c >sql_yacc.cct && mv sql_yacc.cct sql_yacc.cc
sed: can't read y.tab.c: No such file or directory
make[2]: *** [sql_yacc.cc] Error 2

Let us continue with patching mysql source with sphinx storage engine (sphinxse) code and compile/install our new binaries.

cd mysql*
patch -p1 < ../sphinx-0.9.8/mysqlse/sphinx.5.0.37.diff #Make sure everything succeeded.
BUILD/autorun.sh
mkdir sql/sphinx
cp ../sphinx-0.9.8/mysqlse/* sql/sphinx
./configure --prefix=/usr/local/mysql --with-sphinx-storage-engine
make
make install

Now start your mysql installtion and check if engine support is compiled in:

mysql> show engines\G
Engine: SPHINX
Support: YES
Comment: Sphinx storage engine 0.9.8

To read more about how to use Sphinx storage engine, please refer to:  Sphinx documentation for using sphinx storage engine

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Following command will search for the string you specified. Generally this will give you all of the packages which has specified string in title or description. Most of the time you will have to look through a lot of output to find what you are looking for.

yum search string

Probably one of the most important options for yum is provides/whatprovides. If you know what command you need, you can find out what package you have to install in order to have that command available to you.

yum provides (or whatprovides) command

Following command is same as above but with less output.

yum -d 1 provides command

So for example if you are trying to figure out what you need to install to use bunzip2, type:

yum -d 1 provides bunzip2

you will get a similar output as below.

# yum -d 1 provides bunzip2
bzip2.x86_64 1.0.2-13.EL4.3 base
bzip2.x86_64 1.0.2-13.EL4.3 base
man-pages-fr.noarch 0.9.7-13.el4 base
man-pages-ja.noarch 20050215-2.EL4.3 base
man-pages-pl.noarch 0.23-5 base

As you can see bunzip2 is part of bzip2 package. So now you can you just install bzip2.x86_64 to get bunzip2.

To learn more about what else is available, read man yum.

# /usr/local/apache2/bin/apachectl -t -D DUMP_MODULES
Loaded Modules:
...
deflate_module (static)
...
Syntax OK

If you don’t have have deflate_module, you would have to recompile your apache with “–enable-deflate” option.

Going forward, I am going to assume you have deflate_module. Add the following to your apache conf file:

<Location />
SetOutputFilter DEFLATE
BrowserMatch ^Mozilla/4\.0[678] no-gzip\
BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html
# Don't compress images
SetEnvIfNoCase Request_URI \
\.(?:gif|jpe?g|png)$ no-gzip dont-vary
</Location>

The main thing you need to configure is the line which says “no-gzip dont-vary” also in bold above. This tells apache to not compress certain type of files. I have noticed on some of my sites that swf (flash) files do not work as expected if they are compressed. So if you have swf files in your site, you may want to add |swf right after png.

This is all what it takes for you to enable gzip compression in Apache2. Once you restart your apache so it reads the conf file, you can test if your site is getting compressed or not by using this tool: http://www.gidnetwork.com/tools/gzip-test.php

Here are the results for my blog:

Results for: http://crazytoon.com
Web page compressed? Yes
Compression type? gzip
Size, Markup (bytes) 57,337
Size, Compressed (bytes) 11,666
Compression % 79.7

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

So today we decided that we are going to migrate one of our master database servers to new hardware.  Since we got the hardware this morning and we wanted to move on to it asap, we decided that we will take our slave down, copy data from it, and bring it up on future master server.  At that point, we will let it run as slave to the current master server until its time for us to take it down.  Reason we did that instead of mysqldump/import was to avoid the lag mysqldump creates on our server.

After we did all this and put up the new master server, we started to notice odd issues.  After looking around and comparing old db with new, we found out that new db was missing data.  How it happened is beyond me and is the reason why I am writing this.  We never had issues with the slave which would cause data to be lost; so what happened to those missing rows?  Is this something which is common?  Can we not trust our slave enough to use it as master if master died?  Can we not run backups off the slave with confident that our data is protected and up to date so to keep load down on our master?  All these questions which keep me awake and wondering…

Note: you must be logged in as root to do following.

vi /etc/passwd
find joe and change it to john, save/exit

vi /etc/group
find joe and change it to john, save/exit

vi /etc/shadow
find joe and change it to john. This file is read only and you have to force overwrite it. In vi it is :w! once saved, quit.

cd /home
mv joe john

And that should do the trick.

[Edited] Right after I posted this post, I was contacted and was told to look at utility called usermod. Read more about it man usermod. Got to love the quick responses.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Slave server options:

• replicate-do-db = dbname (or) replicate-do-db = dbname1, dbname2, …, dbnameN
  This option is used on slave server to tell the server to only replicate dbname db on this particular host. You would want to use this if you have a master which is replicating to multiple slaves and each slave may contain different database for read performance reasons.
• replicate-ignore-db = dbname (or) replicate-ignore-db = dbname1, dbname2, …, dbnameN
  This option is used on slave server to tell server to ignore database(s) listed. This is used when you want to replicate every database except certain individual ones. For example, you may want to replicate-ignore-db=testdb
• replicate-do-table = dbname.tablename
  This specifies a table from a database to be replicated.
• replicate-ignore-table = dbname.tablename
  This is very useful and often ignored. If you have logging table which you only do writes to but never read from, there is no real point to replicate that table to slave(s). This way you ignore specific tables.
• replicate-wild-do-table=dbname.tablename%
  This is another option which can prove itself to be very useful. Let’s say you have database with multiple type of open source installations (phpbb, wordpress, drupal, etc), and you want to designate slave(s) to only be used for phpbb, you would do: replicate-wild-do-table=dbname.phpbb%
• replicate-wild-ignore-table=dbname.tablename%
  Another option which can be used instead of above example. Say you wanted everything on this particular slave BUT phpbb tables. You would put this in your config: replicate-wild-ignore-table=dbname.phpbb%

NOTE:  for wildcards, you can use generic mysql wildcards.

Master server options:

• binlog-do-db = dbname (or) binlog-do-db = dbname1, dbname2, …, dbnameN
  This option is used on master server to tell the master server to only log queries for dbname db. This is helpful if your master has multiple databases but you only want to replicate selected few to all slaves.
• binlog-ignore-db = dbname (or) binlog-ignore-db = dbname1, dbname2, …, dbnameN
  Ignore specified database and do not log queries referring to list database(s).

In order for mysql to know which database queries to log (or not to log) you have to make sure you specify what is your default db by doing mysql_select_db() or issuing “use dbname” command. For example, if you had binlog-do-db=forums and you issue a query: delete from main.users (basically saying, delete everything from users table in main database), it will still log the command even though main database is not in the binlog-do-db list. To get around this, you should do the following: use main; delete from users; If you do not do that, your query “delete from main.users” will be logged and executed on slave(s). There is a high probability that you do not have a test database on your slave.

NOTE: This post is part of series on MySQL Replication. Here is the original post: MySQL Replication Series: How does MySQL replication work?

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Master1 server ip: 10.0.0.1
Master2 server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Let us go through the steps you must take on Master1 to enable it to act as master and slave by using following configuration which goes under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=1
# Replication Master Server
# binary logging is required for replication
log-bin=master1-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 1
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.2
master-user = slaveuser
master-password = slavepw
master-port = 3306

Following is the configuration which goes on master2 under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=2
# Replication Master Server
# binary logging is required for replication
log-bin=master2-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 2
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.1
master-user = slaveuser
master-password = slavepw
master-port = 3306

On master1 server, go to mysql> prompt and add the appropriate user:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

On master2 server do the same but allow right ip:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.1' identified by 'slavepw';

Restart both of the master servers and check slave status:

mysql> show slave status\G

That is all you have to do to set up the replication. Of course there are a lot more configuration options but this should get your replication going and you can tweak from here on.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

1) Offload some of the queries from one server to another and spread the load: One of the biggest advantages to have master-slave set up in MySQL is to be able to use master for all of the inserts and send some, if not all, select queries to slave. This will most probably speed up your application without having to diving into optimizing all the queries or buying more hardware.

2) Do backups from slave: One of the advantages people overlook is that you can use MySQL slave to do backups from. That way site is not affected at all when doing backups. This becomes a big deal when your database has grown to multiple gigs and every time you do backups using mysqldump, site lags when table locks happen. For some sites, this could mean that site goes down for few secs to minutes. If you have slave, you just take slave out of rotation (should be built into code) and run backups off the slave. You can even stop slave MySQL instance and copy the var folder instead of doing mysqldump.

Ok let us dive into how to setup master-slave replication under MySQL. There are many configuration changes you can do to optimize your MySQL set up. I will just touch on very basic ones to get the replication to work. Here are some assumptions:

Master server ip: 10.0.0.1
Slave server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Put the following in your master my.cnf file under [mysqld] section:

# changes made to do master
server-id = 1
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
log-bin = /usr/local/mysql/var/mysql-bin
# end master

Copy the following to slave’s my.cnf under [mysqld] section:

# changes made to do slave
server-id = 2
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
# end slave setup

Create user on master:
mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

Do a dump of data to move to slave
mysqldump -u root --all-databases --single-transaction --master-data=1 > masterdump.sql

import dump on slave
mysql < masterdump.sql

After dump is imported go in to mysql client by typing mysql. Let us tell the slave which master to connect to and what login/password to use:
mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.1', MASTER_USER='slaveuser', MASTER_PASSWORD='slavepw';

Let us start the slave:
mysql> start slave;

You can check the status of the slave by typing
mysql> show slave status\G

The last row tells you how many seconds its behind the master. Don’t worry if it doesn’t say 0, the number should be going down over time until it catches up with master (at that time it will show Seconds_Behind_Master: 0) If it shows NULL, it could be that slave is not started (you can start by typing: start slave) or it could be that it ran into an error (shows up in Last_errno: and Last_error under show slave status\G).

Related posts:

• How do I dump all tables in a database into separate files?
• How do I import individual table dump files in to MySQL using shell script?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

python -V

Let us download and install readline:

wget ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
tar zxf readline-5.2.tar.gz
cd readline-5.2
./configure
make install

Now let us install python with readline support:

wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
tar zxf Python-2.5.1.tgz
cd Python-2.5.1


I had some problems on one of the servers where it would not compile readline support in to python. I was able to compile reading support in to python by:

echo "readline readline.c -lreadline -ltermcap" >> Modules/Setup.local

Now let us continue with python installation.

./configure --prefix=/usr/local/python-2.5.1 --enable-readline
make -i install

If you do not do “make -i install”, install may fail with following error (-i means ignore any errors):
Compiling /usr/lib/python2.5/zipfile.py ...
make: *** [libinstall] Error 1

If you scroll up, you will find following error which seems to be the root cause:

Compiling /usr/lib/python2.5/test/test_multibytecodec.py ...
Sorry: UnicodeError: ("\\N escapes not supported (can't load unicodedata module)",)

Once you do python install with make -i install, that library (unicodedata.so which is not built until later stage of build process) gets installed. If you want, you can type make install once again (without ignoring errors) and it will complete without errors.

Once you have python installed, you would want to use the new version. I like to keep a backup of old files in case I have to use older version for any reason. Run following which creates symbolic links and makes backups of current files:

for binaries in `find /usr/local/python-2.5.1/bin/*` ; do
mv /usr/bin/`basename ${binaries}` /usr/bin/`basename ${binaries}`.bak
ln -s ${binaries} /usr/bin/`basename ${binaries}`
done

Now let us get phpsh and try it out:

wget http://www.phpsh.org/phpsh-latest.tgz
tar zxf phpsh-latest.tgz
cd ../phpsh
chmod +x phpsh
./phpsh

At this point you should be at the shell: php>

Following is a snippet from README file which comes with phpsh. You should take a look since it has more details on how to use phpsh:

Type php commands and they will be evaluated each time you hit enter. Ex:
php> $msg = "hello world"
Put = at the beginning of a line as syntactic sugar for return. Ex:
php> = 2 + 2
If you end a line with a backlash (\), you can enter multi-line input.
For example,
php> print "like \
... this"
like this
php>

There we go. Now you have a great interactive php shell prompt. Note that there is interactive mode built into php as well. I personally do not like it as much but you can try it out for yourself by typing:

php -a
Interactive mode enabled
echo "Hello world";
Hello world

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

#!/bin/bash
db=$1
if [ "$db" = "" ]; then
echo "Usage: $0 db_name"
exit 1
fi
mkdir done
clear
for sql_file in *.sql; do
echo "Importing $sql_file";
mysql $db< $sql_file;
mv $sql_file done;
done

Related posts:

• How do I dump all tables in a database into separate files?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

[NDBD]
id=2
HostName=10.0.0.2   # the IP of the first data node
DataMemory=6G
IndexMemory=512M

But when I tried the: 2 restart in the mysql cluster managment console (ndb_mgm), it didn’t re-read the new configuration.  DataMemory was initially set to 2G and it continued using that configuration.  I even tried: 2 stop, and than logged on the node server and started ndbd manually, but it refused to read the new configuration.  It is only when I did ndbd –initial, nodes read in the configuration and were able to use more than 2 gigs of ram.

Did anybody else run into this?  Am I reading the documentation on MySQL site incorrectly?

perl -w -i -p -e "s/search_text/replace_text/g" filename

-w turns warnings on
-i makes Perl operate on files in-place (if you would like to make backups, use -i.bak, this will save filename.bak files)
-p loops over the whole input
-e specifies Perl expression
filename works on one file at a time

Once we get the results we want, we can now pass it multiple files by doing something like:

perl -w -i -p -e "s/search_text/replace_text/g" *.php

This will search and replace within all php files in the directory you are in.

Now, let us say you want to go through your whole web directory and replace every place where you have “Perl is good” to “Perl is great”, you would use following command:

find /www_root -name "*.php"|xargs perl -w -i -p -e "s/Perl is good/perl is great/g"

find will start at /www_root, look for filenames which have .php extension, xargs takes that filename and passes it to perl as an arguement.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

yum -y install gmp-devel
wget http://freshmeat.net/redir/clamav/29355/url_tgz/clamav-0.91.2.tar.gz
adduser -M -s /bin/false clamav
tar zxf clamav-0.91.2.tar.gz
cd clamav-0.91.2
./configure --prefix=/usr/local/clamav
make install
for binaries in `find /usr/local/clamav/bin/*` ; do ln -s ${binaries} /usr/bin/; done

At this point Clam AntiVirus is installed and ready for use. Edit the configuration file and remove the line which says: Example It is there to ensure. If you want, you can look at other options but we don’t need to change anything else here to make ClamAV work for us.

vi /usr/local/clamav/etc/freshclam.conf #remove Example

Now let us run the freshclam which will download virus database and bring our virus database up to date. We should do this manually and make sure it didn’t give any errors. If this works, you will a lot of “downloading” messages.

/usr/bin/freshclam

If everything checks out, let us add this to our crontab to ensure our virus database is updated hourly. I chose to be updated every 9 minutes in to every hour. You can change to fit your needs or leave it as it is.

crontab -e

9 * * * * /usr/bin/freshclam –quiet

At this point our ClamAV virus database is up to date and now we can scan whichever directory we want. Go to the directory you want to scan and type:

clamscan -r -i

Once it is done scanning, it will display something similar to below.
-r parameter tells clamscan to recurse into directories
-i will print out infected filenames

----------- SCAN SUMMARY -----------
Known viruses: 159855
Engine version: 0.91.2
Scanned directories: 1437
Scanned files: 8836
Infected files: 0
Data scanned: 464.83 MB
Time: 103.678 sec (1 m 43 s)

Happy scanning!
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Note: I have installed their MySQL from source and installed MySQL in to /usr/local/mysql folder. Your installation path might be different.

service mysql stop
cd /usr/local/mysql
mv var var.bak
mkdir var
bin/mysql_install_db
chown -R mysql.mysql var
chmod 700 -R var
service mysql start

At this point we should have fresh MySQL db setup.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mkdir /backups/repos -p
for repostobackup in `ls /svn/` ; do
filename=`basename ${repostobackup}`
if [ ! $filename = "" ]; then
hot-backup.py --archive-type=bz2 /svn/${filename} /backups/repos
fi
done

This will create your backup files and compress them using bzip2. You can also use zip or gz if you prefer. Another thing you should do is to edit the hot-backup.py script and change num_backups to be whatever you think is a good number. I personally use: num_backups = 10

That’s all there is to it. Just add this script to your crons and your backups will be automated. You may want to consider moving your backups to another server by adding ftp, scp or rsync at the end of the script.

070815 15:31:46 InnoDB: ERROR: the age of the last checkpoint is 9433957,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

I poked around and found out our log files were set to default (5 mb) and they needed to be increased. After doing some calculations and research, I decided going to 50 mb at this point would be a good way to go. So I made the change in my.cnf file and added: innodb_log_file_size = 50M

I then stopped our mysql server and restarted it. And to my horror I saw following error messages show up in the mysql error logs:

070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'
070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'

I stopped the mysql server right away. I then remember, by help of a friend, that I have to remove the old log files if I change the size by changing innodb_log_file_size. So I issued a command: mv ib_logfile? ..

This command moves both ib_logfile1 and ib_logfile2 down one directory. I didn’t want to just remove ‘em so instead I moved them. After that, I restarted mysql again and to my comfort, everything came back up without errors. So lesson learned: if you change ib_logfile size by using innodb_log_file_size setting, make sure you move existing log files before you start mysql again.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.