For my purposes, I started with very basic installation of CentOS 5.5. I have done this on existing installations of CentOS without any issues as well. Ok so let’s start with getting the OpenVZ repository added to our system. This means that we will be going down “yum” path instead of “rpm” path. This is the quickest and easiest way to get OpenVZ installed.

cd /etc/yum.repos.d
wget http://download.openvz.org/openvz.repo
rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ

Before we move on, let’s turn off selinux:

vi /etc/selinux/config
:0,$s/=enforcing/=disabled/g
:wq

Go ahead and reboot the box and check:

[root@tooncent ~]# getenforce
Disabled

Now since we have the OpenVZ repository added and selinux disabled, we can go ahead and move on with installation part:
yum install vzkernel.x86_64 #install kernel with openvz support

echo '
# ----------added for OpenVZ installation----------------
# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
' >> /etc/sysctl.conf

Note: by executing above snippet, you will end up creating duplicate entries. It’s always a good idea to clean up duplicates to avoid confusion/mistakes later.

Following step is optional and mostly for clarity. Edit your grub.conf and change CentOS to say OpenVZ:

vi /etc/grub.conf

Change the first “title CentOS” to “title OpenVZ” (without the quotes). For example:

title CentOS (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

becomes:

title OpenVZ (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

Ok at this point we are ready to reboot and use the new kernel. Issue a “reboot”. Let’s install the utilities to manage OpenVZ:
yum install vzctl.x86_64 vzquota.x86_64

We need to start up vz manually once.

/etc/init.d/vz start

Alright! we got OpenVZ running. Now, let’s get a precreated template from OpenVZ site.

cd /vz/template/cache/
wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

Ok now comes the fun part; let’s get our first container up and running:
Note: ostemplate name is the file you downloaded minus .tar.gz. centos-5-x86_64.tar.gz

[root@tooncent cache]# vzctl create 101 --ostemplate centos-5-x86_64
Creating container private area (centos-5-x86_64)
Performing postcreate actions
Container private area was created
[root@tooncent cache]#

Container files are created under /vz/private/101 < – 101 is the id you give it. You can put any numeric number above 100. Documentation says not to use anything below 101 since its a reserved range.

Let’s set a hostname so there is no confusion:

vzctl set 101 --hostname toontest.tooncent.com --save
Set hostname: toontest.tooncent.com
Saved parameters for CT 101

To start your new container:

[root@tooncent ~]# vzctl start 101
Starting container ...
Container is mounted
Setting CPU units: 1000
Set hostname: toontest.tooncent.com
Container start in progress...
[root@tooncent ~]#

To access your container:

vzctl enter 101

To exit the container, type: exit

To stop the container:

# vzctl stop 101
Stopping container ...
Container was stopped
Container is unmounted

Congratulations! You have successfully setup a container.

As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be running the exact same version of the
build, it makes sense that I should only allow a user of my deployment
scripts to link a build which exists on all servers. But how do you
accomplish this in bash?

In most other languages, you have access to numerous array helping
functions that allow you to perform intersects, uniqs, and merges. My goal
was to do the same thing in bash without forking out to any external
binary. So how do you ensure that a particular thing exists on N number of
servers? Here it is:

function in_array() {
 local x
 ENTRY=$1
 shift 1
 ARRAY=( "$@" )
 [ -z "${ARRAY}" ] && return 1
 [ -z "${ENTRY}" ] && return 1
 for x in ${ARRAY[@]}; do
   [ "${x}" == "${ENTRY}" ] && return 0
 done
 return 1
}
 
MASTER=()
CURRENT=()
FIRST=1
for SERVER in ${SERVERS}; do
 # collect all builds from server and populate CURRENT list
 COMMAND="${LS} -1fd ${WEBROOT}/${SITE}.*"
 BUILDS=`${SSH} ${SSHOPTS} root@${SERVER} "${COMMAND}"`
 for BUILD in ${BUILDS}; do
   CURRENT=( ${CURRENT[@]-} ${BUILD} )
 done
 
 # if this is our first time around, copy CURRENT to MASTER
 if [ ${FIRST} -eq 1 ]; then
   MASTER=( ${CURRENT[@]} )
   FIRST=0
 fi
 
 # now we do a compare between MASTER and CURRENT to see what builds
 # are common
 INTERSECT=()
 for ENTRY in ${CURRENT[@]}; do
   in_array "${ENTRY}" "${MASTER[@]}"
   RET=$?
   if [ "${RET}" -eq 0 ]; then
     INTERSECT=( ${INTERSECT[@]-} ${ENTRY} )
   fi
 done
 MASTER=( ${INTERSECT[@]} )
 
 # clear the CURRENT array
 CURRENT=()
done

Let me take a moment to explain the code above:

• In order to check for array intersection, you need an in_array()
  function
  • The first argument as the “needle” and the second is the
    “haystack”
  • We verify that both parameters were passed
  • We simply loop through the haystack checking for the needle
  • If we find it, return success. Otherwise, eventually return
    false
• We need to loop through each server eventually, but we’ll start with
  the first one
  • Run an SSH command to get a listing of builds
  • Populate an array ($CURRENT) with the builds that were found
  • Since the first server has no previous server to compare with, so we
    just copy it to $MASTER
• We then loop to the 2nd server, and put the result of getting builds
  into $CURRENT
  • Now that we have the first server’s builds in $MASTER, we perform an
    intersect with $CURRENT
  • We realize the need for an $INTERSECT array to hold the intersections
    found above
  • $INTERSECT becomes $MASTER since it only contains similar builds from
    the 1st and 2nd server
• Looping to the 3rd server, we get the builds and put them in $CURRENT
  • Since $MASTER contains only the similar builds thus far, we again
    compare it with $CURRENT
  • The intersect can now be used to compare against builds on the 4th
    server, and so on
• Once you finish looping through all servers, your $MASTER should
  contain only similar builds

There are a few guides out there which show you how to do this via
forking, but I thought someone may appreciate the elegance of using 100%
bash to accomplish this. I hope this helps someone else out there!

for i in `find -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

I am sure there are many different ways to show file count for each folder in a directory and I am curious to see what people do so please do post comments with what you do.

Above command is pretty simple and can be expanded to do whatever you need.  For example, you can throw it into a bash script and be able to pass parameters.  For example:  count_files /home/  In this case your command line would look like:

for i in `find /home/ -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

only difference would be that /home/ would be argument you passed and therefore will be $1.  Here is a sample script for above example:

#!/bin/bash
for i in `find $1 -maxdepth 1 -type d`; do
echo -n $i " ";
find $i|wc -l;
done

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

./configure –prefix=/usr/local/curl –disable-ipv6

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Note:  you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz
tar zxpfv fuse-*.gz
cd fuse*
./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found
configure: error:
*** Please specify the location of the kernel source with
*** the '--with-kernel=SRCDIR' option
configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/
total 4.0K
drwxr-xr-x 18 root root 4.0K Oct  7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64
make && make install
cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz
tar zxpfv sshfs*
cd sshfs-fuse-*
./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

OR

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0′ found
No package ‘gthread-2.0′ found

You need to install glib2.  Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure
make && make install

After installation is done, we can move on with testing the installation:

cd /mnt
mkdir test
sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse
FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt
sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
...
sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test
...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

wget http://www.whatismyip.org

Well that does the trick.  But being lazy as I am, I did not want to have to cat the output file to find out the ip (plus there is no point of creating extra files and doing extra work to remove them).  So if you are ssh’ed in, you can issue following command (I am sure there is another way of doing it, but this is the quickest way I could think of):

wget -q -O - http://www.whatismyip.org

-O tells wget to redirect output to the following file (- being the standard out ).  So it basically echo’s output to our console.

-q makes wget run in  quiet mode so you do not see all of the connection/download/etc output.

That is it!  I am curious to know what other ways people use to get the same information.  Please share your way if possible.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mkdir temp
cd temp
touch Foo FooO "Foo Bar" "FOO BAaR"
\ls | while read -r FILENAME
do
mv -v "$FILENAME" `echo $FILENAME| tr ' ' '_'| tr '[A-Z]' '[a-z]'`
done

Note:  I intentionally have slash in front of ls (\ls).  \ls means that we want to make sure there is no ls alias overwriting our command. This is needed if your system has alias setup to display ls in a different way instead of default listing.  mv -v shows us the filenames being renamed as your script goes through the whole dir.  Your output should be like:

`Foo' -> `foo'
`FOO BAaR' -> `foo_baar'
`Foo Bar' -> `foo_bar'
`FooO' -> `fooo'

One of the very powerful commands in this post is the “tr” command. This command is not as popular as sed or awk but it is very useful and simple to use (read more about tr).

If you only needed to convert spaces to underscores and you are using CentOS/Fedora/Redhat, you can use this simpler method. NOTE: this command is not available on all distributions: rename " " "_" *

Learn more about rename command
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Install perl modules:

install Term::ReadKey
install File::Tail
install Time::HiRes

Now copy paste the script below and save it in a location and set +x attribute on it so you can execute it.  On my setup, I have the script under /usr/bin/httptop:

#!/usr/bin/perl -w
use Time::HiRes qw( time );
use File::Tail (  );
use Term::ReadKey;
use Getopt::Std;
use strict;
### Defaults you might be interested in adjusting.
my $Update = 2; # update every n secs
my $Backtrack = 250; # backtrack n lines on startup
my @Paths = qw(
%
/title/%/logs/access_log
/var/log/httpd/%/access_log
/usr/local/apache/logs/%/access_log
);
my $Log_Format = "combined";
my %Log_Fields = (
combined => [qw/ Host x x Time URI Response x Referer Client /],
vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
);
### Constants & other thingies. Nothing to see here. Move along.
my $Version = "0.4.1";
sub by_hits_per (  ) { $b->{Rate} <=> $a->{Rate} }
sub by_total (  ) { $b->{Total} <=> $a->{Total} }
sub by_age (  ) { $a->{Last} <=> $b->{Last} }
my $last_field = "Client";
my $index = "Host";
my $show_help = 0;
my $order = \&by_hits_per;
my $Help = "htlwufd?q";
my %Keys = (
h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
w => [ "Show remote host" => sub { $index = "Host" } ],
u => [ "Show requested URI" => sub { $index = "URI" } ],
f => [ "Show referring URL" => sub { $index = "Referer" } ],
d => [ "Show referring domain" => sub { $index = "Domain" } ],
'?' => [ "Help (this thing here)" => sub { $show_help++ } ],
q => [ "Quit" => sub { exit } ]
);
my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
my @Record_Fields = qw/ Host URI Referer Domain /;
my $Max_Index_Width = 50;
my $Initial_TTL = 50;
my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
my %Term = (
HOME => "\033[H",
CLS => "\033[2J",
START_TITLE => "\033]0;", # for xterms etc.
END_TITLE => "\007",
START_RV => "\033[7m",
END_RV => "\033[m"
);
my ( %hist, %opt, $spec );
$SIG{INT} = sub { exit };
END { ReadMode 0 };
### Subs.
sub refresh_output
{
my ( $cols, $rows ) = GetTerminalSize;
my $show = $rows - 3;
my $count = $show;
my $now = (shift || time);
for my $type ( values %hist ) {
for my $peer ( values %$type ) {
# if ( --$peer->{_Ttl} > 0 ) {
my $delta = $now - $peer->{Start};
if ( $delta >= 1 ) {
$peer->{ Rate } = $peer->{ Total } / $delta;
} else {
$peer->{ Rate } = 0
}
$peer->{ Last } = int( $now - $peer->{ Date } );
# } else {
# delete $type->{$peer}
# }
}
}
$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values %{$hist{$index}};
my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
my $first = 0;
$first = ( $first <= $_ ? $_ + 1 : $first ) for map { $_ ? length($_->{$index}) : 0 } @list;
$first = $Max_Index_Width if $Max_Index_Width < $first;
print $Term{START_TITLE}, "Monitoring $spec at: ", scalar localtime, $Term{END_TITLE} if $ENV{TERM} eq "xterm"; # UGLY!!!
my $help = "Help/?";
my $head = sprintf( "%-${first}s %6s %4s %4s %s (%d total)",
$index, qw{ Hits/s Tot Last }, $last_field,
scalar keys %{$hist{$index}}
);
#
# Truncate status line if need be
#
$head = substr($head, 0, ($cols - length($help)));
print @Term{"HOME", "START_RV"}, $head, " " x ($cols - length($head) - length($help)), $help, $Term{END_RV}, "\n";
for ( @list ) {
# $_->{_Ttl}++;
my $line = sprintf( "%-${first}s %6.3f %4d %3d %s",
substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last }, $last_field)} );
if ( length($line) > $cols ) {
substr( $line, $cols - 1 ) = "";
} else {
$line .= " " x ($cols - length($line));
}
print $line, "\n";
}
print " " x $cols, "\n" while $count++ < $show;
}
sub process_line
{
my $line = shift;
my $now = ( shift || time );
my %hit;
chomp $line;
@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /"([^"]+)"|\[([^]]+)\]|\s/o, $line ) );
$hit{ URI } =~ s/HTTP\/1\S+//gos;
$hit{ Referer } = "<unknown>" if not $hit{Referer} or $hit{Referer} eq "-";
( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
$hit{ Client } ||= "<none>";
$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
$hit{ Client } =~ s/[^\x20-\x7f]//gos;
# if $now is negative, try to guess how old the hit is based on the time stamp.
if ( $now < 0 ) {
my @hit_t = ( split( m![:/\s]!o, $hit{ Time } ))[ 0 .. 5 ];
my @now_t = ( localtime )[ 3, 4, 5, 2, 1, 0 ];
my @mag = ( 3600, 60, 1 );
# If the hit didn't parse right, or didn't happen today, the hell with it.
return unless $hit_t[2] == ( $now_t[2] + 1900 )
and $hit_t[1] eq $Months[ $now_t[1] ]
and $hit_t[0] == $now_t[0];
splice( @hit_t, 0, 3 );
splice( @now_t, 0, 3 );
# Work backward to the UNIX time of the hit.
$now = time;
$now -= (shift( @now_t ) - shift( @hit_t )) * $_ for ( 3600, 60, 1 );
}
$hit{ Date } = $now;
for my $field ( @Record_Fields ) {
my $peer = ( $hist{$field}{$hit{$field}} ||= { Start => $now, _Ttl => $Initial_TTL } );
@$peer{ @Display_Fields } = @hit{ @Display_Fields };
$peer->{ Total }++;
}
}
sub display_help {
my $msg = "httptop v.$Version";
print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, "\n\n";
print " " x 4, $_, " " x 8, $Keys{$_}[0], "\n" for ( split "", $Help );
print "\nPress any key to continue.\n";
}
### Init.
getopt( 'frb' => \%opt );
$Backtrack = $opt{b} if $opt{b};
$Update = $opt{r} if $opt{r};
$Log_Format = $opt{f} if $opt{f};
$spec = $ARGV[0];
die <<End unless $spec and $Log_Fields{$Log_Format};
Usage: $0 [-f <format>] [-r <refresh_secs>] [-b <backtrack_lines>] <logdir | path_to_log>
Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
End
for ( @Paths ) {
last if -r $spec;
( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
}
die "No access_log $ARGV[0] found.\n" unless -r $spec;
my $file = File::Tail->new(
name => $spec,
interval => $Update / 2,
maxinterval => $Update,
tail => $Backtrack,
nowait => 1
) or die "$spec: $!";
my $last_update = time;
my ( $line, $now );
# Backtracking.
while ( $Backtrack-- > 0 ) {
last unless $line = $file->read;
process_line( $line, -1 );
}
$file->nowait( 0 );
ReadMode 4; # Echo off.
print @Term{"HOME", "CLS"}; # Home & clear.
refresh_output;
### Main loop.
while (defined( $line = $file->read )) {
$now = time;
process_line( $line, $now );
while ( $line = lc ReadKey(-1) ) {
$show_help = 0 if $show_help;
$Keys{$line}[1]->(  ) if $Keys{$line};
}
if ( $show_help == 1 ) {
display_help;
$show_help++; # Don't display help again.
} elsif ( $now - $last_update > $Update and not $show_help ) {
$last_update = $now;
refresh_output( $now );
}
}

Save/exit and make sure you make it executable by setting it to +x (chmod +x httptop)

Now you can run httptop by typing:  httptop -f combined -r 1 /usr/local/apache2/logs/access_log

NOTE:  Your access_log file might be in different location.  Point to the right location.  This sets the refresh rate to 1 sec (-r 1).  Now you can run httptop any time you want to checkout how your http traffic is doing.  Remember to press “?” to get help once you are in.

—————

DISCLAIMER: As always, if you find any inaccurate information, please comment and let me know. When you do comment, make sure you give me some references to confirm.

Here are the instructions on how to get sphinxse compiled with your mysql installation on CentOS x64.  I am sure same instructions will work for other flavors but I have not tested it.  I will be compiling the most current version of sphinx (0.9.8) with most current stable version of mysql (5.0.51b) at the time of the writing.  Let’s get the appropriate packages first:

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8.tar.gz
wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/
tar zxpf sphinx*
tar zxpf mysql*

You will also need “bison”, “patch”, “automake” and “libtool” installed.  Let us just do a yum install for it.

yum -y install bison patch automake libtool

NOTE:  if you don’t install bison, you will get the following error:
sed '/^#/ s|y\.tab\.c|sql_yacc.cc|' y.tab.c >sql_yacc.cct && mv sql_yacc.cct sql_yacc.cc
sed: can't read y.tab.c: No such file or directory
make[2]: *** [sql_yacc.cc] Error 2

Let us continue with patching mysql source with sphinx storage engine (sphinxse) code and compile/install our new binaries.

cd mysql*
patch -p1 < ../sphinx-0.9.8/mysqlse/sphinx.5.0.37.diff #Make sure everything succeeded.
BUILD/autorun.sh
mkdir sql/sphinx
cp ../sphinx-0.9.8/mysqlse/* sql/sphinx
./configure --prefix=/usr/local/mysql --with-sphinx-storage-engine
make
make install

Now start your mysql installtion and check if engine support is compiled in:

mysql> show engines\G
Engine: SPHINX
Support: YES
Comment: Sphinx storage engine 0.9.8

To read more about how to use Sphinx storage engine, please refer to:  Sphinx documentation for using sphinx storage engine

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Following command will search for the string you specified. Generally this will give you all of the packages which has specified string in title or description. Most of the time you will have to look through a lot of output to find what you are looking for.

yum search string

Probably one of the most important options for yum is provides/whatprovides. If you know what command you need, you can find out what package you have to install in order to have that command available to you.

yum provides (or whatprovides) command

Following command is same as above but with less output.

yum -d 1 provides command

So for example if you are trying to figure out what you need to install to use bunzip2, type:

yum -d 1 provides bunzip2

you will get a similar output as below.

# yum -d 1 provides bunzip2
bzip2.x86_64 1.0.2-13.EL4.3 base
bzip2.x86_64 1.0.2-13.EL4.3 base
man-pages-fr.noarch 0.9.7-13.el4 base
man-pages-ja.noarch 20050215-2.EL4.3 base
man-pages-pl.noarch 0.23-5 base

As you can see bunzip2 is part of bzip2 package. So now you can you just install bzip2.x86_64 to get bunzip2.

To learn more about what else is available, read man yum.

mysql> CREATE DATABASE testdump;
mysql> USE testdump
mysql> CREATE TABLE `testtable` (
`id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`testfield` VARCHAR( 15 ) NOT NULL ,
`testfield2` VARCHAR( 15 ) NOT NULL
) ENGINE = innodb COMMENT = 'test table for dumping each row to file';
mysql> INSERT INTO `testtable` values ('','test1','test2'),('','test3','test4'),('','test5','test6');
mysql> select * from testtable;
+----+-----------+------------+
| id | testfield | testfield2 |
+----+-----------+------------+
| 1 | test1 | test2 |
| 2 | test3 | test4 |
| 3 | test5 | test6 |
+----+-----------+------------+
3 rows in set (0.00 sec)

Now we create a php script which we can use to select data from MySQL database and save it to a file under /tmp directory. You can easily modify this script to save it somewhere else, pull from multiple tables and/or save with different delimiter.

<?php
$link = mysql_connect('localhost', 'username', 'password');
if (!$link) {
die('Could not connect: ' . mysql_error());
}
$db = mysql_select_db('testdump',$link);
//specify what query to use to pull data.
$select = "SELECT * FROM `testtable`";
$result = mysql_query($select);
while($myrow = mysql_fetch_array($result))
{
extract($myrow);
//specify what query to use for create text files with csv data
//in our case we create files with testtable_ and ID which is unique
//per row. We use the same ID in where clause to limit one record at a time
//modify this part to your needs. Eg change delimiter from , to - or something else
$export = "
SELECT * INTO OUTFILE '/tmp/testtable_$id.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
FROM testtable where id=$id";
mysql_query($export);
}//loop until we are done exporting all records.
mysql_close($link);
?>

You can run this script from command line if you have php cli installed by issuing: php filename.php

Above example is a very simple example. t is merely a proof of concept. You can modify it to do whatever you need it to do. I would also put some error checking, not use extract() function, etc before using it in production environment.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Note: you must be logged in as root to do following.

vi /etc/passwd
find joe and change it to john, save/exit

vi /etc/group
find joe and change it to john, save/exit

vi /etc/shadow
find joe and change it to john. This file is read only and you have to force overwrite it. In vi it is :w! once saved, quit.

cd /home
mv joe john

And that should do the trick.

[Edited] Right after I posted this post, I was contacted and was told to look at utility called usermod. Read more about it man usermod. Got to love the quick responses.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

One of the things worth mentioning is that MySQL documentation states: “Symlinks are fully supported only for MyISAM tables. For files used by tables for other storage engines, you may get strange problems if you try to use symbolic links.” Keeping that in mind, if you have a innodb table and would like to create symlinks for, you should change the engine type to myisam before symlinking. Although, I have heard and personally used symlinks for innodb databases with no side affects.

First let us find out if your MySQL configuration allows symlinks:

mysql> SHOW VARIABLES LIKE 'have_symlink'\G
*************************** 1. row ***************************
Variable_name: have_symlink
Value: YES

I am going to make following assumptions:

• MySQL installation is in /usr/local/mysql.
• You want to create new tables under /mnt/another_partition/mysql/var
  1. you must create the dir: mkdir /mnt/another_partition/mysql/var -p
  2. Make it owned by mysql user: chown mysql /mnt/another_partition/mysql/var

We will create some test data. Following are the steps:

mysql> create database db1;
Query OK, 1 row affected (0.05 sec)
mysql> use db1;
Database changed
mysql> CREATE TABLE `users` ( `id` int(11) default NULL, `name` varchar(255) default NULL ) ENGINE=myisam, data directory="/mnt/another_partition/mysql/var", index directory="/mnt/another_partition/mysql/var";
Query OK, 0 rows affected (0.09 sec)
mysql> show tables;
+---------------+
| Tables_in_db1 |
+---------------+
| users |
+---------------+

Drop back to command prompt and “ls” your db1 directory as shown below. Here you can see that users table’s data file (users.MYD) is a symbolic link to the location we specified. You can see that users index file (users.MYI) is symlinked to the location we specified as well.

# ls -al /usr/local/mysql/var/db1/
total 24K
drwx------ 2 mysql mysql 4.0K May 2 00:03 ./
drwx------ 23 mysql mysql 4.0K May 1 23:59 ../
-rw-rw---- 1 mysql mysql 65 May 1 09:33 db.opt
-rw-rw---- 1 mysql mysql 8.4K May 2 00:03 users.frm
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYD -> /mnt/another_partition/mysql/var/users.MYD
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYI -> /mnt/another_partition/mysql/var/users.MYI

Here are the real files which above symlinks are pointing to:

# ls -al /mnt/another_partition/mysql/var/
total 12K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 0 May 2 00:03 users.MYD
-rw-rw---- 1 mysql mysql 1.0K May 2 00:03 users.MYI

If you go back in to mysql prompt and insert couple rows, you can see the size of users.MYD increase:

mysql> insert into db1.users values (1,"test1");
Query OK, 1 row affected (0.01 sec)
mysql> insert into db1.users values (2,"test2");
Query OK, 1 row affected (0.01 sec)
# ls -al /mnt/another_partition/mysql/var/
total 16K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 40 May 2 00:08 users.MYD
-rw-rw---- 1 mysql mysql 1.0K May 2 00:08 users.MYI

If you already have data file and index file you would like to move, I recommend shutting down your MySQL server and moving the files manually and create symbolic links by using “ln -s” command. You can read more about ln by doing man ls

NOTE: Even though MySQL documentation says this is only supported 100% for myisam tables, I have read few posts where people say they have done this with innodb tables without any problems. If anybody else can comment on this, I would greatly appreciate it.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

CREATE TABLE `t1` (
`a` char(1) default NULL,
`b` smallint(6) default NULL
);
insert into t1 values ('y','1');

I have a table t1 which has column a and b, I want column a to be updated to ‘n’ when column b = 0. Here is the first version I created:

DELIMITER |
CREATE TRIGGER trigger1 AFTER UPDATE ON t1
FOR EACH ROW UPDATE t1 SET a= 'n' WHERE b=0;
|
DELIMITER ;

The trigger created successfully but I got this error when I tried to do an update on column b on table t1:
mysql> update t1 set b=0;
ERROR 1442 (HY000): Can't update table 't1' in stored function/trigger because it is already used by statement which invoked this stored function/trigger.


After searching online for a while and trying different solutions, I finally found a way to update the table which has trigger on it:

drop trigger trigger1;
DELIMITER |
CREATE TRIGGER trigger1 BEFORE UPDATE ON t1
FOR EACH ROW
BEGIN
IF NEW.b=0 THEN
SET NEW.a = 'n';
END IF;
END
|
DELIMITER ;

After the new trigger is in, I issued the same update query and “ERROR 1442 (HY000): Can’t update table ‘t1′ in stored function/trigger because it is already used by statement which invoked this stored function/trigger.” didn’t show up and it updated the col a value to “n” as it suppose to.

mysql> update t1 set b=0;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from t1\G
*************************** 1. row ***************************
a: n
b: 0

Therefore, if you want to create a trigger on the table which will update itself, make sure you use the NEW.column_name to refer to the row after it’s updated and don’t do the full update statement!

However, if you are updating some other table, then you can use the regular update statement:

DELIMITER |
CREATE TRIGGER trigger1 AFTER UPDATE ON t1
FOR EACH ROW UPDATE t2 SET a= 'n' WHERE b=0;
|
DELIMITER ;

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Master1 server ip: 10.0.0.1
Master2 server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Let us go through the steps you must take on Master1 to enable it to act as master and slave by using following configuration which goes under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=1
# Replication Master Server
# binary logging is required for replication
log-bin=master1-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 1
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.2
master-user = slaveuser
master-password = slavepw
master-port = 3306

Following is the configuration which goes on master2 under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=2
# Replication Master Server
# binary logging is required for replication
log-bin=master2-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 2
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.1
master-user = slaveuser
master-password = slavepw
master-port = 3306

On master1 server, go to mysql> prompt and add the appropriate user:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

On master2 server do the same but allow right ip:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.1' identified by 'slavepw';

Restart both of the master servers and check slave status:

mysql> show slave status\G

That is all you have to do to set up the replication. Of course there are a lot more configuration options but this should get your replication going and you can tweak from here on.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

1) Offload some of the queries from one server to another and spread the load: One of the biggest advantages to have master-slave set up in MySQL is to be able to use master for all of the inserts and send some, if not all, select queries to slave. This will most probably speed up your application without having to diving into optimizing all the queries or buying more hardware.

2) Do backups from slave: One of the advantages people overlook is that you can use MySQL slave to do backups from. That way site is not affected at all when doing backups. This becomes a big deal when your database has grown to multiple gigs and every time you do backups using mysqldump, site lags when table locks happen. For some sites, this could mean that site goes down for few secs to minutes. If you have slave, you just take slave out of rotation (should be built into code) and run backups off the slave. You can even stop slave MySQL instance and copy the var folder instead of doing mysqldump.

Ok let us dive into how to setup master-slave replication under MySQL. There are many configuration changes you can do to optimize your MySQL set up. I will just touch on very basic ones to get the replication to work. Here are some assumptions:

Master server ip: 10.0.0.1
Slave server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Put the following in your master my.cnf file under [mysqld] section:

# changes made to do master
server-id = 1
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
log-bin = /usr/local/mysql/var/mysql-bin
# end master

Copy the following to slave’s my.cnf under [mysqld] section:

# changes made to do slave
server-id = 2
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
# end slave setup

Create user on master:
mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

Do a dump of data to move to slave
mysqldump -u root --all-databases --single-transaction --master-data=1 > masterdump.sql

import dump on slave
mysql < masterdump.sql

After dump is imported go in to mysql client by typing mysql. Let us tell the slave which master to connect to and what login/password to use:
mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.1', MASTER_USER='slaveuser', MASTER_PASSWORD='slavepw';

Let us start the slave:
mysql> start slave;

You can check the status of the slave by typing
mysql> show slave status\G

The last row tells you how many seconds its behind the master. Don’t worry if it doesn’t say 0, the number should be going down over time until it catches up with master (at that time it will show Seconds_Behind_Master: 0) If it shows NULL, it could be that slave is not started (you can start by typing: start slave) or it could be that it ran into an error (shows up in Last_errno: and Last_error under show slave status\G).

Related posts:

• How do I dump all tables in a database into separate files?
• How do I import individual table dump files in to MySQL using shell script?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Memcached is a very popular open source object caching server. It was developed to speed up livejournal.com by Danga Interactive.  We use memcached for a lot of our sites.  We use it for different purposes but one main purpose is to cache query results so we don’t have to keep hitting database.  As most of the people who work with databases know it is costly to keep hitting database for same information over and over.

When you run the Memcached daemon, it runs and listens on a specific port. One of the things Memcached does lack is security. Memcached will let anybody who can make a connection to its port have full access to all objects. So you would have to run a firewall to block unauthorized access. It is usually wise to do put firewall on it even if you trust everybody on the same network since accidents do happen. That said, let’s get memcached installed!

Let’s get libevent which is required by Memcached:

wget http://monkey.org/~provos/libevent-1.3e.tar.gz
tar zxpfv libevent*
cd libevent*
./configure
make install

Now let’s download the newest Memcached source (at time of update to this post 1.4.5 was the latest)

wget http://memcached.googlecode.com/files/memcached-1.4.5.tar.gz
tar zxpfv memcached*
cd memcached*
./configure
make install

Let’s add Memcached user to run daemon as since we don’t need it to run as root privileges.

adduser memcached

We will start the server to use 48 megs of ram (-m 48), listen on ip 10.0.0.2 (-l 10.0.0.2) and run on port 11211 (-p 11211) as user memcached (-u memcached)

./memcached -u memcached -d -m 48 -l 10.0.0.2 -p 11211

If you get the following error (which you will get if you are doing this under CentOS 64 bit):

./memcached: error while loading shared libraries: libevent-1.3e.so.1: cannot open shared object file: No such file or directory

You can fix this by simply doing this:

ln -s /usr/local/lib/libevent-1.3e.so.1 /lib64/

That is all there is to it. You can see if daemon is running by telneting to the port.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

python -V

Let us download and install readline:

wget ftp://ftp.cwru.edu/pub/bash/readline-5.2.tar.gz
tar zxf readline-5.2.tar.gz
cd readline-5.2
./configure
make install

Now let us install python with readline support:

wget http://www.python.org/ftp/python/2.5.1/Python-2.5.1.tgz
tar zxf Python-2.5.1.tgz
cd Python-2.5.1


I had some problems on one of the servers where it would not compile readline support in to python. I was able to compile reading support in to python by:

echo "readline readline.c -lreadline -ltermcap" >> Modules/Setup.local

Now let us continue with python installation.

./configure --prefix=/usr/local/python-2.5.1 --enable-readline
make -i install

If you do not do “make -i install”, install may fail with following error (-i means ignore any errors):
Compiling /usr/lib/python2.5/zipfile.py ...
make: *** [libinstall] Error 1

If you scroll up, you will find following error which seems to be the root cause:

Compiling /usr/lib/python2.5/test/test_multibytecodec.py ...
Sorry: UnicodeError: ("\\N escapes not supported (can't load unicodedata module)",)

Once you do python install with make -i install, that library (unicodedata.so which is not built until later stage of build process) gets installed. If you want, you can type make install once again (without ignoring errors) and it will complete without errors.

Once you have python installed, you would want to use the new version. I like to keep a backup of old files in case I have to use older version for any reason. Run following which creates symbolic links and makes backups of current files:

for binaries in `find /usr/local/python-2.5.1/bin/*` ; do
mv /usr/bin/`basename ${binaries}` /usr/bin/`basename ${binaries}`.bak
ln -s ${binaries} /usr/bin/`basename ${binaries}`
done

Now let us get phpsh and try it out:

wget http://www.phpsh.org/phpsh-latest.tgz
tar zxf phpsh-latest.tgz
cd ../phpsh
chmod +x phpsh
./phpsh

At this point you should be at the shell: php>

Following is a snippet from README file which comes with phpsh. You should take a look since it has more details on how to use phpsh:

Type php commands and they will be evaluated each time you hit enter. Ex:
php> $msg = "hello world"
Put = at the beginning of a line as syntactic sugar for return. Ex:
php> = 2 + 2
If you end a line with a backlash (\), you can enter multi-line input.
For example,
php> print "like \
... this"
like this
php>

There we go. Now you have a great interactive php shell prompt. Note that there is interactive mode built into php as well. I personally do not like it as much but you can try it out for yourself by typing:

php -a
Interactive mode enabled
echo "Hello world";
Hello world

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

#!/bin/bash
db=$1
if [ "$db" = "" ]; then
echo "Usage: $0 db_name"
exit 1
fi
mkdir done
clear
for sql_file in *.sql; do
echo "Importing $sql_file";
mysql $db< $sql_file;
mv $sql_file done;
done

Related posts:

• How do I dump all tables in a database into separate files?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

perl -w -i -p -e "s/search_text/replace_text/g" filename

-w turns warnings on
-i makes Perl operate on files in-place (if you would like to make backups, use -i.bak, this will save filename.bak files)
-p loops over the whole input
-e specifies Perl expression
filename works on one file at a time

Once we get the results we want, we can now pass it multiple files by doing something like:

perl -w -i -p -e "s/search_text/replace_text/g" *.php

This will search and replace within all php files in the directory you are in.

Now, let us say you want to go through your whole web directory and replace every place where you have “Perl is good” to “Perl is great”, you would use following command:

find /www_root -name "*.php"|xargs perl -w -i -p -e "s/Perl is good/perl is great/g"

find will start at /www_root, look for filenames which have .php extension, xargs takes that filename and passes it to perl as an arguement.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

yum -y install gmp-devel
wget http://freshmeat.net/redir/clamav/29355/url_tgz/clamav-0.91.2.tar.gz
adduser -M -s /bin/false clamav
tar zxf clamav-0.91.2.tar.gz
cd clamav-0.91.2
./configure --prefix=/usr/local/clamav
make install
for binaries in `find /usr/local/clamav/bin/*` ; do ln -s ${binaries} /usr/bin/; done

At this point Clam AntiVirus is installed and ready for use. Edit the configuration file and remove the line which says: Example It is there to ensure. If you want, you can look at other options but we don’t need to change anything else here to make ClamAV work for us.

vi /usr/local/clamav/etc/freshclam.conf #remove Example

Now let us run the freshclam which will download virus database and bring our virus database up to date. We should do this manually and make sure it didn’t give any errors. If this works, you will a lot of “downloading” messages.

/usr/bin/freshclam

If everything checks out, let us add this to our crontab to ensure our virus database is updated hourly. I chose to be updated every 9 minutes in to every hour. You can change to fit your needs or leave it as it is.

crontab -e

9 * * * * /usr/bin/freshclam –quiet

At this point our ClamAV virus database is up to date and now we can scan whichever directory we want. Go to the directory you want to scan and type:

clamscan -r -i

Once it is done scanning, it will display something similar to below.
-r parameter tells clamscan to recurse into directories
-i will print out infected filenames

----------- SCAN SUMMARY -----------
Known viruses: 159855
Engine version: 0.91.2
Scanned directories: 1437
Scanned files: 8836
Infected files: 0
Data scanned: 464.83 MB
Time: 103.678 sec (1 m 43 s)

Happy scanning!
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Before we went ahead and re-imported, we made a decision to move to innodb for *most* of our crucial tables. After altering on master, we did a dump/import. Converting tables to innodb got rid of our data corruption problem but our missing rows issue is still there. Hardware on the master is 4×1.86Ghz, 8 gigs of ram; slaves hardware is: 4×1.86, 4 gigs of ram.

Here is the config running on our master:
< some of the part of the config has been cut off to keep it short/simple and to the point >
skip-locking
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 1G
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_concurrency = 8
skip-name-resolve
max_allowed_packet = 32M
#low-priority-updates
table_cache = 1024
thread_cache = 8
query_cache_size = 64M
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 16
log_slow_queries
log_long_format
#
# INNODB Settings
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 4G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Here is the config running on our slave:

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql1.sock
pid-file = /usr/local/mysql/var/mysql.pid
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
server-id = 31
skip-slave-start
skip-locking
skip-name-resolve
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 512M
max_allowed_packet = 1M
table_cache = 512M
myisam_sort_buffer_size = 64M
thread_cache_size = 4
read-only
#slave-skip-errors=all
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
read_buffer_size = 2M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
query_cache_size = 16M # changed to 16 3/1/07
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
log_slow_queries
log_long_format
#
# INNODB Settings
default_table_type = INNODB
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 2G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Please feel free to comment/scrutinize any of these settings. If you think they should be different, please give quick summary about why they should be different. Some of these settings were inherited from old setup of mysql (where they had it hosted before).

Right now we are just trying to find out why we have missing rows and why our slaves lag behind constantly. We have refreshed data almost every day for last two weeks. We barely have load average above 1 or 2. What can we do to track down the problem? What are the things we should be looking out for?

Note: I have installed their MySQL from source and installed MySQL in to /usr/local/mysql folder. Your installation path might be different.

service mysql stop
cd /usr/local/mysql
mv var var.bak
mkdir var
bin/mysql_install_db
chown -R mysql.mysql var
chmod 700 -R var
service mysql start

At this point we should have fresh MySQL db setup.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mkdir /backups/repos -p
for repostobackup in `ls /svn/` ; do
filename=`basename ${repostobackup}`
if [ ! $filename = "" ]; then
hot-backup.py --archive-type=bz2 /svn/${filename} /backups/repos
fi
done

This will create your backup files and compress them using bzip2. You can also use zip or gz if you prefer. Another thing you should do is to edit the hot-backup.py script and change num_backups to be whatever you think is a good number. I personally use: num_backups = 10

That’s all there is to it. Just add this script to your crons and your backups will be automated. You may want to consider moving your backups to another server by adding ftp, scp or rsync at the end of the script.

070815 15:31:46 InnoDB: ERROR: the age of the last checkpoint is 9433957,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

I poked around and found out our log files were set to default (5 mb) and they needed to be increased. After doing some calculations and research, I decided going to 50 mb at this point would be a good way to go. So I made the change in my.cnf file and added: innodb_log_file_size = 50M

I then stopped our mysql server and restarted it. And to my horror I saw following error messages show up in the mysql error logs:

070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'
070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'

I stopped the mysql server right away. I then remember, by help of a friend, that I have to remove the old log files if I change the size by changing innodb_log_file_size. So I issued a command: mv ib_logfile? ..

This command moves both ib_logfile1 and ib_logfile2 down one directory. I didn’t want to just remove ‘em so instead I moved them. After that, I restarted mysql again and to my comfort, everything came back up without errors. So lesson learned: if you change ib_logfile size by using innodb_log_file_size setting, make sure you move existing log files before you start mysql again.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.