Here are the instructions on how to get sphinxse compiled with your mysql installation on CentOS x64.  I am sure same instructions will work for other flavors but I have not tested it.  I will be compiling the most current version of sphinx (0.9.8) with most current stable version of mysql (5.0.51b) at the time of the writing.  Let’s get the appropriate packages first:

wget http://www.sphinxsearch.com/downloads/sphinx-0.9.8.tar.gz
wget http://dev.mysql.com/get/Downloads/MySQL-5.0/mysql-5.0.51b.tar.gz/from/http://mysql.he.net/
tar zxpf sphinx*
tar zxpf mysql*

You will also need “bison”, “patch”, “automake” and “libtool” installed.  Let us just do a yum install for it.

yum -y install bison patch automake libtool

NOTE:  if you don’t install bison, you will get the following error:
sed '/^#/ s|y\.tab\.c|sql_yacc.cc|' y.tab.c >sql_yacc.cct && mv sql_yacc.cct sql_yacc.cc
sed: can't read y.tab.c: No such file or directory
make[2]: *** [sql_yacc.cc] Error 2

Let us continue with patching mysql source with sphinx storage engine (sphinxse) code and compile/install our new binaries.

cd mysql*
patch -p1 < ../sphinx-0.9.8/mysqlse/sphinx.5.0.37.diff #Make sure everything succeeded.
BUILD/autorun.sh
mkdir sql/sphinx
cp ../sphinx-0.9.8/mysqlse/* sql/sphinx
./configure --prefix=/usr/local/mysql --with-sphinx-storage-engine
make
make install

Now start your mysql installtion and check if engine support is compiled in:

mysql> show engines\G
Engine: SPHINX
Support: YES
Comment: Sphinx storage engine 0.9.8

To read more about how to use Sphinx storage engine, please refer to:  Sphinx documentation for using sphinx storage engine

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

So if you just want to replace these with appropriate symbols, you may do it with a simple sql query. Note that below queries are without where clause. You may what to test it with one of your rows before making changes to the whole table. Of course, you should always backup your data before you try this out. If you have a dev system, that is even better. I put all my sql queries into a file ex: fix.sql and sourced it with mysql client.

vi fix.sql

update table_name set fieldname = replace(fieldname, '’', '\'');
update table_name set fieldname = replace(fieldname, '…','...');
update table_name set fieldname = replace(fieldname, '–','-');
update table_name set fieldname = replace(fieldname, '“','"');
update table_name set fieldname = replace(fieldname, '”','"');
update table_name set fieldname = replace(fieldname, '‘','\'');
update table_name set fieldname = replace(fieldname, '•','-');
update table_name set fieldname = replace(fieldname, '‡','c');

Save/exit.

# mysql
mysql> source fix.sql;

I am not sure if I am missing any other chars. If you know of any other chars, please comment with them and I will add on to the script here.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Following are the instructions on how to install innotop on CentOS x64/Fedora/RHEL (Redhat enterprise). Most probably same instructions can be used on all flavors of Linux. If not, leave me a comment and I will research a solution for you. Let us start with downloading innotop. I used version 1.6.0 which is the latest at the time of writing.

wget http://internap.dl.sourceforge.net/sourceforge/innotop/innotop-1.6.0.tar.gz

Now let us go ahead and unzip and create the MakeFile to get it ready for install

tar zxpf innotop-1.6.0.tar.gz
cd innotop-1.6.0
perl Makefile.PL

At this point if you get the following output, you are good to continue:

Checking if your kit is complete...
Looks good
Writing Makefile for innotop

If you get something similar to following, you will need to take care of the prerequisites:

Looks good
Warning: prerequisite DBD::mysql 1 not found.
Warning: prerequisite DBI 1.13 not found.
Warning: prerequisite Term::ReadKey 2.1 not found.
Writing Makefile for innotop

Just because they are warnings does not mean you ignore them. So let us install those prerequisites. We will use perl’s cpan shell to get this installed (visit my post on how to install perl modules for more details). If it is your first time starting this up, you will have to answer some questions. Defaults will work fine in all cases.

perl -MCPAN -eshell
install Term::ReadKey
install DBI
install DBD::mysql

Note: you must install DBI before you can install DBD::mysql.

If you get an error like following when you are installing DBD::mysql:

Error: Can't load '/root/.cpan/build/DBD-mysql-4.007/blib/arch/auto/DBD/mysql/mysql.so' for module DBD::mysql: libmysqlclient.so.15: cannot open shared object file: No such file or directory at /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/DynaLoader.pm line 230.

You will have to create a symlink to the object file in your lib64 (or lib if you are not using x64 version) folder:

ln -s /usr/local/mysql/lib/mysql/libmysqlclient.so.15 /lib64/

Once all prerequisites are done, type perl Makefile.PL and you should have no warnings. Continue the install:

make install

At this point you should have innotop installed on your system. Let us do some quick set up so you can start using innotop. We start with configuring your .my.cnf to include connection directives.

vi ~/.my.cnf

Add the following (edit to reflect your install) and save/exit

[mysql]
port = 3306
socket = /tmp/mysql.sock

Start up innotop by typing innotop at your shell prompt. First prompt will ask you to “Enter a name:”. I just put localhost since this will be used to connect locally. Next prompt asks you about DSN entry. I use: DBI:mysql:;mysql_read_default_group=mysql

This tells innotop to read .my.cnf file and use group [mysql] directives. Next prompt is optional (I just press enter). Next two prompts you enter information if you need to.

At this point your innotop installation / testing is complete. You can read man innotop to get more details on how to use innotop.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

So today we decided that we are going to migrate one of our master database servers to new hardware.  Since we got the hardware this morning and we wanted to move on to it asap, we decided that we will take our slave down, copy data from it, and bring it up on future master server.  At that point, we will let it run as slave to the current master server until its time for us to take it down.  Reason we did that instead of mysqldump/import was to avoid the lag mysqldump creates on our server.

After we did all this and put up the new master server, we started to notice odd issues.  After looking around and comparing old db with new, we found out that new db was missing data.  How it happened is beyond me and is the reason why I am writing this.  We never had issues with the slave which would cause data to be lost; so what happened to those missing rows?  Is this something which is common?  Can we not trust our slave enough to use it as master if master died?  Can we not run backups off the slave with confident that our data is protected and up to date so to keep load down on our master?  All these questions which keep me awake and wondering…

mysql> CREATE DATABASE testdump;
mysql> USE testdump
mysql> CREATE TABLE `testtable` (
`id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`testfield` VARCHAR( 15 ) NOT NULL ,
`testfield2` VARCHAR( 15 ) NOT NULL
) ENGINE = innodb COMMENT = 'test table for dumping each row to file';
mysql> INSERT INTO `testtable` values ('','test1','test2'),('','test3','test4'),('','test5','test6');
mysql> select * from testtable;
+----+-----------+------------+
| id | testfield | testfield2 |
+----+-----------+------------+
| 1 | test1 | test2 |
| 2 | test3 | test4 |
| 3 | test5 | test6 |
+----+-----------+------------+
3 rows in set (0.00 sec)

Now we create a php script which we can use to select data from MySQL database and save it to a file under /tmp directory. You can easily modify this script to save it somewhere else, pull from multiple tables and/or save with different delimiter.

<?php
$link = mysql_connect('localhost', 'username', 'password');
if (!$link) {
die('Could not connect: ' . mysql_error());
}
$db = mysql_select_db('testdump',$link);
//specify what query to use to pull data.
$select = "SELECT * FROM `testtable`";
$result = mysql_query($select);
while($myrow = mysql_fetch_array($result))
{
extract($myrow);
//specify what query to use for create text files with csv data
//in our case we create files with testtable_ and ID which is unique
//per row. We use the same ID in where clause to limit one record at a time
//modify this part to your needs. Eg change delimiter from , to - or something else
$export = "
SELECT * INTO OUTFILE '/tmp/testtable_$id.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
FROM testtable where id=$id";
mysql_query($export);
}//loop until we are done exporting all records.
mysql_close($link);
?>

You can run this script from command line if you have php cli installed by issuing: php filename.php

Above example is a very simple example. t is merely a proof of concept. You can modify it to do whatever you need it to do. I would also put some error checking, not use extract() function, etc before using it in production environment.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

One of the things worth mentioning is that MySQL documentation states: “Symlinks are fully supported only for MyISAM tables. For files used by tables for other storage engines, you may get strange problems if you try to use symbolic links.” Keeping that in mind, if you have a innodb table and would like to create symlinks for, you should change the engine type to myisam before symlinking. Although, I have heard and personally used symlinks for innodb databases with no side affects.

First let us find out if your MySQL configuration allows symlinks:

mysql> SHOW VARIABLES LIKE 'have_symlink'\G
*************************** 1. row ***************************
Variable_name: have_symlink
Value: YES

I am going to make following assumptions:

• MySQL installation is in /usr/local/mysql.
• You want to create new tables under /mnt/another_partition/mysql/var
  1. you must create the dir: mkdir /mnt/another_partition/mysql/var -p
  2. Make it owned by mysql user: chown mysql /mnt/another_partition/mysql/var

We will create some test data. Following are the steps:

mysql> create database db1;
Query OK, 1 row affected (0.05 sec)
mysql> use db1;
Database changed
mysql> CREATE TABLE `users` ( `id` int(11) default NULL, `name` varchar(255) default NULL ) ENGINE=myisam, data directory="/mnt/another_partition/mysql/var", index directory="/mnt/another_partition/mysql/var";
Query OK, 0 rows affected (0.09 sec)
mysql> show tables;
+---------------+
| Tables_in_db1 |
+---------------+
| users |
+---------------+

Drop back to command prompt and “ls” your db1 directory as shown below. Here you can see that users table’s data file (users.MYD) is a symbolic link to the location we specified. You can see that users index file (users.MYI) is symlinked to the location we specified as well.

# ls -al /usr/local/mysql/var/db1/
total 24K
drwx------ 2 mysql mysql 4.0K May 2 00:03 ./
drwx------ 23 mysql mysql 4.0K May 1 23:59 ../
-rw-rw---- 1 mysql mysql 65 May 1 09:33 db.opt
-rw-rw---- 1 mysql mysql 8.4K May 2 00:03 users.frm
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYD -> /mnt/another_partition/mysql/var/users.MYD
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYI -> /mnt/another_partition/mysql/var/users.MYI

Here are the real files which above symlinks are pointing to:

# ls -al /mnt/another_partition/mysql/var/
total 12K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 0 May 2 00:03 users.MYD
-rw-rw---- 1 mysql mysql 1.0K May 2 00:03 users.MYI

If you go back in to mysql prompt and insert couple rows, you can see the size of users.MYD increase:

mysql> insert into db1.users values (1,"test1");
Query OK, 1 row affected (0.01 sec)
mysql> insert into db1.users values (2,"test2");
Query OK, 1 row affected (0.01 sec)
# ls -al /mnt/another_partition/mysql/var/
total 16K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 40 May 2 00:08 users.MYD
-rw-rw---- 1 mysql mysql 1.0K May 2 00:08 users.MYI

If you already have data file and index file you would like to move, I recommend shutting down your MySQL server and moving the files manually and create symbolic links by using “ln -s” command. You can read more about ln by doing man ls

NOTE: Even though MySQL documentation says this is only supported 100% for myisam tables, I have read few posts where people say they have done this with innodb tables without any problems. If anybody else can comment on this, I would greatly appreciate it.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Slave server options:

• replicate-do-db = dbname (or) replicate-do-db = dbname1, dbname2, …, dbnameN
  This option is used on slave server to tell the server to only replicate dbname db on this particular host. You would want to use this if you have a master which is replicating to multiple slaves and each slave may contain different database for read performance reasons.
• replicate-ignore-db = dbname (or) replicate-ignore-db = dbname1, dbname2, …, dbnameN
  This option is used on slave server to tell server to ignore database(s) listed. This is used when you want to replicate every database except certain individual ones. For example, you may want to replicate-ignore-db=testdb
• replicate-do-table = dbname.tablename
  This specifies a table from a database to be replicated.
• replicate-ignore-table = dbname.tablename
  This is very useful and often ignored. If you have logging table which you only do writes to but never read from, there is no real point to replicate that table to slave(s). This way you ignore specific tables.
• replicate-wild-do-table=dbname.tablename%
  This is another option which can prove itself to be very useful. Let’s say you have database with multiple type of open source installations (phpbb, wordpress, drupal, etc), and you want to designate slave(s) to only be used for phpbb, you would do: replicate-wild-do-table=dbname.phpbb%
• replicate-wild-ignore-table=dbname.tablename%
  Another option which can be used instead of above example. Say you wanted everything on this particular slave BUT phpbb tables. You would put this in your config: replicate-wild-ignore-table=dbname.phpbb%

NOTE:  for wildcards, you can use generic mysql wildcards.

Master server options:

• binlog-do-db = dbname (or) binlog-do-db = dbname1, dbname2, …, dbnameN
  This option is used on master server to tell the master server to only log queries for dbname db. This is helpful if your master has multiple databases but you only want to replicate selected few to all slaves.
• binlog-ignore-db = dbname (or) binlog-ignore-db = dbname1, dbname2, …, dbnameN
  Ignore specified database and do not log queries referring to list database(s).

In order for mysql to know which database queries to log (or not to log) you have to make sure you specify what is your default db by doing mysql_select_db() or issuing “use dbname” command. For example, if you had binlog-do-db=forums and you issue a query: delete from main.users (basically saying, delete everything from users table in main database), it will still log the command even though main database is not in the binlog-do-db list. To get around this, you should do the following: use main; delete from users; If you do not do that, your query “delete from main.users” will be logged and executed on slave(s). There is a high probability that you do not have a test database on your slave.

NOTE: This post is part of series on MySQL Replication. Here is the original post: MySQL Replication Series: How does MySQL replication work?

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

In a MySQL replication setup master server puts all the commands it is executing which affects data (insert, delete, update, truncate, etc) into a file referred to as bin-log file. Slave(s) than pick up the big-log files and execute those commands on themselves. You can see which bin file is currently being used by typing: show master status, you would see output similar to:

mysql> show master status\G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 519132576

This tells us that right now we are writing to mysql-bin.000001 and it is at position 519132576. Why does this position matter? It matters if you are looking at your slaves and trying to figure out where it is currently as far as position goes. If you were to do mysqldump and do it with master info, it is this number which gets added to the dump file so when you import on slave, it knows where to start from.

At this point in time if you go to slave, and type: show slave status\G, you would see something similiar to below:

mysql> show slave status\G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.0.0.1
Master_User: slaveuser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 519132576

As you can see it is at the same point meaning it is all caught up to the master (this snapshot was taken from a site which was not accepting traffic at the time). Another thing you can look at, even though it is not the most accurate way to measure how behind you are, is to check last row in “show slave status” labeled: Seconds_Behind_Master: 0 <- if this is greater than 0, than you are lagging behind the master. Which in fact just means that it has not played back some of the queries. If it is NULL, it means that your slave is either not started or it has errors which needs attention. You can start and stop the slave by typing: start slave; and stop slave; respectively.

Hopefully this gives an overview of how MySQL replication works. As always, please feel free to correct me, add on to what I have said or just make comments. I am, like many others, learning new things every day.

Tip #1: What should be replicated and what should not be replicated?
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

CREATE TABLE `t1` (
`a` char(1) default NULL,
`b` smallint(6) default NULL
);
insert into t1 values ('y','1');

I have a table t1 which has column a and b, I want column a to be updated to ‘n’ when column b = 0. Here is the first version I created:

DELIMITER |
CREATE TRIGGER trigger1 AFTER UPDATE ON t1
FOR EACH ROW UPDATE t1 SET a= 'n' WHERE b=0;
|
DELIMITER ;

The trigger created successfully but I got this error when I tried to do an update on column b on table t1:
mysql> update t1 set b=0;
ERROR 1442 (HY000): Can't update table 't1' in stored function/trigger because it is already used by statement which invoked this stored function/trigger.


After searching online for a while and trying different solutions, I finally found a way to update the table which has trigger on it:

drop trigger trigger1;
DELIMITER |
CREATE TRIGGER trigger1 BEFORE UPDATE ON t1
FOR EACH ROW
BEGIN
IF NEW.b=0 THEN
SET NEW.a = 'n';
END IF;
END
|
DELIMITER ;

After the new trigger is in, I issued the same update query and “ERROR 1442 (HY000): Can’t update table ‘t1′ in stored function/trigger because it is already used by statement which invoked this stored function/trigger.” didn’t show up and it updated the col a value to “n” as it suppose to.

mysql> update t1 set b=0;
Query OK, 1 row affected (0.01 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> select * from t1\G
*************************** 1. row ***************************
a: n
b: 0

Therefore, if you want to create a trigger on the table which will update itself, make sure you use the NEW.column_name to refer to the row after it’s updated and don’t do the full update statement!

However, if you are updating some other table, then you can use the regular update statement:

DELIMITER |
CREATE TRIGGER trigger1 AFTER UPDATE ON t1
FOR EACH ROW UPDATE t2 SET a= 'n' WHERE b=0;
|
DELIMITER ;

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Master1 server ip: 10.0.0.1
Master2 server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Let us go through the steps you must take on Master1 to enable it to act as master and slave by using following configuration which goes under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=1
# Replication Master Server
# binary logging is required for replication
log-bin=master1-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 1
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.2
master-user = slaveuser
master-password = slavepw
master-port = 3306

Following is the configuration which goes on master2 under [mysqld] section:

# let's make it so auto increment columns behave by having different increments on both servers
auto_increment_increment=2
auto_increment_offset=2
# Replication Master Server
# binary logging is required for replication
log-bin=master2-bin
binlog-ignore-db=mysql
binlog-ignore-db=test
# required unique id between 1 and 2^32 - 1
server-id = 2
#following is the slave settings so this server can connect to master2
master-host = 10.0.0.1
master-user = slaveuser
master-password = slavepw
master-port = 3306

On master1 server, go to mysql> prompt and add the appropriate user:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

On master2 server do the same but allow right ip:

mysql> grant replication slave on *.* to slaveuser@'10.0.0.1' identified by 'slavepw';

Restart both of the master servers and check slave status:

mysql> show slave status\G

That is all you have to do to set up the replication. Of course there are a lot more configuration options but this should get your replication going and you can tweak from here on.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

1) Offload some of the queries from one server to another and spread the load: One of the biggest advantages to have master-slave set up in MySQL is to be able to use master for all of the inserts and send some, if not all, select queries to slave. This will most probably speed up your application without having to diving into optimizing all the queries or buying more hardware.

2) Do backups from slave: One of the advantages people overlook is that you can use MySQL slave to do backups from. That way site is not affected at all when doing backups. This becomes a big deal when your database has grown to multiple gigs and every time you do backups using mysqldump, site lags when table locks happen. For some sites, this could mean that site goes down for few secs to minutes. If you have slave, you just take slave out of rotation (should be built into code) and run backups off the slave. You can even stop slave MySQL instance and copy the var folder instead of doing mysqldump.

Ok let us dive into how to setup master-slave replication under MySQL. There are many configuration changes you can do to optimize your MySQL set up. I will just touch on very basic ones to get the replication to work. Here are some assumptions:

Master server ip: 10.0.0.1
Slave server ip: 10.0.0.2
Slave username: slaveuser
Slave pw: slavepw
Your data directory is: /usr/local/mysql/var/

Put the following in your master my.cnf file under [mysqld] section:

# changes made to do master
server-id = 1
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
log-bin = /usr/local/mysql/var/mysql-bin
# end master

Copy the following to slave’s my.cnf under [mysqld] section:

# changes made to do slave
server-id = 2
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
# end slave setup

Create user on master:
mysql> grant replication slave on *.* to slaveuser@'10.0.0.2' identified by 'slavepw';

Do a dump of data to move to slave
mysqldump -u root --all-databases --single-transaction --master-data=1 > masterdump.sql

import dump on slave
mysql < masterdump.sql

After dump is imported go in to mysql client by typing mysql. Let us tell the slave which master to connect to and what login/password to use:
mysql> CHANGE MASTER TO MASTER_HOST='10.0.0.1', MASTER_USER='slaveuser', MASTER_PASSWORD='slavepw';

Let us start the slave:
mysql> start slave;

You can check the status of the slave by typing
mysql> show slave status\G

The last row tells you how many seconds its behind the master. Don’t worry if it doesn’t say 0, the number should be going down over time until it catches up with master (at that time it will show Seconds_Behind_Master: 0) If it shows NULL, it could be that slave is not started (you can start by typing: start slave) or it could be that it ran into an error (shows up in Last_errno: and Last_error under show slave status\G).

Related posts:

• How do I dump all tables in a database into separate files?
• How do I import individual table dump files in to MySQL using shell script?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

#!/bin/bash
db=$1
if [ "$db" = "" ]; then
echo "Usage: $0 db_name"
exit 1
fi
mkdir done
clear
for sql_file in *.sql; do
echo "Importing $sql_file";
mysql $db< $sql_file;
mv $sql_file done;
done

Related posts:

• How do I dump all tables in a database into separate files?
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mysqldump database_name table1 > table1.sql
mysqldump database_name table2 > table2.sql

This could be very painful if you have 10’s or 100’s of tables. Until today, I never ran into a situation where I had to deal with dumping more than few tables at a time. Today I had to do a dump of 181 tables. I was not going to sit there and type in that command with 181 table names. It is not just time consuming but it is also stupid. So I wrote this script to help me with this task. We still use mysqldump command as described above, except we do it programmatically to make it easier on us:

#!/bin/bash
db=$1
if [ "$db" = "" ]; then
echo "Usage: $0 db_name"
exit 1
fi
mkdir $$
cd $$
clear
for table in `mysql $db -e 'show tables' | egrep -v 'Tables_in_' `; do
echo "Dumping $table"
mysqldump --opt -Q $db $table > $table.sql
done
if [ "$table" = "" ]; then
echo "No tables found in db: $db"
fi

You can also compress your files by adding bzip2, zip or any other compression commands after mysqldump command. Here is the same script with bzip2 command added:
#!/bin/bash
db=$1
if [ "$db" = "" ]; then
echo "Usage: $0 db_name"
exit 1
fi
mkdir $$
cd $$
clear
for table in `mysql $db -e 'show tables' | egrep -v 'Tables_in_' `; do
echo "Dumping $table"
mysqldump --opt -Q $db $table > $table.sql
bzip2 $table.sql
done
if [ "$table" = "" ]; then
echo "No tables found in db: $db"
fi

I do not recommend doing compression on a production server since most compression program put descent amount of load on the server. Also note that this will delay your dump considerably. You may also want to use different parameters for running mysqldump. Type man mysqldump in your shell to read more.

Related posts:

• MySQL database backup file compress gzip vs bzip2
• MySQL backups using mysqldump

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

[NDBD]
id=2
HostName=10.0.0.2   # the IP of the first data node
DataMemory=6G
IndexMemory=512M

But when I tried the: 2 restart in the mysql cluster managment console (ndb_mgm), it didn’t re-read the new configuration.  DataMemory was initially set to 2G and it continued using that configuration.  I even tried: 2 stop, and than logged on the node server and started ndbd manually, but it refused to read the new configuration.  It is only when I did ndbd –initial, nodes read in the configuration and were able to use more than 2 gigs of ram.

Did anybody else run into this?  Am I reading the documentation on MySQL site incorrectly?

Before we went ahead and re-imported, we made a decision to move to innodb for *most* of our crucial tables. After altering on master, we did a dump/import. Converting tables to innodb got rid of our data corruption problem but our missing rows issue is still there. Hardware on the master is 4×1.86Ghz, 8 gigs of ram; slaves hardware is: 4×1.86, 4 gigs of ram.

Here is the config running on our master:
< some of the part of the config has been cut off to keep it short/simple and to the point >
skip-locking
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 1G
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_concurrency = 8
skip-name-resolve
max_allowed_packet = 32M
#low-priority-updates
table_cache = 1024
thread_cache = 8
query_cache_size = 64M
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 16
log_slow_queries
log_long_format
#
# INNODB Settings
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 4G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Here is the config running on our slave:

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql1.sock
pid-file = /usr/local/mysql/var/mysql.pid
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
server-id = 31
skip-slave-start
skip-locking
skip-name-resolve
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 512M
max_allowed_packet = 1M
table_cache = 512M
myisam_sort_buffer_size = 64M
thread_cache_size = 4
read-only
#slave-skip-errors=all
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
read_buffer_size = 2M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
query_cache_size = 16M # changed to 16 3/1/07
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
log_slow_queries
log_long_format
#
# INNODB Settings
default_table_type = INNODB
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 2G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Please feel free to comment/scrutinize any of these settings. If you think they should be different, please give quick summary about why they should be different. Some of these settings were inherited from old setup of mysql (where they had it hosted before).

Right now we are just trying to find out why we have missing rows and why our slaves lag behind constantly. We have refreshed data almost every day for last two weeks. We barely have load average above 1 or 2. What can we do to track down the problem? What are the things we should be looking out for?

Note: I have installed their MySQL from source and installed MySQL in to /usr/local/mysql folder. Your installation path might be different.

service mysql stop
cd /usr/local/mysql
mv var var.bak
mkdir var
bin/mysql_install_db
chown -R mysql.mysql var
chmod 700 -R var
service mysql start

At this point we should have fresh MySQL db setup.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

070815 15:31:46 InnoDB: ERROR: the age of the last checkpoint is 9433957,
InnoDB: which exceeds the log group capacity 9433498.
InnoDB: If you are using big BLOB or TEXT rows, you must set the
InnoDB: combined size of log files at least 10 times bigger than the
InnoDB: largest such row.

I poked around and found out our log files were set to default (5 mb) and they needed to be increased. After doing some calculations and research, I decided going to 50 mb at this point would be a good way to go. So I made the change in my.cnf file and added: innodb_log_file_size = 50M

I then stopped our mysql server and restarted it. And to my horror I saw following error messages show up in the mysql error logs:

070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'
070815 17:37:40 [ERROR] /usr/local/mysql/libexec/mysqld: Incorrect information in file: './dbname/table_name.frm'

I stopped the mysql server right away. I then remember, by help of a friend, that I have to remove the old log files if I change the size by changing innodb_log_file_size. So I issued a command: mv ib_logfile? ..

This command moves both ib_logfile1 and ib_logfile2 down one directory. I didn’t want to just remove ‘em so instead I moved them. After that, I restarted mysql again and to my comfort, everything came back up without errors. So lesson learned: if you change ib_logfile size by using innodb_log_file_size setting, make sure you move existing log files before you start mysql again.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

mysql> set @@long_query_time=3;
Query OK, 0 rows affected (0.00 sec)mysql> show variables like 'long%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| long_query_time | 3 |
+-----------------+-------+
1 row in set (0.00 sec)

Once you are done checking for slow queries, you can reset it back to whatever value you want by repeating the command with higher number (default is 10).

mysql> set @@long_query_time=10;
Query OK, 0 rows affected (0.00 sec
mysql> show variables like 'long%';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| long_query_time | 10 |
+-----------------+-------+
1 row in set (0.00 sec)

There are many more variables you can change.  Type show variables in the mysql client to see the list and what all those variables are set to.

1. Replace all the bulbs in your house with energy savings one. Energy savings bulbs will save you upto 70% in utility bills and at the same time help out to make world greener by using less electricity.
2. Unplug appliances which are not in use. Even some of the chargers like your laptop charger, consumes some energy while plugged in to outlet even when there is nothing is plugged in the other side. That home entertainment system or just a cable box you have sitting around, consumes energy. All this could add upto significant $$’s. One of the products worth looking at: Smart Strip Power Strip
3. If you can afford to and have a house, put solar panels on your roof. Even though this is sort of an expensive way to go but in long (or short, depends on costs etc.) run it can save you money an maybe even make you some money. You can also depreciate this investment just like any other investments. Talk to your tax adviser for more details on tax credits. You can read more details about return on your investment at: motherearthnews.com
4. Windmills. Yeah..not the coolest thing around but if you are living in an area with lots of yard space and do qualify to put windmills there, its worth looking into.
5. Buy hybrids! Not only that you get some tax credit for buying hybrids, you can also pay A LOT less on gas stations. Of course there are many more alternatives then hybrids. You can also buy electric cars, cars which run on ethanol (http://www.msnbc.msn.com/id/11603199/) and cars which are powered by solar energy. Hybrid seems to be more mainstream choice even though for short commutes electric cars make more sense. One of the car I know of is Tesla Roadster which by no means is a cheap car! It could cost you are $100k to buy this puppy but before you say anything, check out the specs. You also have to wait until next year to get it so maybe that is too much a turn off and you rather have one of the many available hybrids.
6. Use less paper! Opt in for online statements. This alone will save a lot paper and in turn will protect the trees! Other places you can cut back would be at your work. Ask for your paystubs electronically instead of paper. You really don’t need a piece of paper to see how much you made. But if you really want to, just log in to the website and look. File your taxes via computer using e-file. All these are little things which does not make our life harder but helps with making sure we are not destroying our nature!
7. Use recycled material! Promote recycling. Ask for recycling at your work, your apartment, church, where ever you can!
8. Paper or plastic? ALWAYS pick paper. Plastic bags are VERY hard to recycle.
9. If you live close to stores, work or any other destinations, walk there instead of driving. You save on some pollution and at the same time its good for you to go for walk.
10. Make your home energy efficient. There are many ways to make this happen. Of course you are already doing #1 by now, right? Other ways would be to buy appliances which qualify. Install windows which insulate your house better. Put insulation in the outside walls. All these will make it so you don’t use all that electricity you are producing by following #3. Which in turn means you are pumping electricity back in the power grid and making some money from doing it!

This list is very brief but should be enough to get you started and thinking about things you can do to make this world a better place to live! Please feel free to add comments with your own suggestions, comments, etc.
Tagged as mysql so hopefully more people can benefit from this

After searching for a bit on google about this problem, apparently only way you can do that is by exporting your mysql databases, delete ibdata1 file, import databases. This creates new ibdata file with correct space usage. Atleast there is a way to get around this issue. But honestly, too much pain on production boxes where we might be trying to remove old databases to reclaim some of the hard drive space.

An preventive measure one can use is to use option: innodb_file_per_table (‘put innodb_file_per_table’ in your my.cnf file under [mysqld] section). This will create individual files for tables under database directory. So now when I delete the database, all the space is returned since the directory is now deleted along with database along with all the tables inside the directory. In my test after you put option innodb_file_per_table your my.cnf, you will have to still do export/import to be able to minimize disk usage and have the ability to delete database at your leisure without worrying about reclaiming the disk space. Here are the steps I took. DISCLAIMER: Please make backup of your data and use following steps at your own risk. Doing it on test server is HIGHLY recommended. Please don’t come back and tell me that you lost your data because you followed my steps. They work for me and they may not work for you!

That said, here are the steps:

1. Add innodb_file_per_table in your my.cnf file under [mysqld] section
2. run following commands at the prompt. Your path to binaries might be different.

#note: following assumes you are logged in as root
mkdir -p /temp #temp dir to save our sql dump
#lets make a backup of current database. -p is used if there is pw set
/usr/local/mysql/bin/mysqldump -R -q -p --all-databases > /temp/all.sql
#stop mysql so we can remove all the files in the dir
/etc/init.d/mysql stop
rm -fr /usr/local/mysql/var/* #remove all the files
/usr/local/mysql/bin/mysql_install_db #install default dbs
#change ownership so mysql user can read/write to/from files
chown -R mysql.mysql /usr/local/mysql/var/
#start mysql so we can import our dump
/etc/init.d/mysql start
#note there is no -p since defaults don't have mysql pw set
/usr/local/mysql/bin/mysql < /temp/all.sql
/etc/init.d/mysql restart

This should be all you need to do. At this point when you remove a database, it will delete the directory of the db and all the data contained within which in turn will give you your disk space back.

REMEMBER: Backup your data and be smart about using code found on internet. If you don’t know what you are doing, hire a consultant who does.

I went and did few tests to see if its beneficial for us to use Bzip2 instead of gzip. I could’ve tried 7z also but that is not something installed on most linux machines I work on and I didn’t want use a solution which will require me to add more software. Here are the results:

Original sql file size: 584M
Using gzip: 166M
Time spent on compressing 1 minute 25 secs
Using bzip2: 125M
Time spent on compressing 1 minute 50 secs

As we can see that it takes longer to compress but file size is much smaller (adds up fast for multiple backups). As our database grows bigger and bigger, the size difference will matter quite a bit. Since we also ftp the file off the server to onsite/offsite location hourly as well.

Does anybody know of any backup techniques which we can use for free and we can do incrementals with? comments are always welcome.  To learn more about gzip or bzip2, see man gzip or man bzip2 respectively.

We took out the wait timeout and everything seems to started to work fine. Did anybody ever notice this behavior where you would loose connection to the mysql server due to timeout? The script which processes line by line and deletes line by line takes fraction of second to process that particular line. Does wait timeout starts counting from the starting of the connection? Does it mean that wait timeout is actually a max connection time limit? Suggestions/comments?

Edit 5/31/09: Friend of mine was getting this error: database error: Lost connection to MySQL server during query
he got around it by adjusting wait_timeout setting.

Intially I was doing backup by using: mysqldump dbname > weekdayHour.dbname.sql hourly. This allowed us to have week worth of backups done every hour and auto overwriting old backups. Eventually we added stored procedures and triggers to the mix and all of the sudden this dump wasn’t getting all the stored procedures and triggers. So we started using mysqldump with -R parameter which in man states:

Dump stored routines (functions and procedures) from the dumped databases.

Recently when we started to see more and more traffic, we noticed that our server was under heavy load on the hour. Ofcourse we quickly found it was due to mysqldump running on the hour which was causing the lag. Eventually we had enough traffic on the site where this was causing connection problems with mysql. So I went through man mysqldump again to find solution. And thanks to those smart people in mysqldump dev team, there I found couple more parameters to make this backup process little less painful. At this point we are using: mysqldump -R -q –single-transaction > weekdayHour.dbname.sql This seems to have decreased the load on the server and we haven’t got errors connecting to mysql. Since we use innodb tables in this particular db, we could use single-transaction parameter. Here is what our friendly man mysqldump has to say about these two parameters:

-q …This option is useful for dumping large tables. It forces mysqldump to retrieve rows for a table from the server a row at a time rather than retrieving the entire row set and buffering it in memory before writing it out…

–single-transaction …This option issues a BEGIN SQL statement before dumping data from the server. It is useful only with transactional tables such as InnoDB and BDB, because then it dumps the consistent state of the database at the time when BEGIN was issued without blocking any applications…

If you are thinking about using these two parameters, please spend couple minutes reading through man mysqldump and look at other notes which might pertain to your setup.

One last but very important thing we do after our backups are run is to move the new backup files off the server just in case server dies. I achieved this by using ncftp package which includes ncftpput command line utility. With ncftp, you can store ip/login/pw in a file and tell ncftp to use that for login information. Lets look at the script as a whole:

NOTE: comments in this script is for information purposes only. You can remove them if you like

#!/bin/bash
#
#http://crazytoon.com/2007/01/23/mysql-backups-using-mysqldump/
#
DATE=`date '+%u%H'` # this sets up weekly rotation of backup files
BACKUP_DIR="/admin/backups/domain.com/" # this dir will be created if it doesn't exist
HOST=`hostname` #you may want to hard code this if you hostname returns wrong information
mkdir ${BACKUP_DIR}mysql/ -p
/usr/local/mysql/bin/mysqldump -R -q --single-transaction --databases dbname1 dbname2 -ppassword > ${BACKUP_DIR}mysql/$HOST$DATE.sql
rm ${BACKUP_DIR}mysql/$HOST$DATE.sql.gz > /dev/null 2>&1
gzip -9 ${BACKUP_DIR}mysql/$HOST$DATE.sql > /dev/null 2>&1
/usr/bin/ncftpput -R -f ${BACKUP_DIR}hostinfo / ${BACKUP_DIR}mysql/$HOST$DATE.sql.gz

This is what your hostinfo file should contain (just make sure you edit it and put your own server ip, username, and password):
host 123.123.123.123
user loginname
pass loginpassword

To read more about mysqldump, see man mysqldump