MySQL: How do I dump each record from a table to a separate files in csv format?

I honestly do not know why somebody would want to export each record from a table in to its’ own files in a csv format. I am sure people have their own reasons. But since I got request from couple people, I figure I would post a solution here. Same script can be used to dump the whole table in to one csv file as well, with little tweaking. I will start with creating database with a table. I then insert three rows with test data into the table just to show three separate files creation.

mysql> CREATE DATABASE testdump;
mysql> USE testdump
mysql> CREATE TABLE `testtable` (
`id` TINYINT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY ,
`testfield` VARCHAR( 15 ) NOT NULL ,
`testfield2` VARCHAR( 15 ) NOT NULL
) ENGINE = innodb COMMENT = 'test table for dumping each row to file';
mysql> INSERT INTO `testtable` values ('','test1','test2'),('','test3','test4'),('','test5','test6');
mysql> select * from testtable;
+----+-----------+------------+
| id | testfield | testfield2 |
+----+-----------+------------+
| 1 | test1 | test2 |
| 2 | test3 | test4 |
| 3 | test5 | test6 |
+----+-----------+------------+
3 rows in set (0.00 sec)

Now we create a php script which we can use to select data from MySQL database and save it to a file under /tmp directory. You can easily modify this script to save it somewhere else, pull from multiple tables and/or save with different delimiter.

<?php
$link = mysql_connect('localhost', 'username', 'password');
if (!$link) {
die('Could not connect: ' . mysql_error());
}
$db = mysql_select_db('testdump',$link);
//specify what query to use to pull data.
$select = "SELECT * FROM `testtable`";
$result = mysql_query($select);
while($myrow = mysql_fetch_array($result))
{
extract($myrow);
//specify what query to use for create text files with csv data
//in our case we create files with testtable_ and ID which is unique
//per row. We use the same ID in where clause to limit one record at a time
//modify this part to your needs. Eg change delimiter from , to - or something else
$export = "
SELECT * INTO OUTFILE '/tmp/testtable_$id.txt'
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"'
LINES TERMINATED BY '\n'
FROM testtable where id=$id";
mysql_query($export);
}//loop until we are done exporting all records.
mysql_close($link);
?>

You can run this script from command line if you have php cli installed by issuing: php filename.php

Above example is a very simple example. t is merely a proof of concept. You can modify it to do whatever you need it to do. I would also put some error checking, not use extract() function, etc before using it in production environment.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Linux: How do you rename a user account in linux?

In Linux, there is no command which will rename a user account. If you make a mistake creating a user account, user changes their name or if user does not like his user name, there is no real easy way of going and making the change. Only thing I know you can do is to go through some files and rename user manually. Let us say that we have a user who is named joe and we want to rename him to john.

Note: you must be logged in as root to do following.

vi /etc/passwd
find joe and change it to john, save/exit

vi /etc/group
find joe and change it to john, save/exit

vi /etc/shadow
find joe and change it to john. This file is read only and you have to force overwrite it. In vi it is :w! once saved, quit.

cd /home
mv joe john

And that should do the trick.

[Edited] Right after I posted this post, I was contacted and was told to look at utility called usermod. Read more about it man usermod. Got to love the quick responses.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL: How do you use symlinks with MySQL tables?

Creating table symlinks in MySQL is very easy and it is probably one of the features a lot of MySQL users overlook or never think about. You may ask, why would I want to use symlinks? Some of the reasons you would want symlinks is if you are running low on disk space on the partition where your data generally is and/or you want to move one (or more) table(s) on to a different disk/partition for performance reasons.

One of the things worth mentioning is that MySQL documentation states: “Symlinks are fully supported only for MyISAM tables. For files used by tables for other storage engines, you may get strange problems if you try to use symbolic links.” Keeping that in mind, if you have a innodb table and would like to create symlinks for, you should change the engine type to myisam before symlinking. Although, I have heard and personally used symlinks for innodb databases with no side affects.

First let us find out if your MySQL configuration allows symlinks:

mysql> SHOW VARIABLES LIKE 'have_symlink'\G
*************************** 1. row ***************************
Variable_name: have_symlink
Value: YES

I am going to make following assumptions:

  • MySQL installation is in /usr/local/mysql.
  • You want to create new tables under /mnt/another_partition/mysql/var
    1. you must create the dir: mkdir /mnt/another_partition/mysql/var -p
    2. Make it owned by mysql user: chown mysql /mnt/another_partition/mysql/var

We will create some test data. Following are the steps:

mysql> create database db1;
Query OK, 1 row affected (0.05 sec)
mysql> use db1;
Database changed
mysql> CREATE TABLE `users` ( `id` int(11) default NULL, `name` varchar(255) default NULL ) ENGINE=myisam, data directory=”/mnt/another_partition/mysql/var”, index directory=”/mnt/another_partition/mysql/var”;
Query OK, 0 rows affected (0.09 sec)
mysql> show tables;
+—————+
| Tables_in_db1 |
+—————+
| users |
+—————+

Drop back to command prompt and “ls” your db1 directory as shown below. Here you can see that users table’s data file (users.MYD) is a symbolic link to the location we specified. You can see that users index file (users.MYI) is symlinked to the location we specified as well.

# ls -al /usr/local/mysql/var/db1/
total 24K
drwx------ 2 mysql mysql 4.0K May 2 00:03 ./
drwx------ 23 mysql mysql 4.0K May 1 23:59 ../
-rw-rw---- 1 mysql mysql 65 May 1 09:33 db.opt
-rw-rw---- 1 mysql mysql 8.4K May 2 00:03 users.frm
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYD -> /mnt/another_partition/mysql/var/users.MYD
lrwxrwxrwx 1 mysql mysql 42 May 2 00:03 users.MYI -> /mnt/another_partition/mysql/var/users.MYI

Here are the real files which above symlinks are pointing to:

# ls -al /mnt/another_partition/mysql/var/
total 12K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 0 May 2 00:03 users.MYD
-rw-rw—- 1 mysql mysql 1.0K May 2 00:03 users.MYI

If you go back in to mysql prompt and insert couple rows, you can see the size of users.MYD increase:

mysql> insert into db1.users values (1,"test1");
Query OK, 1 row affected (0.01 sec)
mysql> insert into db1.users values (2,"test2");
Query OK, 1 row affected (0.01 sec)
# ls -al /mnt/another_partition/mysql/var/
total 16K
drwxr-xr-x 2 mysql root 4.0K May 2 00:03 ./
drwxr-xr-x 3 root root 4.0K May 1 23:39 ../
-rw-rw---- 1 mysql mysql 40 May 2 00:08 users.MYD
-rw-rw—- 1 mysql mysql 1.0K May 2 00:08 users.MYI

If you already have data file and index file you would like to move, I recommend shutting down your MySQL server and moving the files manually and create symbolic links by using “ln -s” command. You can read more about ln by doing man ls

NOTE: Even though MySQL documentation says this is only supported 100% for myisam tables, I have read few posts where people say they have done this with innodb tables without any problems. If anybody else can comment on this, I would greatly appreciate it.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Linux: How do you display specific information at login prompt in Linux on the console?

Buddy of mine asked me a question over chat today: “how do I show my machines’ IP at login prompt with out logging in?” He is referring to his Virtual Machine in this case. He does not want to have to log in to the server to see what ip it has (since its on dhcp) for him to ssh in or hit it from the browser. I could have answered him with a simple how to but what is the fun in that? So I decided to give some background on how login prompts are done and show what can be done.

When Linux server boots up, it calls a program called mingetty. This program creates that infamous login prompt as show in a screenshot:

Screenshot of default CentOS login prompt on x86_64 server

You can see how server calls the mingetty program by looking at /etc/inittab. You will see a block like below:

# Run gettys in standard runlevels
1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6

When mingetty is called, it reads a file called /etc/issue. There are a lot of existing variables you use. For example the prompt shown above in example is:

CentOS release 5 (Final)
Kernel \r on an \m

All of the available variables are list here on this page: http://www.lamp-tips.com/man-pages/mingetty/

If you were paying attention in the beginning of the post and looked at all of the available variables you can use, you would notice there is nothing in there which talks about IP. Since there is no variables available, you would have to sort of hack it. Put following in your /etc/rc.local file:

echo "CentOS release 5 (Final)" > /etc/issue
echo "Kernel \r on an \m" >> /etc/issue
echo `ifconfig eth0 | grep inet | cut -d : -f 2 | cut -d \ -f 1` >> /etc/issue

Getting an ip part can be modified to check another adapter such as eth1, eth2, etc. Or you can get it another way if you want. That all there is too. Reboot, and you should see your changes.

[correction 4.24.08] you do not have to reboot to see your changes.  Just press enter at the login prompt in console and it will re-read the issue file.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL Replication Series (tip #1): what should be replicated and what should not be replicated?

Welcome to Tip #1 in MySQL Replication Series. In this tip we will go over what to do when you only want to replicate certain data to slave(s). Most general way to tell what is replicated to which slave is to include following configuration directive in my.cnf file depending on your environment and your goals. We will start with slave server side options since you have more flexibility on slave on what to replicate and what not to.

Slave server options:

  • replicate-do-db = dbname (or) replicate-do-db = dbname1, dbname2, …, dbnameN
    This option is used on slave server to tell the server to only replicate dbname db on this particular host. You would want to use this if you have a master which is replicating to multiple slaves and each slave may contain different database for read performance reasons.
  • replicate-ignore-db = dbname (or) replicate-ignore-db = dbname1, dbname2, …, dbnameN
    This option is used on slave server to tell server to ignore database(s) listed. This is used when you want to replicate every database except certain individual ones. For example, you may want to replicate-ignore-db=testdb
  • replicate-do-table = dbname.tablename
    This specifies a table from a database to be replicated.
  • replicate-ignore-table = dbname.tablename
    This is very useful and often ignored. If you have logging table which you only do writes to but never read from, there is no real point to replicate that table to slave(s). This way you ignore specific tables.
  • replicate-wild-do-table=dbname.tablename%
    This is another option which can prove itself to be very useful. Let’s say you have database with multiple type of open source installations (phpbb, wordpress, drupal, etc), and you want to designate slave(s) to only be used for phpbb, you would do: replicate-wild-do-table=dbname.phpbb%
  • replicate-wild-ignore-table=dbname.tablename%
    Another option which can be used instead of above example. Say you wanted everything on this particular slave BUT phpbb tables. You would put this in your config: replicate-wild-ignore-table=dbname.phpbb%

NOTE:  for wildcards, you can use generic mysql wildcards.

Master server options:

  • binlog-do-db = dbname (or) binlog-do-db = dbname1, dbname2, …, dbnameN
    This option is used on master server to tell the master server to only log queries for dbname db. This is helpful if your master has multiple databases but you only want to replicate selected few to all slaves.
  • binlog-ignore-db = dbname (or) binlog-ignore-db = dbname1, dbname2, …, dbnameN
    Ignore specified database and do not log queries referring to list database(s).

In order for mysql to know which database queries to log (or not to log) you have to make sure you specify what is your default db by doing mysql_select_db() or issuing “use dbname” command. For example, if you had binlog-do-db=forums and you issue a query: delete from main.users (basically saying, delete everything from users table in main database), it will still log the command even though main database is not in the binlog-do-db list. To get around this, you should do the following: use main; delete from users; If you do not do that, your query “delete from main.users” will be logged and executed on slave(s). There is a high probability that you do not have a test database on your slave.

NOTE: This post is part of series on MySQL Replication. Here is the original post: MySQL Replication Series: How does MySQL replication work?

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

« Previous PageNext Page »