Category Archives: Enterprise level solutions

MySQL Cluster: Changing datamemory requires node restart with –initial?

7 Replies

I ran into something with mysql cluster today which boggles my mind.  On http://dev.mysql.com/doc/refman/5.0/en/mysql-cluster-config-params-ndbd.html page, it is documented that if we you change datamemory parameter in the config.ini under mysql-cluster like below, you would have to restart nodes to reread the configuration.

[NDBD]
id=2
HostName=10.0.0.2   # the IP of the first data node
DataMemory=6G
IndexMemory=512M

But when I tried the: 2 restart in the mysql cluster managment console (ndb_mgm), it didn’t re-read the new configuration.  DataMemory was initially set to 2G and it continued using that configuration.  I even tried: 2 stop, and than logged on the node server and started ndbd manually, but it refused to read the new configuration.  It is only when I did ndbd –initial, nodes read in the configuration and were able to use more than 2 gigs of ram.

Did anybody else run into this?  Am I reading the documentation on MySQL site incorrectly?

Linux: How do I mass find and replace text in files under linux using perl?

408 Replies

Few friends have asked me how to do mass find and replace text in files under linux. There are quite a bit of options in linux to achieve mass replacing of text in files. If you are doing it file by file, you can achieve that in vi by opening, running replace and closing and going to next file. But sometimes that can be very tedious and you would rather do mass replacement on all files containing certain extension. We can do this by using sed or perl. Since most people are familiar with perl (at least most system admins and programmers), I will show you a perl way of doing it which you can use with sed as well if you wish. First step is to get perl to do what we want on one file

perl -w -i -p -e "s/search_text/replace_text/g" filename

-w turns warnings on
-i makes Perl operate on files in-place (if you would like to make backups, use -i.bak, this will save filename.bak files)
-p loops over the whole input
-e specifies Perl expression
filename works on one file at a time

Once we get the results we want, we can now pass it multiple files by doing something like:

perl -w -i -p -e "s/search_text/replace_text/g" *.php

This will search and replace within all php files in the directory you are in.

Now, let us say you want to go through your whole web directory and replace every place where you have “Perl is good” to “Perl is great”, you would use following command:

find /www_root -name "*.php"|xargs perl -w -i -p -e "s/Perl is good/perl is great/g"

find will start at /www_root, look for filenames which have .php extension, xargs takes that filename and passes it to perl as an arguement.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Linux virus scan: How do I check my linux installation for viruses using ClamAV? (CentOS, Linux, Redhat)

3 Replies

There are quite a bit of antivirus software exist for Linux nowadays. One of the popular antivirus software is ClamAV. We will install Clam AntiVirus software from source and install it to a custom directory. We will also install gmp-devel package which installs GMP library. GMP library is used to verify the digital signature of the virus database.

yum -y install gmp-devel
wget http://freshmeat.net/redir/clamav/29355/url_tgz/clamav-0.91.2.tar.gz
adduser -M -s /bin/false clamav
tar zxf clamav-0.91.2.tar.gz
cd clamav-0.91.2
./configure --prefix=/usr/local/clamav
make install
for binaries in `find /usr/local/clamav/bin/*` ; do ln -s ${binaries} /usr/bin/; done

At this point Clam AntiVirus is installed and ready for use. Edit the configuration file and remove the line which says: Example It is there to ensure. If you want, you can look at other options but we don’t need to change anything else here to make ClamAV work for us.

vi /usr/local/clamav/etc/freshclam.conf #remove Example

Now let us run the freshclam which will download virus database and bring our virus database up to date. We should do this manually and make sure it didn’t give any errors. If this works, you will a lot of “downloading” messages.

/usr/bin/freshclam

If everything checks out, let us add this to our crontab to ensure our virus database is updated hourly. I chose to be updated every 9 minutes in to every hour. You can change to fit your needs or leave it as it is.

crontab -e

9 * * * * /usr/bin/freshclam –quiet

At this point our ClamAV virus database is up to date and now we can scan whichever directory we want. Go to the directory you want to scan and type:

clamscan -r -i

Once it is done scanning, it will display something similar to below.
-r parameter tells clamscan to recurse into directories
-i will print out infected filenames

----------- SCAN SUMMARY -----------
Known viruses: 159855
Engine version: 0.91.2
Scanned directories: 1437
Scanned files: 8836
Infected files: 0
Data scanned: 464.83 MB
Time: 103.678 sec (1 m 43 s)

Happy scanning!
————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

MySQL: Slave lag behind master and data corruption.

12 Replies

I am reaching out to mysql experts out there! We just inherited a site which uses MySQL master/slave replication. We set up the master and slaves per MySQL documents and did some of our own tweaking in my.cnf to get things working. Things were great when we were testing the site but than after we put it up, we started to notice missing data on slaves. At first, we thought maybe we missed something with the import of data on slaves, so we re-imported. All tables in our db, at that time, were myisam. After a day or so, we started to see data corruption on some of the slaves. We went through and repaired all the tables. Few hours after the repair, we checked row counts and the repaired tables didn’t have same count as master did. Eventually most of our slaves were at the same state so we decided to re-import data from master again.

Before we went ahead and re-imported, we made a decision to move to innodb for *most* of our crucial tables. After altering on master, we did a dump/import. Converting tables to innodb got rid of our data corruption problem but our missing rows issue is still there. Hardware on the master is 4×1.86Ghz, 8 gigs of ram; slaves hardware is: 4×1.86, 4 gigs of ram.

Here is the config running on our master:
< some of the part of the config has been cut off to keep it short/simple and to the point >
skip-locking
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 1G
read_buffer_size = 2M
read_rnd_buffer_size = 8M
myisam_sort_buffer_size = 64M
thread_concurrency = 8
skip-name-resolve
max_allowed_packet = 32M
#low-priority-updates
table_cache = 1024
thread_cache = 8
query_cache_size = 64M
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 16
log_slow_queries
log_long_format
#
# INNODB Settings
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 4G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Here is the config running on our slave:

# The MySQL server
[mysqld]
port = 3306
socket = /tmp/mysql1.sock
pid-file = /usr/local/mysql/var/mysql.pid
relay-log = /usr/local/mysql/var/mysql-relay-bin
relay-log-index = /usr/local/mysql/var/mysql-relay-bin.index
log-error = /usr/local/mysql/var/mysql.err
master-info-file = /usr/local/mysql/var/mysql-master.info
relay-log-info-file = /usr/local/mysql/var/mysql-relay-log.info
datadir = /usr/local/mysql/var
server-id = 31
skip-slave-start
skip-locking
skip-name-resolve
ft_min_word_len=2
long_query_time=30
max_connect_errors=10000
max_connections=1000
#skip-grant-tables
skip-bdb
delay-key-write=ALL
key_buffer = 512M
max_allowed_packet = 1M
table_cache = 512M
myisam_sort_buffer_size = 64M
thread_cache_size = 4
read-only
#slave-skip-errors=all
#
# added by sunny >
wait_timeout = 120
tmp_table_size = 128M
read_buffer_size = 2M
sort_buffer_size = 128M # Sort buffer is used to perform sorts for some ORDER BY and GROUP BY queries.
read_rnd_buffer_size = 8M # Read the rows in sorted order by using the row
# pointers in the result file. To optimize this,
# we read in a big block of row pointers, sort them,
# and use them to read the rows in sorted order into a row buffer.
thread_cache_size = 8
query_cache_size = 16M # changed to 16 3/1/07
#
# Try number of CPU's*2 for thread_concurrency
thread_concurrency = 8
log_slow_queries
log_long_format
#
# INNODB Settings
default_table_type = INNODB
innodb_additional_mem_pool_size = 32M # was 16
innodb_buffer_pool_size = 2G
innodb_file_per_table
innodb_log_file_size = 50M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50
innodb_flush_method=O_DIRECT

Please feel free to comment/scrutinize any of these settings. If you think they should be different, please give quick summary about why they should be different. Some of these settings were inherited from old setup of mysql (where they had it hosted before).

Right now we are just trying to find out why we have missing rows and why our slaves lag behind constantly. We have refreshed data almost every day for last two weeks. We barely have load average above 1 or 2. What can we do to track down the problem? What are the things we should be looking out for?

MySQL: How do I reset MySQL to have only the databases when I first installed it? (CentOS, Redhat, Fedora, Linux)

7 Replies

One of my clients asked me today to make their MySQL installation go back to default database install. Basically they wanted me to get rid of all their databases (in this case test databases) so they can start fresh and go live with only the databases they needed. So here are the steps I followed.

Note: I have installed their MySQL from source and installed MySQL in to /usr/local/mysql folder. Your installation path might be different.

service mysql stop
cd /usr/local/mysql
mv var var.bak
mkdir var
bin/mysql_install_db
chown -R mysql.mysql var
chmod 700 -R var
service mysql start

At this point we should have fresh MySQL db setup.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.