Comments on: MySQL: Slave lag behind master and data corruption. http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/ Enterprise level solutions, LAMP, Linux, Apache, MySQL, PHP, Perl, Windows, Cache, Optimization Mon, 16 Apr 2012 22:54:49 +0000 http://wordpress.org/?v=2.9.1 hourly 1 By: Paul M http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-327 Paul M Mon, 22 Oct 2007 04:14:47 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-327 Noticed Baron (Xaprb) is too modest, use the MySQL toolkit and do checksums on the slave and you can also use the sync function to sync missing rows. You are going to have to use mysqlbinlog to mine the binlogs to determine what SQL was not replicated. If you combine table checksum from the toolkit, you should be able to search for the SQL (on the master) which should have updated the slave. Other stuff. Review the use of these settings on master/slave. innodb_flush_log_at_trx_commit = 0 innodb_lock_wait_timeout = 50 From the MySQL 5.0 doco http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html Note: For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1, sync_binlog=1 Note: Both these are going to slow down the transaction per sec performance of the master, so benchmark first and change to see the difference. Have Fun Paul http://blog.dbadojo.com/2007/09/mysqltoolkit-mysql-table-checksum.html Noticed Baron (Xaprb) is too modest, use the MySQL toolkit and do checksums on the slave and you can also use the sync function to sync missing rows.

You are going to have to use mysqlbinlog to mine the binlogs to determine what SQL was not replicated.
If you combine table checksum from the toolkit, you should be able to search for the SQL (on the master) which should have updated the slave.

Other stuff. Review the use of these settings on master/slave.
innodb_flush_log_at_trx_commit = 0
innodb_lock_wait_timeout = 50

From the MySQL 5.0 doco
http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html

Note: For the greatest possible durability and consistency in a replication setup using InnoDB with transactions, you should use innodb_flush_log_at_trx_commit=1, sync_binlog=1

Note: Both these are going to slow down the transaction per sec performance of the master, so benchmark first and change to see the difference.

Have Fun
Paul
http://blog.dbadojo.com/2007/09/mysqltoolkit-mysql-table-checksum.html

]]> By: CrazyToon http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-323 CrazyToon Wed, 17 Oct 2007 17:11:39 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-323 Mysql version 5.0.45 source install. We fixed our sort_buffer_size. I didn't know that if we put 128M, it would take 128M for each query. I, for some reason, thought it was the ceiling and queries will use whatever it needs until it hit 128M. Apparently I was wrong. Now we set sort_buffer_size=4M I will try out the packet size change. Also will check for the errors in the log for the errors Dmitri mentioned. Thanks for all of your suggestions/comments. Even if your suggestions might not fix the problem, I am sure we are all learning what to do or not to do. Keep them coming. Mysql version 5.0.45 source install.
We fixed our sort_buffer_size. I didn’t know that if we put 128M, it would take 128M for each query. I, for some reason, thought it was the ceiling and queries will use whatever it needs until it hit 128M. Apparently I was wrong. Now we set sort_buffer_size=4M

I will try out the packet size change. Also will check for the errors in the log for the errors Dmitri mentioned.

Thanks for all of your suggestions/comments. Even if your suggestions might not fix the problem, I am sure we are all learning what to do or not to do. Keep them coming.

]]> By: Steve http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-321 Steve Tue, 16 Oct 2007 15:30:40 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-321 I don't recommend setting up the slave via a dump/import. It's better to do it by copying the data files. I don’t recommend setting up the slave via a dump/import. It’s better to do it by copying the data files.

]]> By: Dmitri Mikhailov http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-320 Dmitri Mikhailov Tue, 16 Oct 2007 14:58:06 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-320 Make sure read_buffer_size is less or equal to max_allowed_packet, there were few bugs related to these variables. "Too many open files" or "Can't open file" errors? Tune up open_files_limit variable. - Mixed myisam/innodb config? "Out of memory" errors in the log? Slave crashes with no traces in the logs? Upgrade the memory on the slave or tune up the server variables. - Repair table corrupts data/index file? Make sure myisam_repair_threads is set to 1, just to be safe, there were a few bugs regarding "REPAIR TABLE" corrupting the myisam data/index files. - thread_cache_size is defined twice in the slave config file. This just does not make sense. Make sure read_buffer_size is less or equal to max_allowed_packet, there were few bugs related to these variables. “Too many open files” or “Can’t open file” errors? Tune up open_files_limit variable.

– Mixed myisam/innodb config? “Out of memory” errors in the log? Slave crashes with no traces in the logs? Upgrade the memory on the slave or tune up the server variables.

– Repair table corrupts data/index file? Make sure myisam_repair_threads is set to 1, just to be safe, there were a few bugs regarding “REPAIR TABLE” corrupting the myisam data/index files.

– thread_cache_size is defined twice in the slave config file. This just does not make sense.

]]> By: Dmitri Mikhailov http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-319 Dmitri Mikhailov Tue, 16 Oct 2007 14:53:55 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-319 Few things are missing: - MySQL version; - Error logs; - Slow query log; - Repair table output. Without this information I can only guess... - Replication errors? Set max_allowed_packet to the same size on master and slave (in your case it's set to 32M on master and 1M on slave). Make sure read_buffer_size Few things are missing:

– MySQL version;
– Error logs;
– Slow query log;
– Repair table output.

Without this information I can only guess…

– Replication errors? Set max_allowed_packet to the same size on master and slave (in your case it’s set to 32M on master and 1M on slave). Make sure read_buffer_size

]]> By: GOTG http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-318 GOTG Tue, 16 Oct 2007 14:48:40 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-318 To check binary log use mysqlbinlog. It's unlikely that you have queries bigger than 1M (slave packet size) but I see master having 32M packet size. If large queries are missing from slaves then you may need to increase the packet size on slaves. To check binary log use mysqlbinlog. It’s unlikely that you have queries bigger than 1M (slave packet size) but I see master having 32M packet size. If large queries are missing from slaves then you may need to increase the packet size on slaves.

]]> By: dan http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-317 dan Tue, 16 Oct 2007 13:18:00 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-317 what is the mysql version? we had similar problem when we were on 4.0 and it seemed to be fixed when we moved to 4.1.xx what is the mysql version? we had similar problem when we were on 4.0 and it seemed to be fixed when we moved to 4.1.xx

]]> By: CrazyToon http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-316 CrazyToon Tue, 16 Oct 2007 04:11:11 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-316 GOTG: we initially thought it was the myisam tables so we moved to innodb. Our total increase in disk usage is about 1.5x. We also checked the server IDs and they are all unique. Ryan: how does one go about looking into binlogs? All good ideas. Keep them coming. GOTG: we initially thought it was the myisam tables so we moved to innodb. Our total increase in disk usage is about 1.5x. We also checked the server IDs and they are all unique.
Ryan: how does one go about looking into binlogs?

All good ideas. Keep them coming.

]]> By: ryan http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-314 ryan Tue, 16 Oct 2007 02:51:37 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-314 One thing to check is if the missing rows are being written to the binlog at all. I've run into a few cases where certain statements are applied to the master but not written to the binlog so of course are not relayed to the slaves. And if they are, it would be worth checking to watch the binlog to try to discern a pattern from the missing data. Is it a unusually large statement? Does it use a particular syntax that is unlike your other code? Is it at a regular time interval? These will give important clues to figuring out the root of the problem. One thing to check is if the missing rows are being written to the binlog at all. I’ve run into a few cases where certain statements are applied to the master but not written to the binlog so of course are not relayed to the slaves.

And if they are, it would be worth checking to watch the binlog to try to discern a pattern from the missing data. Is it a unusually large statement? Does it use a particular syntax that is unlike your other code? Is it at a regular time interval? These will give important clues to figuring out the root of the problem.

]]> By: GOTG http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-313 GOTG Tue, 16 Oct 2007 02:30:40 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-313 BTW, missing data on slaves sound like non-unique server IDs. BTW, missing data on slaves sound like non-unique server IDs.

]]> By: GOTG http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-312 GOTG Tue, 16 Oct 2007 02:07:49 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-312 Slaves lag if you run heavy queries on myisam tables. Myisam tables lock the whole table at insert/update so you need to use something else for heavy insert/update tables (innodb would be the logical choice). Warning: innodb tables are about 6 times the size of myisam tables. Slaves lag if you run heavy queries on myisam tables. Myisam tables lock the whole table at insert/update so you need to use something else for heavy insert/update tables (innodb would be the logical choice). Warning: innodb tables are about 6 times the size of myisam tables.

]]> By: Xaprb http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/comment-page-1/#comment-311 Xaprb Tue, 16 Oct 2007 01:27:56 +0000 http://crazytoon.com/2007/10/15/mysql-slave-lag-behind-master-and-data-corruption/#comment-311 Your sort_buffer_size is too large; it is allocated for every query that does a filesort. You should keep it smaller and only increase it just before each query that does a filesort: set @@session.sort_buffer_size = (bigger value); (query) set @@session.sort_buffer_size = default; You should probably do a little investigative work to see what you really need to tune. Any chance you can benchmark before tuning? It might save you a lot of work. Otherwise try mysqlreport or something for some basic direction on it. As far as the slaves being behind and missing data, I don't see anything suspicious (I was initially thinking you'd have some replication filters not doing what you thought). Any errors in the master or slave error logs? You might consider posing your question on the mysql general mailing list. Blog comments are hard to use for a conversation about something like this. Your sort_buffer_size is too large; it is allocated for every query that does a filesort. You should keep it smaller and only increase it just before each query that does a filesort:

set @@session.sort_buffer_size = (bigger value);
(query)
set @@session.sort_buffer_size = default;

You should probably do a little investigative work to see what you really need to tune. Any chance you can benchmark before tuning? It might save you a lot of work. Otherwise try mysqlreport or something for some basic direction on it.

As far as the slaves being behind and missing data, I don’t see anything suspicious (I was initially thinking you’d have some replication filters not doing what you thought). Any errors in the master or slave error logs?

You might consider posing your question on the mysql general mailing list. Blog comments are hard to use for a conversation about something like this.

]]>