Category Archives: CentOS

[CentOS] OpenVZ – container based virtualization quick guide. Part 1 of 2

1 Reply

This quick guide will walk you through setting up OpenVZ on CentOS. I followed these steps on CentOS release 5.5 x64 version. If you want more detail on install via different methods and/or have different flavor of CentOS and run into an issue, you can follow instructions provided by OpenVZ site. Once you are done with this guide, you will have CentOS container running for you to play with.

For my purposes, I started with very basic installation of CentOS 5.5. I have done this on existing installations of CentOS without any issues as well. Ok so let’s start with getting the OpenVZ repository added to our system. This means that we will be going down “yum” path instead of “rpm” path. This is the quickest and easiest way to get OpenVZ installed.

cd /etc/yum.repos.d
wget http://download.openvz.org/openvz.repo
rpm --import http://download.openvz.org/RPM-GPG-Key-OpenVZ

Before we move on, let’s turn off selinux:

vi /etc/selinux/config
:0,$s/=enforcing/=disabled/g
:wq

Go ahead and reboot the box and check:

[root@tooncent ~]# getenforce
Disabled

Now since we have the OpenVZ repository added and selinux disabled, we can go ahead and move on with installation part:
yum install vzkernel.x86_64 #install kernel with openvz support

echo '
# ----------added for OpenVZ installation----------------
# On Hardware Node we generally need
# packet forwarding enabled and proxy arp disabled
net.ipv4.ip_forward = 1
net.ipv6.conf.default.forwarding = 1
net.ipv6.conf.all.forwarding = 1
net.ipv4.conf.default.proxy_arp = 0
# Enables source route verification
net.ipv4.conf.all.rp_filter = 1
# Enables the magic-sysrq key
kernel.sysrq = 1
# We do not want all our interfaces to send redirects
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.all.send_redirects = 0
' >> /etc/sysctl.conf

Note: by executing above snippet, you will end up creating duplicate entries. It’s always a good idea to clean up duplicates to avoid confusion/mistakes later.

Following step is optional and mostly for clarity. Edit your grub.conf and change CentOS to say OpenVZ:

vi /etc/grub.conf

Change the first “title CentOS” to “title OpenVZ” (without the quotes). For example:

title CentOS (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

becomes:

title OpenVZ (2.6.18-194.17.1.el5.028stab070.7)
root (hd0,0)
kernel /vmlinuz-2.6.18-194.17.1.el5.028stab070.7 ro root=/dev/VolGroup00/LogVol00
initrd /initrd-2.6.18-194.17.1.el5.028stab070.7.img

Ok at this point we are ready to reboot and use the new kernel. Issue a “reboot”. Let’s install the utilities to manage OpenVZ:
yum install vzctl.x86_64 vzquota.x86_64

We need to start up vz manually once.

/etc/init.d/vz start

Alright! we got OpenVZ running. Now, let’s get a precreated template from OpenVZ site.

cd /vz/template/cache/
wget http://download.openvz.org/template/precreated/centos-5-x86_64.tar.gz

Ok now comes the fun part; let’s get our first container up and running:
Note: ostemplate name is the file you downloaded minus .tar.gz. centos-5-x86_64.tar.gz

[root@tooncent cache]# vzctl create 101 --ostemplate centos-5-x86_64
Creating container private area (centos-5-x86_64)
Performing postcreate actions
Container private area was created
[root@tooncent cache]#

Container files are created under /vz/private/101 < – 101 is the id you give it. You can put any numeric number above 100. Documentation says not to use anything below 101 since its a reserved range.

Let’s set a hostname so there is no confusion:

vzctl set 101 --hostname toontest.tooncent.com --save
Set hostname: toontest.tooncent.com
Saved parameters for CT 101

To start your new container:

[root@tooncent ~]# vzctl start 101
Starting container ...
Container is mounted
Setting CPU units: 1000
Set hostname: toontest.tooncent.com
Container start in progress...
[root@tooncent ~]#

To access your container:

vzctl enter 101

To exit the container, type: exit

To stop the container:

# vzctl stop 101
Stopping container ...
Container was stopped
Container is unmounted

Congratulations! You have successfully setup a container.

[Bash] Performing array intersection with Bash

4 Replies

I am currently working on a project to deploy new website builds to a
small number of servers. I needed something simple and reliable that could
be built in a very short period of time. I decided to whip something up in
bash with the intent of refining it in Python later.

As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be running the exact same version of the
build, it makes sense that I should only allow a user of my deployment
scripts to link a build which exists on all servers. But how do you
accomplish this in bash?

In most other languages, you have access to numerous array helping
functions that allow you to perform intersects, uniqs, and merges. My goal
was to do the same thing in bash without forking out to any external
binary. So how do you ensure that a particular thing exists on N number of
servers? Here it is:

function in_array() {
 local x
 ENTRY=$1
 shift 1
 ARRAY=( "$@" )
 [ -z "${ARRAY}" ] && return 1
 [ -z "${ENTRY}" ] && return 1
 for x in ${ARRAY[@]}; do
   [ "${x}" == "${ENTRY}" ] && return 0
 done
 return 1
}

MASTER=()
CURRENT=()
FIRST=1
for SERVER in ${SERVERS}; do
 # collect all builds from server and populate CURRENT list
 COMMAND="${LS} -1fd ${WEBROOT}/${SITE}.*"
 BUILDS=`${SSH} ${SSHOPTS} root@${SERVER} "${COMMAND}"`
 for BUILD in ${BUILDS}; do
   CURRENT=( ${CURRENT[@]-} ${BUILD} )
 done

 # if this is our first time around, copy CURRENT to MASTER
 if [ ${FIRST} -eq 1 ]; then
   MASTER=( ${CURRENT[@]} )
   FIRST=0
 fi

 # now we do a compare between MASTER and CURRENT to see what builds
 # are common
 INTERSECT=()
 for ENTRY in ${CURRENT[@]}; do
   in_array "${ENTRY}" "${MASTER[@]}"
   RET=$?
   if [ "${RET}" -eq 0 ]; then
     INTERSECT=( ${INTERSECT[@]-} ${ENTRY} )
   fi
 done
 MASTER=( ${INTERSECT[@]} )

 # clear the CURRENT array
 CURRENT=()
done

Let me take a moment to explain the code above:

  • In order to check for array intersection, you need an in_array()
    function

    • The first argument as the “needle” and the second is the
      “haystack”
    • We verify that both parameters were passed
    • We simply loop through the haystack checking for the needle
    • If we find it, return success. Otherwise, eventually return
      false
  • We need to loop through each server eventually, but we’ll start with
    the first one

    • Run an SSH command to get a listing of builds
    • Populate an array ($CURRENT) with the builds that were found
    • Since the first server has no previous server to compare with, so we
      just copy it to $MASTER
  • We then loop to the 2nd server, and put the result of getting builds
    into $CURRENT

    • Now that we have the first server’s builds in $MASTER, we perform an
      intersect with $CURRENT
    • We realize the need for an $INTERSECT array to hold the intersections
      found above
    • $INTERSECT becomes $MASTER since it only contains similar builds from
      the 1st and 2nd server
  • Looping to the 3rd server, we get the builds and put them in $CURRENT
    • Since $MASTER contains only the similar builds thus far, we again
      compare it with $CURRENT
    • The intersect can now be used to compare against builds on the 4th
      server, and so on
  • Once you finish looping through all servers, your $MASTER should
    contain only similar builds

There are a few guides out there which show you how to do this via
forking, but I thought someone may appreciate the elegance of using 100%
bash to accomplish this. I hope this helps someone else out there!

How do you print number of files for each folder in a directory [Linux]

14 Replies

I have been annoyed by the fact that I couldn’t easily print file count for all of the folders in certain directory.  Most of the time I just want to see what space each folder is using (du -hs *) but there are times when I need to know how many files are in each folder (checking cache folder, session folders etc).   So I whipped together a command line which does just that for me:

for i in `find -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

I am sure there are many different ways to show file count for each folder in a directory and I am curious to see what people do so please do post comments with what you do.

Above command is pretty simple and can be expanded to do whatever you need.  For example, you can throw it into a bash script and be able to pass parameters.  For example:  count_files /home/  In this case your command line would look like:

for i in `find /home/ -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

only difference would be that /home/ would be argument you passed and therefore will be $1.  Here is a sample script for above example:

#!/bin/bash
for i in `find $1 -maxdepth 1 -type d`; do
echo -n $i " ";
find $i|wc -l;
done

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

PHP: Curl being slow from php call on CentOS 5.4

1 Reply

I ran into an issue where curl request run within few ms from command line but same url fetch was taking over 10 secs. After doing some debugging and research it turned out that call to check (getaddr ipv6 was timing out. Obviously you can do multiple things to fix this (including enabling ipv6 support) but since I did not have a need for ipv6 (it is disabled in my network configuration), I decided to recompile curl and disable ipv6. Here is the command for recompiling curl and installing into /usr/local/curl directory with ipv6 disabled.

./configure –prefix=/usr/local/curl –disable-ipv6

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

sshfs: How do you install sshfs and fuse? [CentOS/Linux/Redhat]

25 Replies

One may wonder what is sshfs and why would you want it?  Well simply put, sshfs allows you to mount another server’s filesystem into a folder on your local system which in the background is doing ssh commands and transfers.  As a mounted folder, you are able to move about and copy files back and forth as everything was on local server.  As you can see this makes it very easy for you to work with files on multiple servers.

Note:  you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz
tar zxpfv fuse-*.gz
cd fuse*
./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found
configure: error:
*** Please specify the location of the kernel source with
*** the '--with-kernel=SRCDIR' option
configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/
total 4.0K
drwxr-xr-x 18 root root 4.0K Oct  7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64
make && make install
cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz
tar zxpfv sshfs*
cd sshfs-fuse-*
./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

OR

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0’ found
No package ‘gthread-2.0’ found

You need to install glib2.  Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure
make && make install

After installation is done, we can move on with testing the installation:

cd /mnt
mkdir test
sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse
FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt
sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
...
sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test
...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.