Category Archives: Linux Tips

[Bash] Performing array intersection with Bash

I am currently working on a project to deploy new website builds to a
small number of servers. I needed something simple and reliable that could
be built in a very short period of time. I decided to whip something up in
bash with the intent of refining it in Python later.

As I began to write this code, I realized that it probably would have been
quicker to do it in Python from the start. I decided to stick with bash as
somewhat of an academic exercise. The vast majority of these deployment
scripts were trivial; check the code out of git, create a manifest, package
it up, spew it to the servers, etc, etc. The problem came during the last
step. We decided to use a symlink to point to the active build out of a
number of builds that could be available on the server at any given time.
Since all of our servers should be running the exact same version of the
build, it makes sense that I should only allow a user of my deployment
scripts to link a build which exists on all servers. But how do you
accomplish this in bash?

In most other languages, you have access to numerous array helping
functions that allow you to perform intersects, uniqs, and merges. My goal
was to do the same thing in bash without forking out to any external
binary. So how do you ensure that a particular thing exists on N number of
servers? Here it is:

function in_array() {
 local x
 ENTRY=$1
 shift 1
 ARRAY=( "$@" )
 [ -z "${ARRAY}" ] && return 1
 [ -z "${ENTRY}" ] && return 1
 for x in ${ARRAY[@]}; do
   [ "${x}" == "${ENTRY}" ] && return 0
 done
 return 1
}

MASTER=()
CURRENT=()
FIRST=1
for SERVER in ${SERVERS}; do
 # collect all builds from server and populate CURRENT list
 COMMAND="${LS} -1fd ${WEBROOT}/${SITE}.*"
 BUILDS=`${SSH} ${SSHOPTS} root@${SERVER} "${COMMAND}"`
 for BUILD in ${BUILDS}; do
   CURRENT=( ${CURRENT[@]-} ${BUILD} )
 done

 # if this is our first time around, copy CURRENT to MASTER
 if [ ${FIRST} -eq 1 ]; then
   MASTER=( ${CURRENT[@]} )
   FIRST=0
 fi

 # now we do a compare between MASTER and CURRENT to see what builds
 # are common
 INTERSECT=()
 for ENTRY in ${CURRENT[@]}; do
   in_array "${ENTRY}" "${MASTER[@]}"
   RET=$?
   if [ "${RET}" -eq 0 ]; then
     INTERSECT=( ${INTERSECT[@]-} ${ENTRY} )
   fi
 done
 MASTER=( ${INTERSECT[@]} )

 # clear the CURRENT array
 CURRENT=()
done

Let me take a moment to explain the code above:

  • In order to check for array intersection, you need an in_array()
    function

    • The first argument as the “needle” and the second is the
      “haystack”
    • We verify that both parameters were passed
    • We simply loop through the haystack checking for the needle
    • If we find it, return success. Otherwise, eventually return
      false
  • We need to loop through each server eventually, but we’ll start with
    the first one

    • Run an SSH command to get a listing of builds
    • Populate an array ($CURRENT) with the builds that were found
    • Since the first server has no previous server to compare with, so we
      just copy it to $MASTER
  • We then loop to the 2nd server, and put the result of getting builds
    into $CURRENT

    • Now that we have the first server’s builds in $MASTER, we perform an
      intersect with $CURRENT
    • We realize the need for an $INTERSECT array to hold the intersections
      found above
    • $INTERSECT becomes $MASTER since it only contains similar builds from
      the 1st and 2nd server
  • Looping to the 3rd server, we get the builds and put them in $CURRENT
    • Since $MASTER contains only the similar builds thus far, we again
      compare it with $CURRENT
    • The intersect can now be used to compare against builds on the 4th
      server, and so on
  • Once you finish looping through all servers, your $MASTER should
    contain only similar builds

There are a few guides out there which show you how to do this via
forking, but I thought someone may appreciate the elegance of using 100%
bash to accomplish this. I hope this helps someone else out there!

How do you print number of files for each folder in a directory [Linux]

I have been annoyed by the fact that I couldn’t easily print file count for all of the folders in certain directory.  Most of the time I just want to see what space each folder is using (du -hs *) but there are times when I need to know how many files are in each folder (checking cache folder, session folders etc).   So I whipped together a command line which does just that for me:

for i in `find -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

I am sure there are many different ways to show file count for each folder in a directory and I am curious to see what people do so please do post comments with what you do.

Above command is pretty simple and can be expanded to do whatever you need.  For example, you can throw it into a bash script and be able to pass parameters.  For example:  count_files /home/  In this case your command line would look like:

for i in `find /home/ -maxdepth 1 -type d`; do  echo -n $i " ";find $i|wc -l; done

only difference would be that /home/ would be argument you passed and therefore will be $1.  Here is a sample script for above example:

#!/bin/bash
for i in `find $1 -maxdepth 1 -type d`; do
echo -n $i " ";
find $i|wc -l;
done

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

Telnet: shell script to issue commands to telnet session.

This is a quick post to show how one can issue commands to telnet session from a shell script or command line with out going into interactive mode. I use this to get stats from our memcache servers or issue a flush_all via telnet from a script/cron.

So without further delay, following command will telnet to local memcached server on port 11211 and issue one of the memcached commands, stats

(sleep .5;echo stats) | telnet localhost 11211
You may have to play with the sleep timer to get it to work for your environment but in our .5 was the sweet spot. Good luck and let me know if you have another shell command. Obviously we can do this from perl, php, python, etc but the beauty of this is that you do not need any other dependencies plus its a very short command.

Since “jsled” commented about nc (thanks jsled), here is the syntax to do the same thing with nc:

echo "stats" | nc localhost 11211

SVN: How do you use svn command line on Windows with ssh tunneling?

If you ever used svn command line, you know it is not optimal to type in your password every time you do checkout, checkin, info, etc.  In linux world, it is very easy to setup keys to get around this.  Of course in the world of Windows it is not as easy.  Here are the steps you need to follow to get private/public keys working with your SVN under Windows using ssh tunneling.

Assumptions:  you will be connecting as user “root” to svn server located at “10.0.0.1”.  All your files will be saved at c:\ including your svn command line utility

First we will have to generate a key.  We can accomplish this by using a free utility called puttygen.  Run puttygen and click on “Generate” button.  You will have a key similar to below example:

Example of a key generated by puttygen

Example of a key generated by puttygen

Copy this, you will need it in few mins.  At this point, go ahead and create a private key by clicking on:  “Save private key”.  Save this as private.ppk on your C:\. 

Now let us log in to the svn server and add this public key to the authorized_keys2 (see setting up keys for step by step instructions).  I will assume you are using “root” as login.

vi /root/.ssh/authorized_keys2

Make sure when you paste, it is not broken into different lines.  All of the key should be one line.

Ok now back to your Windows machine.  Now we need to set up ssh tunnel to our server.  There are few ways of doing this but for our purpose, we will use another free program provided by the same developers as puttygen: download plink to C:\.  If you do not do this step, you will get following error:

svn: Can't create tunnel: The system cannot find the file specified.

Ok let us set the variables for svn.  Go to command prompt and type (you can also set this in your scripts and inside windows environment.  But since this post is to show you an example, we will just do this):

set SVN_SSH="/plink.exe" -i /private.ppk -l root

Now let’s run this one time manually to cache key:

/plink.exe -i /private.ppk -l root 10.0.0.1

Press “y” when it asks you to save.  Type exit and get back to your prompt.

Ok now we can test our svn utility.

/svn info svn+ssh://10.0.0.1/svn/testrepo/trunk/

This should display output similar to:

Path: trunk
URL: svn+ssh://10.0.0.1/svn/testrepo/trunk
Repository Root: svn+ssh://10.0.0.1/svn/testrepo
Repository UUID: b9143312-b1a1-11ba-a111-11cdcd1d2222
Revision: 10
Node Kind: directory
Last Changed Author: root
Last Changed Rev: 4
Last Changed Date: 2008-11-18 15:18:47 -0800 (Tue, 18 Nov 2008)

Now you are ready to script your checkouts, do checkin’s with out having to type in your password, etc.

————————————-
DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.

sshfs: How do you install sshfs and fuse? [CentOS/Linux/Redhat]

One may wonder what is sshfs and why would you want it?  Well simply put, sshfs allows you to mount another server’s filesystem into a folder on your local system which in the background is doing ssh commands and transfers.  As a mounted folder, you are able to move about and copy files back and forth as everything was on local server.  As you can see this makes it very easy for you to work with files on multiple servers.

Note:  you only have to do the following installations on the server where you are doing the mounts on.

Let us download and install the filesystem framework which is a requirement for sshfs called fuse.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/fuse-2.7.4.tar.gz
tar zxpfv fuse-*.gz
cd fuse*
./configure

If you get the following error, you will either have to point to the location of the kernel source or install it if needed.

checking kernel source directory... Not found
configure: error:
*** Please specify the location of the kernel source with
*** the '--with-kernel=SRCDIR' option
configure: error: ./configure failed for kernel

In our case here, we will be installing the source using yum.

yum -y install kernel-devel

Once installed, you will have to find out the directory it is installed in

ls -l /usr/src/kernels/
total 4.0K
drwxr-xr-x 18 root root 4.0K Oct  7 14:50 2.6.18-92.1.13.el5-x86_64/

./configure --with-kernel=/usr/src/kernels/2.6.18-92.1.13.el5-x86_64
make && make install
cd ..

Now let us get sshfs source and install it.

wget http://voxel.dl.sourceforge.net/sourceforge/fuse/sshfs-fuse-2.1.tar.gz
tar zxpfv sshfs*
cd sshfs-fuse-*
./configure

If you get the following error:

checking for SSHFS... configure: error: The pkg-config script could not be found or is too old.  Make sure it
is in your PATH or set the PKG_CONFIG environment variable to the full
path to pkg-config.

OR

checking for SSHFS... configure: error: Package requirements (fuse >= 2.2 glib-2.0 gthread-2.0) were not met:

No package ‘glib-2.0’ found
No package ‘gthread-2.0’ found

You need to install glib2.  Do the following:

yum install glib2-devel

Once installation is done, continue with configure.

./configure
make && make install

After installation is done, we can move on with testing the installation:

cd /mnt
mkdir test
sshfs 10.0.0.2:/ test

If you get the following error,
sshfs: error while loading shared libraries: libfuse.so.2: cannot open shared object file: No such file or directory
execute this: NOTE: this is for x64 system. If you have 32 bit system, you have to symlink to /lib instead.
ln -s /usr/local/lib/libfuse.so.2 /lib64/
Let us try mounting again:
sshfs 10.0.0.2:/ test
At this point it would be like if you were making a ssh connection to 10.0.0.2 You will have to type in a password to get the mount to happen. You may get the following error: fuse: device not found, try 'modprobe fuse' first

If you do ‘modprobe fuse’, as they tell you to, and you get:
modprobe fuse
FATAL: Module fuse not found.

That means your running kernel is not the same version as the one you compiled with. You have two options here:
1) you can upgrade your kernel by typing: yum update kernel
2) find the source files for the kernel you have running and recompile fuse.

I went with option 1. Once you do the update, reboot and try doing modprobe fuse again.

At this point we can try doing the mount again.
cd /mnt
sshfs 10.0.0.2:/ test

If you do not get any errors, do df -h to see the mount:
...
sshfs#10.0.0.2:/ 1000G 0 1000G 0% /mnt/test
...

At this point you can browse 10.0.0.2 server filesystem as it was local on your server.

————————————-

DISCLAIMER: Please be smart and use code found on internet carefully. Make backups often. And yeah.. last but not least.. I am not responsible for any damage caused by this posting. Use at your own risk.