Epoch date/time convert - Python and Bash

With python you can convert epoch time to the format you want. Hint below!

>>> import os,time
>>> time.time()
1226392970.4683549
>>> time.gmtime(time.time()).tm_mday
11
>>> time.gmtime(time.time()).tm_mon
11
>>> time.gmtime(time.time()).tm_year
2008
>>>

WIth bash the same is accomplished as below.

veveo@shreyas:work$ date  +%s"  <-->  "%c
1226393170  <-->  Tuesday 11 November 2008 02:16:10 PM IST
veveo@shreyas:work$ date -d "-1 month" +%s"  <-->  "%c
1223714775  <-->  Saturday 11 October 2008 02:16:15 PM IST

veveo@shreyas:work$ aMonthBefore=`date -d "-1 month" +%s`
veveo@shreyas:work$ echo $aMonthBefore
1223714816
veveo@shreyas:work$ date --date=@$aMonthBefore +%c
Saturday 11 October 2008 02:16:56 PM IST
veveo@shreyas:work$ date --date=@1223714816 +%c
Saturday 11 October 2008 02:16:56 PM IST

Install ipw3945 (or iwlwifi) - Intel PRO/Wireless 3945ABG - on debian.

I always had these thoughts

1. Why ubuntu install all the necessary drivers for the intel wireless cards to work automagically after the installation
2. Why debian (etch) cant do the same ? why doesn't debian (etch) have the necessary packages in their official repositories.

I was always shying away from installing the firmware, building from the source. But when I found no other way, I ventured to get things done with sourceforge.net and here is how.

Step 1:

• download the ipw3945-1.2.2.tgz from sourceforge.net (http://ipw3945.sourceforge.net/)

Step 2:

• Do the following : 'tar -xzf ipw3945-1.2.2.tgz',  'cd ipw3945-1.2.2'  and 'less INSTALL'

Remember, most important step is the "less INSTALL' and reading through the instructions.

Finishing it off:

Thank your lucky stars, thank the developers who got these packages for you and last but not least thank sourceforge.net for hosting the ipw3945 page.

PS: If you have followed the 'INSTALL' file precisely you should see the wireless networks in your 'nm-applet' list, when you click on it.

I had to use sudo every here and there, especially when i do the './load' within the 'ipw3945-1.2.2' directory. So use it at abundance, shouldn't hurt you much !

This will only get you started with perfecting the wireless interface configuration. Once the mentioned procedure is completed, you might want the wpasupplicant and other accompanying packages like wpa_gui also for completing the setup.

Update:

After updating my Debian etch DVD's to lenny using jigdo, I installed lenny on the laptop. Once done, I modified the sources.list as below and did a "sudo apt-get dist-upgrade"

deb http://security.debian.org/ lenny/updates main contrib non-free
deb-src http://security.debian.org/ lenny/updates main contrib non-free
deb http://http.us.debian.org/debian/ lenny main contrib non-free
deb-src http://http.us.debian.org/debian/ lenny main contrib non-free

This updated my kernel to 2.6.24 and it got me the correct wireless drivers (iwlwifi, this is the next life of deprecated ipw3945 driver). Wireless was working fine after the update and the networks were showing up in the nm-applet.

Configuring syslog server on debian

Like me, if you also wanted to set up a log server for gathering all the logs from your firewall, router, servers etcetera to a single log server, follow me..

I am running a Debian (lenny; In etch also this should work fine) and I got syslog-ng installed replacing the older, sysklogd package. syslog-ng is the next generation log daemon which has added features, noticeably in the log filtering area. But all I wanted was to get start with collecting and logging logs from my Netscreen firewall. So this is what I did.

Installed necessary packages using apt

apt-get install syslog-ng syslog-summary

(I am yet to try out syslog-summary )

Added the following to the configuration file of syslog-ng (/etc/syslog-ng/syslog-ng.conf)

In the source s_all{}, uncommented "udp();"

under the default destinations, added "destination df_netscreen { file("/var/log/netscreen.log"); };"

Under the default filters, added "filter f_netscreen { facility(LOCAL0); };"

Added to the end of the configuration file, the meat of the new log entry,

log { source(s_all); filter(f_netscreen); destination(df_netscreen); };

And then restarted the syslog-ng service.

sudo /etc/init.d/syslog-ng restart

That much did the main work of setting up the log server.  

"f_netscreen" & "df_netscreen" These are mere identifiers and since I am setting this up to get my Juniper Netscreen25 logs to be accumulated here, I gave the name "*netscreen*". "LOCAL0" This is the log facility I used on the Juniper Netscreen25 for setting it up to send the logs to the newly set up log server. Hence the string "LOCAL0" appeared in the facility description in the filter syntax.

Since my firewall log size grows madly, I had to set up the logrotate to save my computer from running out of space. Adding a file called "netscreen" in the "/etc/logrotate.d" folder with contents as below did the magic for me.

/var/log/netscreen.log {
  rotate 6
  weekly
  compress
  missingok
  notifempty
}

This will rotate my firewall logs weekly and each will be kept until 6 weekly rotations before getting deleted. Read through "man logrotate" for more detailed information on setting up logrotate for your logfile and its always good to restart the crond "sudo /etc/init.d/cron restart" since logrotate uses cron.daily to get itself running every day.

update (2008-04-11): Adding the remote log facility "LOCAL0" to the "not" list of "f_messages" filter will avoid accumulating the remote logs in "/var/log/messages". (Addition is in 'bold letters', below)

filter f_messages {
        level(info,notice,warn)
            and not facility(LOCAL0,auth,authpriv,cron,daemon,mail,news);
};

Most importantly, by default syslog-ng does not seem to enable the cron log. I found it only when I wanted to check why the logrotate did not worked. I had to un-comment the below in "/etc/syslog-ng/syslog-ng.conf".

#log { source(s_all); filter(f_cron); destination(df_cron); };

Installing KDE4 on Ubuntu (Hardy Heron)

I had an urge to try kde4 on my Ubuntu installation yesterday. I was using gnome in the laptop and I was a loyal KDE 3.5.x user on my office desktop (with Debian etch upgraded to Lenny a couple of days before) for a long time. With all the new applications details going around about kde4, I wanted to give it a try. After upgrading the Debian etch at my office desk to Lenny, I was a bit over confident that upgradinng my Ubuntu Gutsy to Hardy wouldn't break my heart (applications). I changed the sources.list to do a dist-upgrade. But since my home connection is not that great considering the speed, I decided to update to kde4 to start with, because something was telling me that things wont go so fine with kde4. I ended up with a shiny new kde4 interface in about 40 minutes. I logged into the kde4 and the first impression was not so great. I was welcomed with a windows vista look-alike interface. The new application launcher (K menu) was there with a new design to complete the vista's clone. I didn't like the way KDE4 gave me the desktop. The icons were oversized, menu was difficult than the old style (The classic menu is still there, which you might want to add to the panel). I like it when things have changed for good, but when it makes things difficult for users to adjust, its not improvement. Its disastrous design for the sake of improved flashy interface. The fact that I most dislike is the way kde4 resembles my windows vista. I was with the desktop for about 45 minutes and still have not figured out how to move a application launcher in the panel. Call me ignorant or noob, but I think simple things like these should not be so hidden (It was not in kde3.x). The fact that default gnome optimized window manager gives me more screen real estate made me switch back to gnome in less than 1 hour. Oh yeah and I did tried out the new improved konsole and the file manager. File manager was good (I can always install a similar polished file manager in my old, and the best, KDE - like krusader). Konsole doesn't "look" good and the new settings tab which is said to be improved and easier to use than previous, looked not a bit more easy than the former version. Overall, I did not like KDE4, and I am not going to use it again, not in the near future. What a great desktop kde3.x was, and how the new kde4 put me off from even trying it out. Maybe its just the fact that kde4 looks more like windows vista desktop, that had made me feel uneasy with kde4. I do not understand why there are people who believe, things will be more user friendly when it looks like windows desktop. Well, fine with all the thoughts, all I wanted was to let you know this is what you can expect as a welcome with the new KDE desktop. Try it and ditch it

ffmpeg with AMR (nb & wb), lame, FAAC and FAAd codec.

Below is how I have installed and used ffmpeg to convert avi and wmv files to 3gp format so that I can store and play them in my mobile phone. I have a very rough guide, but hope that will help you in understanding more about the procedure by doing your own research. Consider the below scribble as a pointer to get things started. Now that being said, lets get started..

Download the source for amr_nb (nb stands for narrow band; amr is for the 3gp support). If you can find the sources for both you can use both of them. I was not able to find the source for amr_wb and at last I had to beg in irc://irc.freenode.net (#ffmpeg) to get the source from a fellow who had it uploaded in some file sharing site for me.

For mp3 support, get the lame codec from sourceforge

Download faac, faad2 from sourceforge.net and get them installed.

get the amr and lame codecs compiled and installed.

Get the source for ffmpeg from the svn or search for the tarball.

svn://svn.mplayerhq.hu/ffmpeg/trunk

Get things going with the configure, make & make install.

./configure --enable-libmp3lame  --enable-libvorbis --enable-libamr-nb --enable-libamr-wb --enable-gpl --enable-libfaac --enable-libfaad --enable-shared

voila.. you are ready to convert video files for your mobile ~smile~

Below is the way I use ffmpeg to convert. Figuring out the options for ffmpeg is your job (hint: man ffmpeg)

./ffmpeg -i /home/old_desktops/sabarish.old/laptop_backup/Azureus\ Downloads/Kylie\ Minogue\ -\ Spinning\ Around\ \(xvid\ K-rip\).avi -f 3gp -vcodec mpeg4 -acodec faac -ar 8000 -ac 1 -ab 12.2k -y Kylie.3gp

I have used options, -acodec libamr_wb also but had some issues with the sound in the video, so rolled back using faac audio codecs. You can experiment around and use the best to get things done.

Convert -File System- ext2 to ext3

Lets get started.

If you can un-mount the partition which needs the change, perfect. Else you can remount it in
read-only mode

mount -o remount,ro /dev/xxx

If you want to un-mount the partition and it complains that the device/resource is busy, you can use a little brute force to get things done if you are sure nobody else is using it.

lsof | grep "<mount point>" | awk '{print $2}' | xargs kill -9

The above command may disturb other users logged in and  if they are using the machine. You should be sure that the machine is not shared with anyone else, hence make sure before you do. If the above trick does not work, you know what it does I hope, hence you can try do it yourself some other way. Once done, you can un-mount the partition. Next step is

tune2fs -j /dev/xxx

This will create ext3 file system in the partition specified. Once this is over "without any errors" you should change the "file system type" in /etc/fstab and make it ext3. Then do a

mount -a

To check whether its mounted properly and the file system type, do

mount

Re-read partition table without rebooting OS

I had this situation at office where I was told to re-partition an already existing partition. The situation was to get the below schema

/dev/sdb1 1 3040 24418768+ 83 Linux
/dev/sdb2 3041 6080 24418800 83 Linux
/dev/sdb3 6081 30515 196274137+ 83 Linux

to

/dev/sdb1 1 3040 24418768+ 83 Linux
/dev/sdb2 3041 6080 24418800 83 Linux
/dev/sdb3 6081 30515 196274137+ 5 Extended
/dev/sdb5 6081 18239 97667136 83 Linux
/dev/sdb6 18240 30515 98606938+ 83 Linux

Alright, now the partition /dev/sdb2 was in use and cannot be unmounted. So what happens when you finish the partitioning with fdisk ?

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

This happends, if you are using/mounted any partitions of the HDD which you just re-partitioned. Normally we do reboot the machine to get this right. But this is not always the easy way out, when you have something running on the machine/multiple users are on the machine working etc. So there comes the "parted" for rescue.

The package "parted" provides a tool called "partprobe". This will do the magic for you, you just type in the command (see the man page for more info on partprobe) and this will give the information needed about your newly created partition to your kernel.

The "http_proxy" environment variable - Setup proxy setting in a linux machine.

If you need to use proxy server to access http/https from a linux machine in the office LAN, set the environment variable http_proxy. This will allow wget and python's urllib modules and other applications (yum, apt-get etc) to use this environment variable and access http/https using the settings assigned to the variable http_proxy.

The below would be the ideal way of assigning values for http_proxy variable.

$export http_proxy="http://<proxy-server-ip>:<port>"

I have added this to my ~/.bashrc so that I don't have to export this variable every time I reboot my machine !

And then there is the "ftp_proxy" .....

Linux PPTP VPN client configuration

It is faily easy to install the packages needed and hand configure the pptp client for a linux server. The pptp server is a windows 2003 server running RRAS. How ever from my experience what was more difficult is configuring the firewall to let the pptp traffic go through. Here is what I did to make a connection from Linux PPTP client to the RRAS.

• Installed pptp-linux
  
  
  
  • Optionally you can install pptpconfig which will make life easier with the client configuration part. For that You have to add the below source to your sources.list "deb http://quozl.linux.org.au/pptp/pptpconfig ./"
  
  
  
  


• Collected the server details needed
  
  
  
  • PPTP Server Name - vpn.yourdomain.com
  
  
  • VPN User Name - user1
  
  
  • VPN Password - passw0rd
  
  
  • Connection Name - vpn1
  
  
  
  


• Edited the file /etc/ppp/chap-secrets to add the line "user1 PPTP passw0rd *"


• created a file /etc/ppp/peers/vpn1. Entered the configuration details as below.

pty "pptp vpn.yourdomain.com  - -nolaunchpppd"

name user1

remotename PPTP

require-mppe-128

file /etc/ppp/options.pptp

ipparam vpn1

With this the configuration finished. You can start the tunnel by issueing

• 'sudo pppd call vpn1' or 'pon vpn1'.


• for debug info try 'sudo pon veveo debug dump logfd 2 nodetach'

To stop the pptp tunnel, do a "sudo pkill pppd". You will be able to see the connection status/details in "/var/log/messages"

For me this much was easy. This didnt worked for me straight away. Fair enough, then after a long search I could find that pptp to work through my Juniper Netscreen (ns25) firewall, opening tcp port 1723 is not enough, I should allow the "GRE Protocol" (protocol 47) pass through. I added a custom service for GRE and then everything was working fine.

I am planning to implement a linux SSL vpn server later when I get time. But anyway only after learning the advantages and disadvantages of the SSL VPN.

when you wanted to set up a quick cvs server

follow the below link. I got a cvs server up in 5 minutes. Cant get faster than this can you ?

http://home.wanadoo.nl/cwdegier/cvs.html

Python, Bash yesterday's date

I wanted to get yesterday's date in a neat way for my backup scripts (bash). I found out a simple solution for the same pretty easily.

date -d "-1 day" +%d%m will give me the yesterdays date.

date -d "-1 month" +%d%m will give me date one month previous.

Here is what you do in python for the same.

>>> import datetime
>>> today = datetime.date.today()
>>> today
datetime.date(2008, 3, 20)
>>> yesterday = today - datetime.timedelta(1)
>>> yesterday
datetime.date(2008, 3, 19)
>>> monthbefore = today - datetime.timedelta(31)
>>> monthbefore
datetime.date(2008, 2, 18)
>>>

How much vim installation irritated me!

It happened such that none of the developers here in my office are happy with vi. They all wanted vim and that the latest version also. Now imagine the experience of installing in 30+ servers which runs FC4, with hellua dependancies to satisfy the vim rpm. I choose to do it with a source tarball. Wasnt easy as I thought. The first problem I faced was with the X11 libraries. The vim scripts were looking in /usr/X11R6/lib for the needed libraries in a 64 bit machine, and it complained the libraries are not compatible. Well, as I was in a hurry to finish the job (as always as it happens with me), I did a small rearrangement. Moved the lib in /usr/X11R6 to another name and created a symlink to lib64 with the name lib. Vim was still needy and I had to create symlinks for libXaw.so, libX11.so and libXt.so inside the lib64 folder.

Then the configure and make went fine. I was soo happy that its going on fine. Then after the make i was going to install and issued make install. It gave me a messaage "make: `install` is up to date". I was wondering why it would be giving such an error. I was going through the contents of the Makefile and found a line saying "If there are problems, cd to the src directory and run make there". What the hell, I thought, went inside the src folder and did a make install, it just went fine. The make install and the place where vim scripts search for the library files were funny, but apparently my job was to get vim running in all the machines. Yup, I did it.