Notes

This is not a blog. It is just my way of avoiding pen and paper and storing technology related notes. Some (or most) of the topics might be too in-depth for some people, but then this is meant for a targeted audience.

Here you will find posts on topics related to Computer Science, Scripting, Automation, Unix, SSH or just general geek stuff that might be interesting to techies. Learning twice is the worst crime that we might be guilty of and hopefully this page can save me from it while also being useful to others who find it interesting.


Mac OS X/BSD Scripts and Tools

posted Jul 23, 2011 6:31 AM by Amol Kolhe   [ updated Jul 23, 2011 6:50 AM ]


Memory Usage % on Mac

Usage: ./memory.sh

#!/bin/sh

case $1 in
"") ps -acev | awk '{print $12}' | grep [0-9] | awk 'BEGIN {sum=0}{sum+=$1} END {print "MEM % " sum}' ;;
[0-9])
LINES=$1
echo "%MEM\tProcess"; ps -acev | awk '{print $12 "\t\t" $13}' | sort -nr | head -$LINES
esac


Battery Remaining on a Mac

Usage: ./battery.sh

#!/bin/bash

maxcap=`ioreg -l | grep Capacity |awk '/MaxCapacity/{print $5}'`
curcap=`ioreg -l | grep Capacity |awk '/CurrentCapacity/{print $5}'`

curcap=`echo $curcap*1000|bc`
proc=`echo $curcap / $maxcap |bc`;

len=`echo ${#proc}`;

proc_num=${proc:0:3}
proc_dec=${proc:3:4}

proc_num_len=`echo ${#proc_num}`;

if [ $proc_num_len == "3" ]; then
if [ $proc_num != "100" ]; then
proc_num=${proc:0:2}
proc_dec=${proc:2:3}
fi
fi

echo $proc_num.$proc_dec%


CPU Usage % on a Mac

Usage: ./cpu.sh

#!/bin/sh
case $1 in
"") ps -acev | awk '{print $11}' | grep [0-9] | awk 'BEGIN {sum=0}{sum+=$1} END {print "CPU % " sum}' ;;
[0-9])
LINES=$1
echo "%CPU\t\tProcess"; ps -acev | awk '{print $11 "\t\t" $13}' | sort -nr | head -$LINES
esac

WorldClock script

Usage: ./worldclock.sh

#!/bin/sh
echo "New York | `date +%a\ %m/%d\ \|\ %H:%M`"
echo "India | `date -v+5H -v+30M -u +%a\ %m/%d\ \|\ %H:%M`"


Periodic System Functions

The preriodic command executes To get more information, check manpages for periodic

To execute all periodic scripts
sudo periodic daily weekly monthly

Log files:
/var/log/daily.out
/var/log/monthly.out
/var/log/weekly.out

Following files contain the schedule information for each periodic task:
/System/Library/LaunchDaemons/com.apple.periodic-daily.plist
/System/Library/LaunchDaemons/com.apple.periodic-monthly.plist
/System/Library/LaunchDaemons/com.apple.periodic-weekly.plist

Main Configuration File: /etc/defaults/periodic.conf

Useful Linux Tricks on Macbook

posted Jul 19, 2011 8:33 AM by Amol Kolhe   [ updated Jul 20, 2011 6:59 PM ]

Parts of this section are specific to Ubuntu on a macbook. I have also used some of these tricks on a EEE netbook, but they might not work for others. Many of them are specific to the macbook hardware (mine is macbook 4,1) and the software used with specific versions of Ubuntu.

I do not like Ubuntu's GUI interface, it is too tightly coupled with Gnome and (is in my opinion quite sluggish and packs lot of junk which could be avoided/disabled), but Ubuntu remains my Linux Distro of choice, because it is well tested on a wide array of hardware and mostly works out of the box. In fact, on 11.04 (64 bit), every piece of hardware on my macbook worked beautifully including Wifi using WPA2. Similar experience with 10.10 on a EEE Netbook. If I start with a more minimal distro like Ubuntu Server or Debian or FreeBSD, it is a lot of work setting up each piece of hardware and software. So I like to start with a fresh install of Ubuntu and reverse-engineer it to my liking.


ng 

Disable X-windows from Startup

On Ubuntu, I've found it quite difficult to disable X-windows from startup. The simplest and guaranteed way of accomplishing this is by removing GDM. I dislike gdm the same reasons I dislike Gnome. I usually use linux as a lean-mean alternative to boot into and surf the net or boot into command line and run scripts. I don't need a full blown desktop in linux. Gnome and Gdm are overkill for that, and I feel they are sluggish (especially gnome). 

Following command will have horrible consequences on your Gnome setup so if you're unsure, then don't execute it.
sudo apt-get -y purge gdm

Installing Fluxbox

sudo apt-get -y install fluxbox

Using startx to start GUI

The startx script checks the user's home directory for the file .xinitrc, to determine what window manager to use. On a new install, this file is missing, and by default, the system wide default window manager will be started. And if you removed gdm, this might not go very well. To override this behavior and start Fluxbox for instance, following step is needed.

echo startfluxbox > ~/.xinitrc

Now to start X-windows, just run
startx


Allow non-root user to shutdown/restart

Add a group for shutdown/halt/reboot and add your user(s) to it. Allow this group to execute sudo reboot/halt/shutdown without typing password and then alias the restart commands with alternate versions with sudo built in.

sudo groupadd shutdown

sudo vi /etc/group                #Add your user to the grouping
shutdown:x:1001:amol

sudo vi /etc/sudoers            #Add the following lines
shutdown ALL=(ALL) NOPASSWD:/sbin/shutdown
shutdown ALL=(ALL) NOPASSWD:/sbin/halt
shutdown ALL=(ALL) NOPASSWD:/sbin/reboot

vi ~/.profile
alias reboot="sudo /sbin/reboot"
alias halt="sudo /sbin/halt"
alias shutdown="sudo /sbin/shutdown"

Once all of the steps are done, restart shell, and you should be able to restart your system without ever typing a password.

Changing Screen Brightness from Command line

On a macbook, this can be done by editing the file /sys/class/backlight/mbp_backlight/brightness and putting in a value between 1 and 15, 1 being the least bright and 15 being most bright. If you're trying this on any other hardware, you might need to find the correct path for yous.

I  wrote a script to quickly and easily do so from command line. You can grab it from the attachment section below. I use the same script from X-windows by assigning keyboard shortcuts (to match my laptop's brightness keys). 

Usage: sudo /sbin/brightness [+ - or a value between 1 and 15]
Example:
sudo /sbin/brightness +
sudo /sbin/brightness -
sudo /sbin/brightness 9

Getting Macbook's Brightness Keys to work in Fluxbox

Macbooks has F1 key mapped to decrease Brightness and F2 key mapped to increase Brightness (without pressing the Fn key). F1 and F2 (without Fn) corresponds to Key codes 232 and 233 respectively. You can find this by using the command xev and pressing each key.

Copy the brightness script mentioned above in /sbin or use your own script.

vi ~/.fluxbox/keys    # Add the following lines
232 :Exec sudo /sbin/brightness -
233 :Exec sudo /sbin/brightness +

There is one more thing before above keys will work. Sudo command promps for password, so the shortcut won't really work. To bypass this do the following
sudo groupadd brightness
sudo vi /etc/group            #Add your user to group brightness
brightness:x:1002:amol

sudo vi /etc/sudoers        #Add the following line
brightness ALL=(ALL) NOPASSWD:/sbin/brightness

Restart Fluxbox.

Samba/CIFS/Windows Share

Install samba/CIFS compatibility package
sudo apt-get -y install smbfs

Check if the mount works:
sudo mkdir -p /mnt/samba
sudo chown amol:amol /mnt/samba
sudo mount -t smbfs //MyBookWorld/public /mnt/samba

Where MyBookWorld is the Hostname/IP Address of the Sharing Server and public is the shared path.
Once above works, you should be able to browse, the remote file system from Terminal or File Browser. To auto-mount on system start-up add it to fstab
sudo vi /etc/fstab # Add the following line in the end
//MyBookWorld/public /mnt/samba smbfs username=guest,uid=amol,gid=amol 0 0


FTP script

posted Jul 7, 2011 2:39 PM by Amol Kolhe

#This is a lame ftp script. I don't use it anymore since I use Scp or Sftp

Usage: ftp -i -s:ftp.txt

cat ftp.txt

open remote_host_name
username
password

cd /path/
ls
mdelete file*
mput C:\Temp\file*
quit

Windows Scripting using WSH

posted Jul 7, 2011 2:12 PM by Amol Kolhe   [ updated Jul 7, 2011 2:23 PM ]

This is my attempt to dabble with WSH or Windows Script Host.

An excellent resource on Windows scripting is located here:
http://www.pctools.com/guides/scripting/id/30/?act=reference

Here's Microsoft's scripting documentation website (its not great, but its free):
http://technet.microsoft.com/en-us/library/bb902776.aspx

Here's a good link about all the options for the popup method:
http://msdn.microsoft.com/en-us/library/x83z1d9f%28VS.85%29.aspx

To skip reading and download the final versions follow these links:

Here's a windows batch file that I used in the past, to launch applications on startup. The first thing it does is locks my workstation, so that I can leave it unattended and go grab a coffee, while my system starts. Next it launches apps in a certain order. So, when I'm back with a coffee, my system is up and ready for me to use it. I don't have to sit and launch applications.

#File: LaunchWorkspace.cmd
rundll32.exe user32.dll,LockWorkStation
C:
cd "%USERPROFILE%\Desktop\Amol\Workspace\Shortcuts\Quick Launch"
"C:\Program Files\Microsoft Office Communicator\communicator.exe"
sleep 10
"C:\Program Files\Microsoft Office\Office12\OUTLOOK.EXE" /recycle
sleep 30
start /MIN Firefox.lnk
sleep 60
start /MIN mintty.lnk
sleep 5
start /MIN SecureCRT.lnk
sleep 5
start KeePass.lnk
start /MIN Notepad2.lnk "%USERPROFILE%\Desktop\Amol\Docs\Notes.txt"
start /MIN explorer "%USERPROFILE%\Desktop\Amol"
start X-Multiwindow.lnk

This worked fine for a long time, until I thought, it might be nicer to not show a cmd window on screen. So, I used a vbs script to launch my cmd script. This will not show the command window at all, and still execute the batch file (that can be so dangerous BTW, imagine someone executing random comands on your system without you ever being able to see a window. Don't use this for illegal purposes!!):

# File: LaunchWorkspace.vbs
Set WshShell = CreateObject("WScript.Shell")
WshShell.Run chr(34) & "C:\bin\LaunchWorkspace.cmd" & Chr(34), 0
Set WshShell = Nothing

This worked nice and fine, but I thought it might be nice to learn some windows scripting.

The basic difference between any unix script and a windows script is return codes. In Windows, Successful execution of a command/script returns 1. Failed execution returns 2

For example, if you want to get user input using an ok/cancel popup, ok will return 1 while cancel will return 2.

Here's a small script, which will track an ok/cancel input from user:

set WshShell = CreateObject("WScript.Shell")
Choice = WshShell.Popup("The computer will be locked in 10 seconds. Press Ok to lock it now or Cancel to avoid locking.", 10, "Title", 65)
msgbox Choice

Here's my final script to launch programs based on my previous batch file, but a bit smarter.



'Filename: LaunchWorkspace.vbs

'launch_workspace_timeout = 15
lock_timeout = 10

Set WshShell = WScript.CreateObject("WScript.Shell")

'Choice = WshShell.Popup("Launching Workspace. Press Ok to continue or Cancel to abort. Operation will continue automatically, in " & launch_workspace_timeout & " seconds.", launch_workspace_timeout, "LaunchWorkspace", 65)
'If Choice = 2 Then
' WScript.Quit
' If I'm not connected to Network, or for any reason, I don't wanna run the script
' I can cancel if I want.
' Otherwise, it will just auto-continue.
'End If

'Choice = WshShell.Popup("Launching Workspace. The computer will be locked in " & lock_timeout & " seconds. Press Ok to lock now. Press Cancel to avoid locking. Press No to abort Launching Workspace.", lock_timeout, "LaunchWorkspace", 65)
'Wscript.Echo Choice
'If Choice = 2 Then
' Do nothing
' Just stop the timeout. And not Lock Workstation.
'Else
' Lock Workstation
' WshShell.Run("rundll32.exe user32.dll,LockWorkStation")
'End If

Choice = WshShell.Popup("The computer will be locked in " & lock_timeout & " seconds. Press Ok to lock now, Cancel to avoid locking. Press No to abort Launching Workspace.", lock_timeout, "LaunchWorkspace: initializing ....", 3)
'This will provide a Yes/No/Cancel Dialog.
'Yes returns 6, No returns 7, Cancel returns 2
'Wscript.Echo Choice
'If Choice = 2 Then
' Do nothing
' Just stop the timeout. And not Lock Workstation.
'Else
' Lock Workstation
' WshShell.Run("rundll32.exe user32.dll,LockWorkStation")
'End If

Select Case Choice
   Case 6  'Yes
       WshShell.Run("rundll32.exe user32.dll,LockWorkStation")
   Case 7  'No
       Result = WshShell.Popup("Aborted Launching Workspace.", 2, "LaunchWorkspace", 0)
       WScript.Quit
   Case 2  'Cancel
       ' Do nothing
       ' Just stop the timeout. And not Lock Workstation.
   Case Else
       WshShell.Run("rundll32.exe user32.dll,LockWorkStation")
End Select

'Run works like this:
'   object.Run(strCommand, [intWindowStyle], [bWaitOnReturn])
'   strCommand is the command to execute
'   [intWindowStyle] is an integer between 0-8 which determines the window state
       '0: No Window, 1: Normal Window, 2: Minimized Window, 3: Maximized Window, etc
'   [bWaitOnReturn] is True/False and determines whether to wait for the command to finish (i.e. Application to close)
'       default is false

' Note: If the application name/path has space in it, then we need to use 3 double quotes before and after instead of 1 double quotes.
WshShell.Run """C:\Program Files\Microsoft Office Communicator\communicator.exe""",1,False
WScript.Sleep(5000)

WshShell.Run """C:\Program Files\Microsoft Office\Office12\OUTLOOK.EXE""" & "/recycle",1,False
WScript.Sleep(5000)

WshShell.Run """C:\Program Files\Mozilla Firefox\firefox.exe""",1,False
WScript.Sleep(50000)

WshShell.Run """C:\Program Files\Notepad2\Notepad2.exe""" & "%USERPROFILE%\Desktop\Amol\Docs\Notes.txt",1,False
WScript.Sleep(5000)

WshShell.Run "explorer" & " %USERPROFILE%\Desktop\Amol",1,False
WScript.Sleep(5000)

WshShell.Run "C:\cygwin\bin\mintty.exe - ",1,False
WScript.Sleep(5000)

'Start X-windows and xterm.
WshShell.Run "C:\cygwin\bin\run.exe /usr/bin/xwin -clipboard -trayicon -logverbose 0 -silent-dup-error -lesspointer -screen 0 @2 -multiwindow",1,False
WScript.Sleep(5000)
WshShell.Run "C:\cygwin\bin\run.exe /usr/bin/xhost +",1,False
WshShell.Run "C:\cygwin\bin\run.exe /usr/bin/xterm",1,False
WScript.Sleep(5000)

'WshShell.Run "C:\cygwin\bin\run.exe /usr/local/bin/mrxvt -ls +showmenu",1,False

Result = WshShell.Popup("Workspace Launch Complete.", 3, "LaunchWorkspace", 0)



Echo
'VBScript Example
WScript.Echo "Hello World!"
WScript.Echo "Hello" & " " & "World!"
WScript.Echo "Hello", "World!"

'JScript Example
WScript.Echo("Hello World!")
WScript.Echo("Hello" + " " + "World!")
WScript.Echo("Hello", "World!")

'VBScript Example
set WshShell = CreateObject("WScript.Shell")
WScript.Echo WshShell.ExpandEnvironmentStrings("%SystemRoot%")
WScript.Echo WshShell.ExpandEnvironmentStrings("%WinDir%")

AppActivate
'VBScript Example
Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.AppActivate ("Internet Explorer")

Run
'VBScript Example
Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.Run("%windir%\notepad.exe")
ReturnCode = WshShell.Run("%windir%\notepad.exe", 1, True)

Quit
'VBScript: terminate the current script
WScript.Quit
'JavaScript: terminate the current script with exit code 2
WScript.Quit 2

Sleep
'VBScript Example
WScript.Sleep 2000 'sleep for 2 seconds

'JScript Example
WScript.Sleep(2000) 'sleep for 2 seconds

Send Keys
'VBScript Example
Set WshShell = WScript.CreateObject("WScript.Shell")

WshShell.Run "%windir%\notepad.exe"
WshShell.AppActivate "Notepad"
WshShell.SendKeys "Hello World!"
WshShell.SendKeys "{ENTER}"
WshShell.SendKeys "abc"
WshShell.SendKeys "{CAPSLOCK}"
WshShell.SendKeys "def"

Write to Registry
'VBScript Example
Set WshShell = WScript.CreateObject("WScript.Shell")
WshShell.RegWrite "HKLM\Test\Value", "Test String", "REG_SZ"
WshShell.RegWrite "HKLM\Test\IntValue", 543, "REG_DWORD"

GetObject
Set MyDocument = GetObject("c:\work\order.doc")
MyDocument.SaveAs "c:\work\order_new.doc"

Mac OS X Tips

posted Jul 7, 2011 1:59 PM by Amol Kolhe   [ updated Jul 23, 2011 6:52 AM ]

** These are tested to work on Mac OS X (10.5.8) **

Clear Caches
# Clearing cache defeats the purpose of caching. However, I've noticed that the mac gets slower as this cache grows.
# Clearing the cache, makes everything faster.
# I'm guessing, apple didn't implement it properly, or that the disk I/O speeds on the current generation of hard drives makes it slow.
rm -rf ~/Library/Caches/*
reboot

Make Dock transparent in 3D view
# Backup the dock, in case you feel like restoring later
cd /System/Library/CoreServices; tar -cf Dock.app.tar Dock.app
cd /System/Library/CoreServices/Dock.app/Contents/Resources; rm -rf scurve*.png frontline.png && killall Dock

Make Dock transparent in 2D view
# Backup the dock, in case you feel like restoring later
cd /System/Library/CoreServices; tar -cf Dock.app.tar Dock.app
cd /System/Library/CoreServices/Dock.app/Contents/Resources; rm -rf bottom?.png && killall Dock

Set Dock Size
defaults write com.apple.dock largesize -Number 42 && killall Dock

2D Dock Background
defaults write com.apple.dock no-glass -boolean YES && killall Dock

Dock Pinning Start
defaults write com.apple.dock pinning -string start && killall Dock

Dock Pinning Start
defaults delete com.apple.dock pinning && killall Dock

Dock show transparent icon for hidden applications
defaults write com.apple.dock showhidden -bool true && killall Dock

Add Recent Applications to Dock
defaults write com.apple.dock persistent-others -array-add '{ "tile-data" = { "list-type" = 1; }; "tile-type" = "recents-tile"; }' && killall Dock

Show only running apps in Dock
defaults write com.apple.dock static-only -bool TRUE && killall Dock

Set Login Picture
defaults write /Library/Preferences/com.apple.loginwindow DesktopPicture "/System/Library/CoreServices/Finder.app/Contents/Resources/vortex.png"

Dashboard Disable
defaults write com.apple.dashboard mcx-disabled -boolean YES && killall Dock

Dashboard Enable
defaults write com.apple.dashboard mcx-disabled -boolean NO && killall Dock

Auto Mount disks on startup

sudo defaults write /Library/Preferences/SystemConfiguration/autodiskmount AutomountDisksWithoutUserLogin -bool true 

Disable Ping
defaults write com.apple.iTunes hide-ping-dropdown -bool TRUE

Spacer Tile on Dock
defaults write com.apple.dock persistent-apps -array-add '{"tile-type"="spacer-tile";}'

Disable .DS_Store and other hidden files on network folders
defaults write com.apple.desktopservices DSDontWriteNetworkStores true

Expect

posted Jul 7, 2011 1:51 PM by Amol Kolhe

http://en.wikipedia.org/wiki/Expect

#!/usr/bin/expect -f
expectExpects string from a process or stdin
Stores matched characters in variable $expect_out(0,string)
Stores all the parsed characters in variable $expect_out(buffer)

If no matches are found, the command times out after $timeout seconds, which is 10 seconds by default.
expect "hi\n"
set string "hi\n"; expect $string

expect $string1 -- $string2

expect -re "*$sub_string*"
timeoutA variable which stores the timeout for the expect operation.
Default value of timeout, if not defined by user is 10 (seconds).
If set to -1, expect will wait indefinitely.
If set to 0, expect will timeout immediately
set timeout 3
set timeout 0
set timeout -1
sendSends a response to a process and/or stdoutsend "hello\n"
set response "hello\n"; send $response
spawnspawns the specified program/command spawn ssh root@192.168.20.150
interactStops expect from reading commands and transfers the control to the user. The user can then type commands and receive output.interact
execExecutes the specified unix commandsexec ls


Here's a simple expect script:
#!/usr/bin/expect -f

set timeout 3
spawn ssh root@192.168.20.150

expect {
 "password:" {send "password\r"}
 timeout { error "timed out"; exit }
}
expect "#" {
 send "cd /shares/Public\r"
}
interact


Here's some commonly used expect patterns:
Password or password:expect "\[Pp]assword:"
expect "assword:"
Command Prompt # $ >expect "#" -- "$" -- ">"
expect "\[#$>]"
expect "(#|$|>)"

TCL

posted Jul 7, 2011 1:44 PM by Amol Kolhe   [ updated Jul 7, 2011 1:48 PM ]

http://wiki.tcl.tk/

TCL Keywords
Keyword Format/Sample CodeComments
setset <Name> <Value>
set var1 string
set var1 $var2
set var1 "string1 string2"
set var1 {string1 string2}
Everything is a string. Numbers are also strings. This makes life very easy.
putsputs "string1 string2"
puts $var1
puts "$var1"
puts "string1 string2 $var1"
exprexpr $a + $b
set x [expr $a+$b]
incrincr counter
incr counter 1
incr counter -1
same as:
set counter [expr $count +1]
set counter [expr $count -1]
whileset count 0
while { $count < 10 } {
 set count [expr $count +1]
 puts $count
}  
 
forfor {set count 0} {$count < 10} {incr count -1} {
 puts $count
}
if
else
elseif
if { $count < 10 } {
 puts $count
} elseif { $count = 0 } {
 puts "Err"
} else {
 puts "Hmm"
}
switchswitch -- $count \
 1 { }
 2 { }
 3 - 4 { }
 default { }
break
continue
procproc function_name { } {
}
return
global#within a function, assuming var1 is being set outside the function
global var1
sourcesource /path/file.tcl
listset var1 [list a b c d] same as set var1 "a b c d"
llengthllength "a b c d"
# returns 4
llength $var1
lindexlindex "a b c d" 0
#returns a
lrangelrange "a b c d" 0 2
# returns a b c
foreachforeach var $list1 {
  puts $var
}
concatconcat a b {c d}
#returns a b c d
Eats up 1 starting and 1 ending curly braces
lappend lappend <Variable> <Element to Append>
lappend var1 e
#returns a b c d e
add element to list
linsert linsert $<Variable> <Position> <Element>
linsert $var1 0 z
#returns z a b c d e
lreplacelreplace $<Variable> <Starting Position> <Ending Position> <Elements>
set var [list a b c d e]
lreplace $var1 1 3 x y
#returns a x y e
deletes elements from <staring position> to <ending position> and inserts <elements in their place.
lsearch

lsearch $<Variable> <Element>

lsearch -exact

lsearch $var a
#returns 0 i.e. a is the first element
set var [list a b c d ?]
lsearch $var "?"
#returns 0
lsearch -exact $var "?"
#returns 5
lsortlsort $var
lsort -real
lsort -integer
lsort -decreasing

lsort $var
lsort [list e d c b a]
#returns a b c d e
splitsplit $<Variable> <separator>

split "/a/b/c/d" "/"
#returns {} a b c d
join join <list> <separator>

join "a b c d" "/"
#returns "a/b/c/d"
join "{} a b c d" "/"
#returns "/a/b/c/d"
scan
format set DATE [clock format [clock seconds] -format {%y%m%d %H%M%S}]
string compare string compare <string> <string>

if { [string compare $a $b] == 0} {
 puts "strings are identical"
}
Returns 0 for success or 1 for fail
string matchstring match <regexp> <string>

string compare "*.txt" "1.txt"
Returns 0 for success or 1 for fail
string first string first <substring> <string>Returns the position of string1 in string2 or -1 if not found.  
string last string last <substring> <string>Returns the position of string1 in string2 starting from the end. Or Return -1 for fail.
string lengthstring length <string>
string length $var
Returns the length of string
string indexstring index <string> <position>
string index "abcde" 2
Returns c
string rangestring range <string> <position1> <position2>
string tolowerstring tolower <string>
string trimleftstring trimleft <string> <character
string trimleft $var "-"

#Removes leading -
string trimrightstring trimright <string> <character>
appendappend var "abc"
append var "def" ghi"

sets var to abcdefghi
lappend
Arrayset var(attr1,attr2) value
set var(1,1) 1
unsetunset <Variable>
infoinfo exists <variable>Returns 1 if the variable exists or 0 otherwise
info localsReturns a list of local variables
info globalsReturns a list of global variables
info varsinfo vars
info vars var*
Returns a list of all variables
Returns a list of all variables starting with var
info commandsinfo commandsReturns a list of all commands
info procsinfo procsReturns a list of procedures
info levelinfo level
info level 0
info level -1
info level -2
Returns information about the stack
Returns the commands and arguments of the current procedure
info script Returns the file name for the script being executed
array sizeset array(0) string1
set array(1) string2
array size array
Returns 2  
array namesset array(0) string1
set array(1) string2
array names array
Returns 0 1
catchif [ catch {some_proc} ] {
 puts "Error"
} else {
 puts "Success"
}
errorset error "Some Error"
error $error
Returns an error message which can be caught by the catch command
evalset cmd "puts \"Hello\""
eval $cmd
upvar
openset file [open "$filename" "r"]
#do something with $file
close $file
#or
if [catch {open $filename} error] {
 puts "$error"
 return
} else {
#do something with $file
}
getwhile 1 {
 if {[gets $file line] == -1} {
   break
 } else {
   puts $line
 }
}
Read file one line at a time
readwhile {![eof $file]} {
 set buffer [read $file 100000]
}
Reads 100000 lines from file and stores in variable called buffer
readforeach line [split [read $file] "\n"] {
 #do something with $line
}
Read file one line at a time
write fileif [catch {open $filename w} error] {
 puts "$error"
 return
} else {
 puts $file "Output"
 puts -nonewline $file "Output"
}
Replace w with a to append instead of overwrite
file dirnameset filename /etc/passwd
file dirname $filename

Returns /etc
file tailset filename /etc/passwd
file tail $filename
Returns passwd
file extensionfile extension /etc/resolv.confReturns .conf
file rootnamefile rootname /etc/resolv.confReturns /etc/resolv
file existsfile exists $filenameReturns 0 if file does not exist
file isdirectoryfile isdirectory $filename1/true if file is directory
file isfilefile isfile $filename1/true if file is a file
file executablefile executable $filename1/true if you have execute permissions
file existsfile exists $filename1/true if file exists
file ownedfile owned $filename1/true if you own the file
file readable file readable $filename1/true if you have read permissions on file
file writablefile writable $filename1/true if you have write permissions on file
file sizefile size $filename Returns byte size of file
file atime
file ctime
file mtime
file uid
file gid
file ino
file mode
file nlink
file dev
file atime $filename Returns time in seconds when the file was last accessed
file typefile type $filenameReturns filetype <file | directory | characterSpecial | blockSpecial | link | socket>
file readlinkfile readlink $filenameReturns the target file name for link
execexec ls $filename
exec date
Executed a unix command
argsproc some_proc { args } {
 puts $args
 llength $args
}
stores all the arguments passed to a procedure
argcputs $argcstores the number of arguments passed to a tcl file
argvputs $argv
Array containing all the command line arguments
argv0puts $argv0Returns script name
regexpregexp $string1 $string2 var1 var2Checks if string1 is present in string2
Where string1 can be a String or a Regular Expression
Returns 1 for success or 0 for failure  
regsub

awk

posted Jul 7, 2011 1:41 PM by Amol Kolhe   [ updated Aug 4, 2011 9:23 AM ]

http://en.wikipedia.org/wiki/AWK

awk 'instructions' inputfile

awk -f script inputfile

awk as grep:
awk '/string/' inputfile
awk '/string/ {print}' inputfile

Printing First column in a space or tab separated file
awk '{print $1}' inputfile

Using -v to send variables to awk
awk -v var=value '{print var}' inputfile
awk -v var1=value1 -v var2=value2 '{print var1 var2}' inputfile

Using -F to specify field separator (comma in the example below):
awk -F, '{print $1}' inputfile
awk -F',' '{print $1}' inputfile

Using built-in variable FS to specify field separator:
awk -v FS=, '{print $1}' inputfile

awk '{print $1; print $2; print $3}' inputfile
awk '{printf $1; printf $2; printf $3}' inputfile
awk '{print $1, $2, $3}' inputfile
awk '{print $1 " " $2 " " $3}' inputfile

awk script format:
BEGIN { }
{ }
END { }


it can be embedded in a shell script like so:
awk '
BEGIN { }
{ }
END { }
'


Creating functions in awk:
function name (parameter list) {
   statements
}



Escape sequences
\a Alert using ascii bell
\bBackspace
\f Formfeed
\n Newline
\r Carriage Return
\t Horizontal Tab
\v Vertical Tab
\ddd Character represented as 1 to 3 digit octal value
\xhex Character represented as hexadecimal value
\c Any literal character c


Arithmetic Operators
+ Addition
- Subtraction
* Multiplication
/ Division
% Modulo
^ Exponentiation


Assignment Operators
++Add 1 to the variable
--Subtract 1 from the variable
+=Assign result of addition
-=Assign result of subtraction
*=Assign result of multiplication
/=Assign result of division
%=Assign result of modulo
^=Assign result of exponentiation


Relational operators
<less than
>greater than
<=less than or equal to
>=greater than or equal to
==equal to
!=not equal to
~matches
!~does not match


Boolean Operators
|| logical OR
&& logical AND
! logical NOT


awk system variables
ARGCnumber of arguments on command line
ARGV an array containing the command line arguments
CONVFMTstring conversion format for numbers
ENVIRONAn associated array of environment variables (out of shell command env)
FS field separator, space is default
OFS field separator in output, space is default
NF number of fields (columns) in the input
NR number of current record (lines) in input
RS record (line) separator, newline is default
ORS record separator in output, newline is default
FILENAME name of input file
FNR number of current record relative to the current file


printf format specifiers, e.g. printf("%s", $1}'
cASCII character
dDecimal integer
iDecimal integer
efloating point format [-]d.precisione[+-]dd
Efloating point format [-]d.precisionE[+-]dd
ffloating point format [-]ddd.precision
ge or f whichever is shortest (with trailing zeroes removed)
GE or f whichever is shortest (with trailing zeroes removed)
ounsigned octal
sstring
uunsigned decimal value
xunsigned hex number. uses 0-9 and a-f
Xunsigned hex number. uses 0-9 and A-F
%Literal %


Conditional Statements
ifif ( x==y ) print x

if ( x ~ /pattern/ ) print x

if (expression) {
statement 1
statement 2
}

if(expression) action1; else action2
expr ? action1 : action2
while loopwhile (condition) { action }
do loopdo { action } while (condition)
for loopfor ( set_counter; test_counter; increment_counter ) { action }

for (i=1; i <=NF; i++) { print $i }  


awk's built-in string functions:
gsub(r,s,t) globally substitute s for each match of the regular expression r in the string t. t defaults to $0
sub(r,s,t) substitute s for the first match of the regular expression r in the string t. t defaults to $0
index(str, substr) return position of substring in string
length(str) returns length of string str or $0 if no str is specified
match(s,r) returns 0 for no match, or returns the position of regular expression r in string s  
split(str,array,fs) parses string into elements of array using fs as field separator and returns number of elements in the array.
sprintf("fmt", expr) uses printf format specification for expr
substr(str,beg,len) return substring of string str at beginning position beg and the characters that follow to max specified length len.
tolower() convert to lower case
toupper() convert to uppercase



awk commands:
break
close(filename)
continue
delete
do
exit
for
getline
if
next
print
printf
return
system(command)    executes the specified command and returns its status
while

awk's built in arithmetic functions
int(x) returns the integer value of x
sqrt(x)
rand() returns pseudo-random number between 0 and 1
srand()
srand(expr)
establishes new seed for rand(). defaults to time of day
exp(x) returns e to the power x
log(x) returns natural logarithm of x (base e)
cos(x)returns cosine of x in radians
sin(x)returns sine of x in radians
atan2(y,x) returns arctangent of y/x in the range -pi to pi

Examples:
Print first column awk '{print $1}' file
Print all lines whose 3rd column equals 1awk '$3==1 {print}'
Sum of first columnawk '{ SUM += $1 } END { print SUM }'
Print 3rd thro Last columns awk '{for (i=3;i<=NF;i++) print $i }' file
Specify delimiter (comma or ,) awk -F',' '{print $1}' file
Specify delimiter (tilde or |) awk -F'|' '{print $1}' file
awk as grep awk '/STRING / {print}' file
Search for a string defined by a shell variable, within a file, and print columns 3 thro lastawk '/'$CKSUM' / {for (i=3;i<=NF;i++) printf $i " "; print ""}' file
#argv.awk awk 'BEGIN { for (x=0; x < ARGC; ++x) { print ARGV[x] } print ARGC}'
#env.awk awk 'BEGIN { for (env in ENVIRON) {print env "=" ENVIRON[env] } } '
#random.awk awk 'BEGIN { srand(); num = 1 + rand(); print num }'
#mkdir.awk awk 'BEGIN { if (system("mkdir amol") != 0) { print "command failed" } } '
#clear.awk nawk 'BEGIN { system("clear") } '
Kill SSH processes whose PPID is 1
(For strange reasons, this happens on cygwin and once an ssh process becomes rogue, it takes up PPID=1)
ps -ef | awk '$3==1 && $NF=="/usr/bin/ssh" { system( "kill " $2 ) }'

ps -ef | awk '$3==1 && $NF ~ /ssh/ { system( "kill " $2 ) }'
Print text in evenly formatted columnsawk '{ printf "%-20s %-40s %-50s \n", $1, $2, $3 }'
# prints columns left aligned with widths 20, 40 and 50 respectively.
Round Off a Numberawk '{ printf "%0.2f \n" , $1 }'
Largest Number in a columnawk '$1 > max { max = $1 }; END{print max}'
Skip Empty linesawk 'NF > 0'
Line Countawk 'END{print NR}'
Word Countawk '{w=w+NF} END{print w}'
Character Countawk '{ c = c + length($0)} END {print c}'
Print file contents with line numbersawk '{print NR , $0}'
Get Max-width of each column (for any number of columns)  awk '
 {
nf=NF
for(i=1;i<=NF;i++) {
  if( length($i) > max[i]) {
    max[i]=length($i)
  }
}
 }
 END {
for(i=1;i<=nf;i++) {
  printf max[i] " "} {print ""
}
 }'
tail -1awk '{last_line=$0} END{print last_line}'
tail -1 alternaive if input doesn’t contain blank linesawk 'BEGIN{RS="\r"} {print $NF}'
head -1awk 'NR==1 {print $0}'
grep -c stringawk '/string/ {count=count+1} END{print count}' 1.txt
grep stringawk '$0 ~ /string/'
ps -ef | grep string | grep -v grepps -ef | awk '$0 ~ /string/ && $0 !~ /awk/'
Shell function using awk to get a value of an XML attribute from an XML file of the following format:

<TAG attr_name=”attr_value”></TAG>

Usage: get_attr_value_from_xml filename.xml attribute_name

get_attr_value_from_xml() {
 FILE=$1
 ATTR_NAME=$2

 awk 'BEGIN{RS=" "}
 /'ATTR_NAME'/ {
   gsub( /"/ , "" )
   split( $0, result, "=" )
 } ' $FILE
}


sed

posted Jul 7, 2011 1:39 PM by Amol Kolhe

http://en.wikipedia.org/wiki/Sed

sed -f sedscriptfile inputfile > newfile

sed -e 'instruction' inputfile > newfile

sed -e 'instruction1' -e 'instruction2' inputfile > newfile
sed -e 's/search1/replace1/g' -e 's/search1/replace1/g' inputfile > newfile

sed 'instruction1; instruction2' inputfile > newfile
sed -e 's/search1/replace1/g ; s/search1/replace1/g' inputfile > newfile

sed -n -e 's/search/replace/p' inputfile > newfile

sed as grep:
sed -n '/string/p' inputfile

Syntax for sed:

[address]s/string/replace/flags

flags:
n is a number between 1-512 indicating that the nth replacement should be made for the matches string
g make changes globally. Omitting the g, will result in replacement of only the first match
p print the contents (useful with sed -n)
w file writes the output to file

sed can also be used to perform vi functions append, insert and delete.
FunctionFormat Example
Append[line-address]a\
text
/pattern/a\
text line 1\
text line 2
Insert[line-address]i\
text
/pattern/i\
text line 1\
text line 2
Change[line-address]c\
text
/pattern/c\
text line 1\
text line 2
Delete[line-address]d /pattern/d

Regular Expressions

posted Jul 7, 2011 1:37 PM by Amol Kolhe   [ updated Jul 7, 2011 1:41 PM ]

http://en.wikipedia.org/wiki/Regular_expression

Regular expression describes a pattern or a particular sequence of characters.

Metacharacters are special symbols with special interpretation

Metacharacters:
.matches any single character
*matches one or more occurances of  a single character
[]matches any one of class of characters defined between the brackets
^first character of regular expression, matches the beginning of line
$last character of regular expression, matches the end of line
\Escapes the special character that follows
\{n,m\}Matches a range of occurances of the single character or regular expression that immediately precedes it


Extended Metacharacters (egrep & awk):
+Matches 1 or more occurance of the preceding regular expression
?Matches 0 or more occurance of the preceding regular expression