Software Salariman

Reminder on how to use -exec action in GnuWin32 find command in Windows cmd.exe. For example, if you want to find a string, the format is:

find . -type f -exec grep <pattern> {} ;

If you do any of the following, you can get this cryptic error message: find: missing argument to `-exec'

• Put double-quote marks around the command:

  find . -type f -exec "grep <pattern> {} ;"

• Don't leave a space between braces and semi-colon:

  find . -type f -exec "grep <pattern> {};"

• Use Unix shell escape character:

  find . -type f -exec grep <pattern> {} \;

Finally, if all else fails and you lack time to investigate, use xargs:

find . -type f | xargs grep <pattern>

Labels: GnuWin, Scripting, Windows Cmd

link posted by Kam-Hung Soh @ 7:00 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Julian W. suggested that I write a one line Python loop for my command scripts instead of map(), as in my earlier article. Instead of map(lambda l: expression(l), sys.stdin), I could write for l in sys.stdin: expression(l). An example trivial command to echo all input lines would be:

python -c "import sys; for line in sys.stdin: print line,"

Problem is that the Python interpreter complains:

  File "", line 1
    import sys; for line in sys.stdin: print line,
                  ^
SyntaxError: invalid syntax

Labels: Python, Scripting

link posted by Kam-Hung Soh @ 10:08 AM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Note to self: sed's (Stream EDitor) command y/list1/list2/ to transform / transliterate each character is based on its position in list1 to a character in the same position in list2. list1 and list2 must be an explicit character list, not a regular expression (and hence, not a character class). In other words, if you enter y/[a-z]/[A-Z]/, sed will look for these characters in the input, '[', 'a', '-', 'z' and ']', to replace with '[', 'A', '-', 'Z' and ']' respectively; sed does not expand a character class [a-z] to replace with [A-Z]. Same with Posix character class names such as [:lower:] and [:upper:].

I incorrectly mixed up the idea that sed's transform command with the tr (translate) command, which supports interpreted sequences, e.g. tr [:lower:] [:upper:] will transform all lower case characters to upper case.

Labels: GnuWin, Scripting

link posted by Kam-Hung Soh @ 7:30 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

With Unix shells such as bash and [t]csh, you can set the value of a variable to the result of a command using the back-tick operator (or output replacement). For example, LINES = `wc -l filename`, would set the variable LINES with the result of wc -l, which is the number of lines in filename. This technique is useful when you want pass the value of a computed variable to subsequent commands in a script.

Windows' cmd.exe also supports this feature, in a obscure way, using the for command: for /f %i in ('command') do set VARIABLE=%i. To reproduce the previous example in cmd.exe: for /f %i in ('wc -l filename') do set LINES=%i.

Notes:

• Use %%i in a script.
• Use single-quote marks to delimit a command. If you use double-quote marks, for treats the argument in parentheses as a string.

I saw this and other cmd.exe hacks somewhere but I didn't take a note of it. Grr. Remind myself to update this page when I find that site again.

Labels: Scripting, Windows Cmd

link posted by Kam-Hung Soh @ 7:00 PM   1 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

More uses of GnuWin32 / Cygutils tools getclip and putclip using this recipe: getclip | <command chain> | putclip.

• Copy m'th and n'th column of a table from a browser: cut -fm,n.
• Copy columns from Excel and replace tab character with space: tr \t " ".
• Capitalize letters: tr [:lower:] [:upper:]. (Duh! Enter Shift-F3 in Microsoft Word, thanks to Maria H.).
• Remove indentation from e-mail messages: sed "s/> //".
• Remove indentation from source code in Word document: sed -e "s/^ //" (5-May-2008).
• Join lines broken into multiple lines by e-mail clients: dos2unix | tr -d \n. On a Windows system, tr doesn't recognise CR-LF pairs for terminating a line, so you have to convert them to a Unix-style LF using dos2unix first (6-May-2008).
• Another way to join broken lines: tr -d \r\n using escape codes for carriage return and line feed, respectively (11-May-2008).
• Remove formatting from string: getclip | putclip. This is equivalent to Microsoft Word's Paste Special / Unformatted Text. Also to work-around an annoyance in Outlook 2003, were the Edit / Paste Special is disabled when you are responding to an HTML-formatted document (7-May-2008).
• Remove HTML / XML formatting from input: sed -e "s/<[^>]*>//g" (12-Jun-2008).

A second recipe is (for /f %i in ('getclip') do @command %i) | putclip if command cannot be used in a pipeline. Two examples are basename (return name of file in a path string) and dirname (return path string without file name).

2005-05-01: Don't simply list transformations and filters that can be done with GnuWin32 tools, but ones where existing applications (e.g. Excel, Firefox, Outlook or Word) don't have an easy way to achieve a particular action.

Labels: GnuWin, Scripting, Windows, Windows Cmd

link posted by Kam-Hung Soh @ 2:53 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

GnuWin32 / Cygutils package has two tools for interacting with the Windows clipboard: getclip and putclip. The first copies text from the clipboard to standard output and the second copies text from standard input to the clipboard. These tools are useful when you want to process text from one Windows application before pasting the text into another application, in the following recipe: getclip | <filters> | putclip.

For example, I want to paste all DLL file names in a folder into a document:

1. Navigate to the required folder using 2xExplorer browser.
2. Type Alt+a to select all files.
3. Type Alt+c to copy all file names. 2xExplorer copies the absolute path for each file.
4. Start cmd.exe console.
5. In cmd.exe console, enter: getclip | cut -d\ -fn | grep dll$ | putclip. cut is GnuWin32 tool which selects a column of data given a column delimiter (-d\ defines backslash) and field number (-fn defines column n). grep filters the output to only list files with "dll" in their name.
6. Start editor.
7. Paste the text in the clipboard in destination document.

Of course, you can do the same using Excel:

1. Navigate to the required folder using 2xExplorer browser.
2. Type Alt+a to select all files.
3. Type Alt+c to copy all file names. 2xExplorer copies the absolute path for each file.
4. Start Excel.
5. Paste data in a worksheet column.
6. Select all cells by typing Shift+Space.
7. Open Convert Text to Columns Wizard by typing Alt+d+e.
8. Select Delimited data type by typing Alt+d.
9. Type Alt+n to go to page 2.
10. Select Other delimiter by typing Alt+o, then enter "\" for paths.
11. Type Alt+f run the wizard.
12. Start Auto Filter by typing Alt+d+f+f.
13. Move to filter column using the mouse (no keyboard shortcuts?) then select from the drop down list (Custom …).
14. Select ends width criteria, enter .dll, then press Enter.
15. Move cursor to required column and select it using Control+Space.
16. Copy column by typing Control+C.
17. Start editor.
18. Paste the text in the clipboard in destination document.

The Excel solution has many more steps than the getclip-putclip solution but Excel leads you through to a solution step-by-step. If you're familiar with GNU tools, then getclip-putclip recipe is faster to use and much more extensible.

2008-05-07. I should have remembered that the basename command would output the name of the file without the leading path string. See later article More Uses of Getclip-PutClip about how to use basename in a pipeline.

Labels: GnuWin, Scripting, Windows, Windows Cmd

link posted by Kam-Hung Soh @ 11:20 AM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

When chaining GnuWin32 commands in Windows cmd.exe, you may encounter strange error messages like this:

> ls | grep
…
ls: write error: Invalid argument

The first command reports a write error but the error is really in the second command after the pipe symbol.

You may also encounter a similar write error if the wrong command is found in your PATH variable. For instance, Windows and GnuWin32 both have a find and sort command which support different command-line options, so depending on the order of directories listed in your PATH variable, one version or the other is used. If you enter the wrong command-line options for these commands, they won't start and cause the command earlier in the chain to report some sort of I/O error.

Labels: GnuWin, Scripting, Windows Cmd

link posted by Kam-Hung Soh @ 9:00 AM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Perl has a -n option which implicitly runs a while-loop over all lines in STDIN (while (<>) { }). This mode is handy in a command shell when Perl is the recipient of the output of another command and you don't want to write a script. Can we do the same for Python?

Python has a -c option which runs a command in the string following it. While it's not entirely clear to me what is a Python command, I found that you can write some useful functions using list functions and statements using this template:

python -c "import <package>; print '\n'.join(<list function>(lambda x: <expression>, (s.strip() for s in sys.stdin)))

To use this template, replace <package> with a package name (e.g. os), <list function> with a list function (e.g. filter()) and <expression> with, well, an expression. The rest of the template just constructs a list of strings (without a trailing "\n") from the input and prints the results.

For simple string processing, the list function and expression are not required, resulting in a simplified version of this template:

python -c "import <package>; print '\n'.join(<fn>(s.strip()) for s in sys.stdin)"

While researching this topic, I found an ASPN Python Recipe called Pyline to help write commands. Here's the examples in that recipe rewritten using my template:

Print the first 20 characters of each line:

tail test.txt | python -c "import sys; print '\n'.join(s.strip()[:20] for s in sys.stdin)"

Print the 7th word in each line, assuming the separator is ' ':

tail test.txt | python -c "import sys; print '\n'.join(s.strip().split(' ')[6:7] for s in sys.stdin)"

Note that you can also get columns of text from a file using the cut command. Also note that the reason for using the array slice is to avoid getting an IndexError exception if the string is not long enough.

List all files that are greater than 1024 bytes in size:

ls | python -c "import os, sys; print '\n'.join(filter(lambda x: os.path.isfile(x) and os.stat(x).st_size > 1024, (s.strip() for s in sys.stdin)))

Generate MD5 digest values for a list of files, like md5sum.

ls *.txt | python -c "import md5, sys; print ''.join('%s %s' % md5.new(file(s.strip()).read()).hexdigest(), s) for s in sys.stdin)"

26-Apr-2008: Replaced list comprehension statement (for-in with square brackets) with generator expression (for-in with parentheses) in the template to avoid very large lists stored in memory.

Added MD5 digest example, and realised that we only need to use list functions (e.g. filter()) if you want to change the members of the resulting list. Otherwise, the simpler template suffices.

Labels: Python, Scripting

link posted by Kam-Hung Soh @ 9:51 AM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

To find all palindromes from a list of words in a file, one word per line, you could write a procedural Python program like this:

for row in file('test.txt'):
  s = row.strip()
  if s == s[::-1]:
    print s

Here's a functional Python version, with notes below:

from itertools import imap
filter(lambda s: s == s[::-1], imap(str.strip, file('test.txt')))
  5       3              4       2               1

1. Create a file iterator.
2. When we read a line from a file into a string, the string has a trailing newline character (e.g. 'add\n'). We want to remove that trailing newline character, so we use the itertools.imap() function to create a new iterator that applies the str.strip() to each line read. The result is that we have an iterator that provides strings without the newline character.
3. Define an anonymous function using lambda keyword that returns true if the input string is a palindrome.
4. Python idiom for returning a reversed sequence (a string is a sequence of characters).
5. Use the filter function to return a list of palindromes.

Using this input file …

add
dad
dam
mad
made
madam
set

… the result of running the functional script is:

['dad', 'madam']

Labels: Python, Scripting

link posted by Kam-Hung Soh @ 8:13 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Python has a csv module for reading and writing CSV files (usually exported by Excel or database tables). The basic use of this module is documented in the on-line help. My CSV files usually have a header row, so the idiomatic way to skip this line is to open the CSV file and use the next() function immediately:

from csv import reader
f = open("blah.csv", "rb")
f.next()
for row in reader(f):
  print row

If your CSV files are pretty simple (e.g. only single line data, no quotes, etc.), you can use list comprehension and array slicing:

                   1       2                                      3
for row in [line.strip().split(',') for line in file("blah.csv")][1:]:
  print row

Notes:

1. You have to remove the trailing "\n" from each line.
2. Split the input line using the delimiter, typically a comma.
3. The list comprehension statement returns all lines, so to ignore the first line, you take a slice of the array starting from the second line.

Labels: Python, Scripting

link posted by Kam-Hung Soh @ 3:51 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

I converted my earlier PowerShell script to prune directories to Python:

     1  from os import listdir, rmdir
     2  from os.path import isdir, exists, join
     3
     4  def prune_directory(path):
     5    if len(path) < 1:
     6      print "Empty path"
     7      return
     8    if not exists(path):
     9      print "Invalid path:", path
    10      return
    11    if not isdir(path):
    12      return
    13    if len(listdir(path)) <= 0:
    14      rmdir(path)
    15      return
    16    for elem in listdir(path):
    17      prune_directory(join(path, elem))
    18    if len(listdir(path)) <= 0:
    19      rmdir(path)

It's almost a one-to-one translation from the initial PowerShell version. The difference is that in Python, you have to ensure that path is a directory before you call listdir(path).

Python 2.5 has a new generator function os.walk() for traversing a directory tree. Below, the recursive function call in lines 13-19 is replaced by a for loop in lines 13-14.

     1  from os import listdir, rmdir, walk
     2
     3
     4  def prune_directory_walk(path):
     5    if len(path) < 1:
     6      print "Empty path"
     7      return
     8    if not exists(path):
     9      print "Invalid path:", path
    10      return
    11    if not isdir(path):
    12      return
    13    for curr, dirs, files in walk(path, topdown=False):
    14      if len(listdir(curr)) < 1: rmdir(curr)

In this sample, os.walk() returns a tuple (curr, dirs, files) for each directory it visits. curr is the current directory being traversed, and dirs and files are the directories and files in that directory. Using the parameter topdown=False, os.walk() starts producing these tuples from the lowest descendant directory and working up to the start directory, path.

Note that the loop's conditional statement uses len(listdir(curr)) instead of just len(dirs). os.walk() generates the dirs list before it visits each of the directories in dirs; if all child directories in dirs have been deleted, dirs would still contain an unchanged list and the parent directory, curr, would not be deleted. At least, that's what I think happens; the Python help doesn't say so explicitly.

In earlier versions of Python, there is a similar function called os.path.walk() but os.walk() is much easier to use.

Labels: Programming, Python, Scripting

link posted by Kam-Hung Soh @ 7:54 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Some commands in Windows cmd.exe batch files have a leading tilde (~) character or a trailing colon-tilde (:~) pair of characters attached to environment variable names. What's the purpose of these characters?

Trailing Colon-Tilde Pair

You can find more about colon-tilde in the help using set /?. Briefly, you can …

• Slice a string in a variable: %NAME:~s,n where s is the start position (zero-offset) and n is the number of characters. If s is negative, then the offset starts from the right side minus 1 (i.e. -1 refers to the rightmost character). If n is negative, then length - n characters, are extracted.

• Replace a substring with another string in a variable: %NAME:s1=s2% where s1 is substring to be replaced and s2 is the replacement.

Leading Tilde

The leading tilde is used to decompose elements in a batch file parameter formatted as a path, such as the parent directory or file extension. The best reference is Frequently Asked Questions Regarding The Windows 2000 Command Processor, "How do I parse a file name parameter into its' constituent parts?" (sic). Note that you can only use a leading tilde for batch file parameters, not environment variables (!).

4-Mar-08. Another version of the reference is Frequently Asked Questions Regarding The Windows 2000 Command Processor. 09-Sep-02.

Labels: Scripting, Windows Cmd

link posted by Kam-Hung Soh @ 7:03 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Adding a path to the PATH environment variable in the current PowerShell session is simpler than I thought:

> $env:path += ";path"

Note: remember to prepend the semi-colon to the new path make a valid path list.

You can make your PATH variable persistent using the SetEnvironmentVariable() .Net method:

[System.Environment]::SetEnvironmentVariable("PATH", $Env:Path + ";path", "target")

… where target is "Machine", "User" or "Process". Check the .Net documentation for what these values mean.

Labels: PowerShell, Scripting

link posted by Kam-Hung Soh @ 7:24 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

I made a backup of all files with a certain pattern files from one directory to another. If the pattern was, say PostScript (*.ps) files, you can use the following PowerShell statement:

Copy-Item -recurse -filter *.ps <source> <destination>

Now I had a new directory with the same structure as the original one, but Copy-Item made many new empty directories because there were files in the source directories but these files were not copied. Just to be tidy, I wanted to prune the empty directories in the destination path. The Remove-Item cmdlet does not have an option to remove empty directories, so I wrote the following short PowerShell script:

     1  function prune-directory {
     2    param ([string]$path)
     3    if ($path.Length -le 0) {
     4      write-host "Empty path."
     5      return
     6    }
     7    if (-not (test-path -literalPath $path)) {
     8      write-host "Invalid path:", $path
     9      return
    10    }
    11    if (@(get-childitem $path).Count -le 0) {
    12      remove-item $path
    13      return
    14    }
    15    get-childitem $path | where-object { $_.PsIsContainer} | foreach { prune-directory $_.FullName }
    16    if (@(get-childitem $path).Count -le 0) {
    17      remove-item $path
    18    }
    19  }

To use it, just enter:

prune-directory <path>

You should verify that the function works the way you expect before using it. Once your directories or files are deleted, they're GONE.

prune-directory() is a recursive function that walks a directory tree and deletes any empty directory it finds. Lines 3-10 check for invalid parameters, lines 11-14 delete the current directory if it is empty and line 18 calls this function for all children which are containers in the current directory. Lines 19-22 are required in case the current directory has no children because they were all deleted by line 15.

In line 11 and 16, we use @(…) to force the result of get-childitem $path to be an array, otherwise we may not be able to count the number of children in a directory. It's a known - uh - nuance in PowerShell that if a cmdlet finds zero or one object, it returns an scalar value rather than an array.

2008-05-15: This change should fix the problem of escape characters in the path string: test-path -literalPath $path.

Labels: PowerShell, Programming, Scripting

link posted by Kam-Hung Soh @ 12:31 PM   2 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb

Compiled files in your Windows computer, such as executables and libraries (or files with a .exe and .dll suffix in their names), can contain some additional information stored in a FileVersion structure. You can see these properties in Explorer's Properties dialog, Details tab.

Before releasing a Windows-based product, I wanted to check that the Copyright and Product Version fields in all compiled files were correctly updated. We always increment the number in Product Version and if the product was released in the start of the year, we also update the Copyright field. You can use Windows Explorer to view these properties in several files at once (just select all relevant files and show the Properties dialog), but if a file had a different value from the others, the Properties / Details tab shows the unhelpful multiple values text. Which file has a value different from the others? It isn't that hard to check each field individually but why not automate the test?

You can use the following PowerShell one-liner to list the Copyright field of all compiled files:

> get-childitem * -include *.dll,*.exe | foreach-object { "{0}`t{1}" -f $_.Name, [System.Diagnostics.FileVersionInfo]::GetVersionInfo($_).LegalCopyright }

To test on the Windows PowerShell folder:

gpowershell.exe Copyright (c) Microsoft Corporation. All rights reserved.
powershell.exe  © Microsoft Corporation. All rights reserved.
pwrshmsg.dll    © Microsoft Corporation. All rights reserved.
pwrshsip.dll    © Microsoft Corporation. All rights reserved.

The first command get-childitem * -include *.dll,*.exe retrieves a list of files in the current directory that have a *.dll or *.exe suffix. The Get-ChildItem cmdlet has a -filter option but it only accepts one pattern.

The second command outputs the filename and the Copyright information of each file using the format (-f) operator. We use the .Net [System.Diagnostics.FileVersionInfo]::GetVersionInfo() method to obtain the file's LegalCopyright (or Copyright) field.

To check Product Version field, use .ProductVersion instead of .LegalCopyright. If you are interested in other fields, check MSDN for a complete list of field returned by GetVersionInfo.

Labels: PowerShell, Scripting

link posted by Kam-Hung Soh @ 6:00 PM   0 comments
Digg it!   Del.icio.us   Earthlink   Furl it!   Reddit!   Stumble It!   Slashdot It!   Yahoo MyWeb