Wikipedia:Reference desk/Archives/Computing/2013 April 5

Welcome to the Wikipedia Computing Reference Desk Archives
The page you are currently viewing is an archive page. While you can leave answers for any questions shown below, please ask new questions on one of the current reference desk pages.

April 5

Working on many files in unix

I need to write a bash script to run a program against all the files in a directory tree. The program takes two arguments, the input filename and the output filename where the output filename should be the same as the input filename but with a suffix added. So, for one file it works something like this: program -i filename -o filename.suffix. For many files I thought about using the find command, but can't get either the -exec action of find or xargs to work with the filename appearing twice or with a shell function. So that is something like:

find . -type f -exec program {} {}.suffix ;

or

afunc()
{
   program $1 $1.suffix
}
 
find . -type f -exec afunc {} ;

The first reports "find: missing argument to `-exec'". The second reports "find: `afunc': No such file or directory".

I am also looking for a way to avoid processing a file which has already been processed (and ending up with 2 suffixes) and for a way to report and act on any error messages the program might produce, but without stopping the script. Astronaut (talk) 09:26, 5 April 2013 (UTC)

The first case should end in ;\ so

    find . -type f -exec program {} {}.suffix ;\

The second case can't work because afunc is a name inside this bash script, and program is a subprocess of a subprocess of that shell. You can create another script and have find's exec call that. Personally my bash scripting is weak and I devolve anything but simple tasks to a Python script (where os.walk does the hard work). -- Finlay McWalterჷTalk 10:53, 5 April 2013 (UTC)

Shouldn't it be \; rather than ;\ ? (That is, you want to escape the semicolon argument required by find from being misinterpreted by the shell as a command separator.)—Emil J. 14:11, 5 April 2013 (UTC)

Oops, yes, it should. -- Finlay McWalterჷTalk 14:12, 5 April 2013 (UTC)

• I've written probably thousands of scripts that do things like that (neuroscience data analysis), and I recommend using a "for" loop. You can probably use $(ls -r) or something similar to generate a list of files. I have always used C shell because I know it better, so I can't be very helpful about the specific syntax, but I've always been able to make that method work pretty easily. Looie496 (talk) 15:32, 5 April 2013 (UTC)

If you make your afunc into a shell script (a file, rather than a shell function in memory — sad that it's a redlink, since shell functions are quite useful; we do discuss them a bit at alias (command)), find will be able to invoke it (perhaps as ./afunc.sh). If you write the script as

#!/bin/bash
for t; do
  program "$t" "$t".suffix
done

you will be able to pass multiple files to it (as with most commands), and then you can use find -type f | xargs ./afunc.sh (or, to properly handle unusual file names, find -type f -print0 | xargs -0 ./afunc.sh). (When the actual command, rather than a wrapper script, accepts multiple arguments, using xargs is also more efficient.) There is likely also a syntax -exec ./afunc.sh {} + that is similar to using xargs, including that you cannot do {} {}.suffix this way.

As for not double-processing, you can ask find to exclude files whose names match a pattern: find -type f \! -name \*.suffix .... (The ! and * are like ; in that they are special to the shell and so must be escaped to be delivered intact to find.)

As for error messages, by default they appear on stderr, and it can be a bit of a pain to intercept and interpret that. However, any properly-designed program will have an exit status you can use. Just like -type f specifies a condition, -exec ... \; does too: it "passes" if the command reports that it is successful. In this fashion you can make a sort of "find script" that does things for each file for which processing succeeded or failed:

find -type f -exec program {} {}.suffix \; -print # name the files that worked
find -type f \! -exec program {} {}.suffix \; -print # name the files that failed
# Complicated example: run additional programs on each success/failure:
find -type f \( -exec program {} {}.suffix \; \( -exec success {} \; , -true \) -o -exec failure {} \; \) \)

Alternatively (and more powerfully), you can do this in your shell script (whether it supports multiple files or not):

#!/bin/bash
program "$t" "$t".suffix && echo "$t" # name file that worked
program "$t" "$t".suffix || echo "$t" # name file that failed
# Complicated example: run additional program depending on success/failure:
if program "$t" "$t".suffix; then
  success "$t"
else
  failure "$t"
fi

The shell is also capable of doing things like checking for preexisting suffixes, checking for preexisting files you might not want to overwrite, and capturing stderr if that turns out to be necessary. --Tardis (talk) 16:02, 5 April 2013 (UTC)

online shop stuff

The trouble with selling stuff online is that you can either pay to use various services, such as amazon, ebay, paypal and so on, or it works out cheaper and easier to sell through your own website, but they cost so much more and take more time and effort to set up to start with. So, I had this idea of setting up a website where people pay a small fee for a subdomain and a template to sell through, in particular it gives them a platform through which to automate the use of various payment services, particularly accepting debit and credit card payments and such like. But the question is, this being my website but with other people using it to sell their stuff, would they use my merchant payment accounts, or would each person need to set up their own to arrange these transactions separately?

And in general, would this idea work, or is there something else that I haven't thought of?

Kitutal (talk) 19:35, 5 April 2013 (UTC)

I’m not sure I see how it would differ from existing services of the type you already mentioned. ¦ Reisio (talk) 19:44, 5 April 2013 (UTC)

There already exist services like you describe. Shopify is a big one, but there are a number of others.

So it's certainly possible. APL (talk) 19:45, 5 April 2013 (UTC)

Yahoo also offers a similar service : SmallBusiness.yahoo.com/ecommerce.

APL (talk) 19:49, 5 April 2013 (UTC)