Unix-like Operating Systems

General

• grub boot, change to run level 3
  • Directions here
  • put a 3 on the end of the line, then boot
• md5sum -c *.md5
  • when you have file_whatever.iso and a file_whatever.iso.md5, and you want to check
• file = determine file type and output that to STDOUT
• Here Documents - writing files via command line
  • enclose delimiter in single quotes if you want to turn off backtick, $variable and arithmetic expansion
  • If the redirection operator is `<<-', then all leading tab characters are stripped from input lines and the line containing DELIMITER. This allows here-documents within shell scripts to be indented in a natural fashion.

cat > file_you_want_to_save_text_to.txt <<'EOF'
bunch
of
lines
and
metacharacters ! # ? * "
it's all good
EOF

• gprof
• tar -h option dereferences symbolic links and you get the pointed-to file not the symlink
• Get a list of all files in a tarball:
  • tar -tzf backup.tar.gz (GNU tar)
  • lz backup.tar.gz
• tar zcvf - ./ | ssh user@host "cat > " - "tar over ssh"
• puppetd
• pgrep <string> - return the PIDs of any process whose name greps with "string"
• test
• Unix-like System Startup
• history | grep <search string>; !<history number>
• sed
• gawk - GNU awk
• diff
• Unix Run Processes at Startup
• Unix Network Administration
• umask
• install
• gunzip ... unzip ... uncompress
• screen
• SSH and SSH-Keygen
• BASH terminal prompts
  • PS1="\u@\h \w\n$ " ... user@host pwd newline $prompt
  • PS1="\n\[\e[32m\]\u@\h \[\e[33m\]\w\[\e[0m\]\n\$"
• date
• cron
• echo "hello user2 EOF" | write user2 = send a one-line message to user2 logged in on a machine.
• wall message => send a message to everybody
• echo $[ 34 * 11 ] - command line math
• patch
  • the following command will copy the list of files and directories which start with an uppercase letter in the current directory to destdir: /bin/ls -1d [A-Z]* | xargs -J % cp -rp % destdir
  • implemented differently between gnu and bsd (everyone
• find
• httrack
• ctags
• tidy -indent -wrap 0 index.html > index_new.html
  • wrap 0 means don't wrap long lines
• mount
• nfs
• ps - process server
• rsync
• C-z to stop, jobs to list background processes, fg %1, fg %2 depending on process number
• indent
• sudo su <user>
• chmod
  • example: a+rwX
  • u = user (file owner)
  • g = group
  • o = other (randos!)
  • a = all users
  • r = read
  • w = write
  • x = execute & search for directories
  • X = execute/search only if the file is a directory or already has execute permission for some user
  • s = set user or group ID on execution
  • t = restricted deletion flag or sticky bit
• grep
• top
• df = report file system disk space usage... "macroscopic" view of storage devices
• du -h --max-depth=1 = How big are all the files and directories in this directory... "microscopic" view of file space usage in a specific directory.
• Passing the STDERR to the bit bucket 2>/dev/null
  • 0, 1, and 2 = STDIN, STDOUT, and STDERR, respectively
  • By default, if you don’t name or number one explicitly, you’re talking about STDOUT
  • > is output redirection to a file
  • >> Appends to output to a file, rather than clear out the file
  • >& Redirect both regular and error output to a file
• mail
• proc directory
• lsof
• bash shell keyboard shortcuts
• basename and dirname
• sort -u
• tee - Like a T-shaped pipe joint.
• curl
  • writes to stdout by default
  • -O save the file with the same name as in the URL.
  • -o new_file_name for renaming
  • "-C - ": checks if part of the file has already been downloaded. If the file exists and is complete, curl recognizes this and doesn't download the file again. If the file is partially downloaded, curl will attempt to resume the download from where it left off.
  • -Z download multiple files in parallel
  • -k --insecure

Ghetto Parallelism

Using split

• split --number=l/6 -d --additional-suffix=.txt filelist.txt filelist

function parallelize {
    set -x
    local input_file=$1 # the file.txt containing input lines to be split
    shift
    local n_splits=$1 # the number of chunks/instances of command to run
    shift
    echo "input: $input_file"
    echo "nsplits: $n_splits"
    local base=`basename $input_file .txt`
    echo $base
    local suffix="_${n_splits}_chunk.txt"
    split --numeric-suffixes --number=l/$n_splits --additional-suffix=$suffix $input_file $base
    # no mid-line splits
    set +x

}

Using xargs

• ... | xargs --verbose -i ...\;
  • Default placeholder string token is {}, sometimes I will change it to '{}'
• find . -name "*.png" | xargs -l --max-procs=30 ./convert_to_tiff.sh
  • that's the letter l not 1

# convert_to_tiff.sh
ORIGNAME=$1
DIRNAME=$(dirname $ORIGNAME)
BASENAME=$(basename $ORIGNAME .png)
NEWNAME="../lanczos_tiff/$DIRNAME/$BASENAME.tif"
echo "$ORIGNAME -> $NEWNAME"
convert -verbose $ORIGNAME $NEWNAME

System Monitoring

• mpstat -P ALL 5 | tee profile/test2 - givers per core usage stats every 5 seconds
  • %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
• strace, ltrace
• lsof

wget

• wget --execute="robots = off" --mirror --convert-links --no-parent --wait=5 <URL> - mirror a site locally
• -r retrieve recursively
• --no-parent, -np - Do not ever ascend to the parent directory when retrieving recursively. This is a useful option, since it guarantees that only the files below a certain hierarchy will be downloaded.
• --no-directories, -nd - Do not create a hierarchy of directories when retrieving recursively. With this option turned on, all files will get saved to the current directory, without clobbering
• --timestamping, -N - Turn on time-stamping
• --level=depth, -l - Set the maximum number of subdirectories that Wget will recurse into to depth.
• --mirror - equivalent to -r -N -l inf --no-remove-listing (keeps FTP directory listings.)
• --accept-regex urlregex
• --quiet, -q - used if your output needs to be free of info messages

Examples

• wget -nc -np -r -A "*.xml" -l 2 https://rockyweb.usgs.gov/vdelivery/Datasets/Staged/Elevation/LPC/Project