Concurrent Computing on Remote Servers

From Colettapedia
(Redirected from Amazon Concurrent Computing)
Jump to navigation Jump to search

IPython cluster on lgchrms

  • Get virtualenv
  • Install Python 3.4
mkdir python34
cd python34
git init
git remote add -t 3.4 -f origin https://github.com/python/cpython.git
git checkout 3.4
./configure
make
sudo make altinstall
# install ipython
# then move /usr/local/bin/ipython out of the way -- ipython3 should be there ready to use
  • use IPython.parallel inside an ipython terminal while ssh'ed in
  • TBD : Learn how to start ipython notebook on nih server and access via ssh

Steps

  1. Create new security group - open all ports for my ip address, or certain ports like 8888 for ipython notebook server
  2. Bid price rule of thumb is little less than a penny per hour per core of the 32core machines (~$0.27)

Articles

StarCluster

Using IPython

  1. Open ipython in server ssh session and from IPython.lib import passwd; passwd()
  2. in bash shell: openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
  3. generate nbserver profile
colettace@lgchrm18 ~
$ ipython3 profile create nbserver
[ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_config.py'
[ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_notebook_config.py'
[ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_nbconvert_config.py'

colettace@lgchrm18 ~
$ cd .ipython/profile_nbserver/

colettace@lgchrm18 ~/.ipython/profile_nbserver
$ l
ipython_config.py  ipython_nbconvert_config.py  ipython_notebook_config.py  log/  pid/  security/  startup/  static/
    • if from bash shell type ipython profile create will generate config scripts for profile_default
  1. edit the ~/.ipython/profile_nbserver/ipython_notebook_config.py file
c = get_config()

# Kernel config
c.IPKernelApp.pylab = 'inline'  # if you want plotting support always

# Notebook config
c.NotebookApp.certfile = u'/absolute/path/to/your/certificate/mycert.pem'
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.password = u'sha1:bcd259ccf...[your hashed password here]'
  1. start the server ipython3 notebook --profile=nbserver
  2. In the notebookimport psutil; psutil.cpu_count()
  • Access through ssh tunnel
    • ssh -N -f -L localhost:6000:localhost:7000 username@dest.ination.com links local port 6000 to remote port 7000
    • -N = This flag is used to tell SSH to not execute a remote command; it is used exclusively with port forwarding for situations like this.
    • -f = This flag tells SSH to go into the background before it executes the command but after the port forwards are established. This command also implies "-n" which prevents reading from stdin and is necessary when putting SSH in the background.
    • -L = This command sets up the port forwards, [bind_address:]port:host:hostport

It essentially links an arbitrary local port to a port on the remote machine over an SSH tunnel.