Concurrent Computing on Remote Servers
(Redirected from Amazon Concurrent Computing)
Jump to navigation
Jump to search
IPython cluster on lgchrms
- Get virtualenv
- Install Python 3.4
mkdir python34 cd python34 git init git remote add -t 3.4 -f origin https://github.com/python/cpython.git git checkout 3.4 ./configure make sudo make altinstall # install ipython # then move /usr/local/bin/ipython out of the way -- ipython3 should be there ready to use
- use IPython.parallel inside an ipython terminal while ssh'ed in
- TBD : Learn how to start ipython notebook on nih server and access via ssh
Steps
- Create new security group - open all ports for my ip address, or certain ports like 8888 for ipython notebook server
- Bid price rule of thumb is little less than a penny per hour per core of the 32core machines (~$0.27)
Articles
- Randy Zwitch - is awesome
StarCluster
- STAR: Cluster - "open source toolkit for Amazon EC2"
Using IPython
- Using IPython for parallel computing - Learn everything in here.
- HERE TOO
- Open ipython in server ssh session and
from IPython.lib import passwd; passwd()
- in bash shell:
openssl req -x509 -nodes -days 365 -newkey rsa:1024 -keyout mycert.pem -out mycert.pem
- generate nbserver profile
colettace@lgchrm18 ~ $ ipython3 profile create nbserver [ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_config.py' [ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_notebook_config.py' [ProfileCreate] Generating default config file: '/data/colettace/.ipython/profile_nbserver/ipython_nbconvert_config.py' colettace@lgchrm18 ~ $ cd .ipython/profile_nbserver/ colettace@lgchrm18 ~/.ipython/profile_nbserver $ l ipython_config.py ipython_nbconvert_config.py ipython_notebook_config.py log/ pid/ security/ startup/ static/
- if from bash shell type
ipython profile create
will generate config scripts for profile_default
- if from bash shell type
- edit the ~/.ipython/profile_nbserver/ipython_notebook_config.py file
c = get_config() # Kernel config c.IPKernelApp.pylab = 'inline' # if you want plotting support always # Notebook config c.NotebookApp.certfile = u'/absolute/path/to/your/certificate/mycert.pem' c.NotebookApp.ip = '*' c.NotebookApp.open_browser = False c.NotebookApp.password = u'sha1:bcd259ccf...[your hashed password here]'
- start the server
ipython3 notebook --profile=nbserver
- In the notebook
import psutil; psutil.cpu_count()
- Access through ssh tunnel
ssh -N -f -L localhost:6000:localhost:7000 username@dest.ination.com
links local port 6000 to remote port 7000- -N = This flag is used to tell SSH to not execute a remote command; it is used exclusively with port forwarding for situations like this.
- -f = This flag tells SSH to go into the background before it executes the command but after the port forwards are established. This command also implies "-n" which prevents reading from stdin and is necessary when putting SSH in the background.
- -L = This command sets up the port forwards,
[bind_address:]port:host:hostport
It essentially links an arbitrary local port to a port on the remote machine over an SSH tunnel.