Introduction to Python and environments with miniconda, conda and conda-forge

Why miniconda python?

I recommend the miniconda version of python. It is a minimal install of the anaconda python distribution. It installs in your home directory so you don't need administrator priviledges to install, and you can easily create environments which are independent of each other, so you can segregate projects, and you're less likely to run into issues with dependency problems.

Which version of miniconda?

You can get the installer for miniconda here.

For linux, you probably want the newest 3.x 64 bit installer. if the installer that you downloaded is named "Miniconda3-latest-Linux-x86_64.sh", you can start the installer by typing

 sh ./Miniconda3-latest-Linux-x86_64.sh

For mac OS X, you can get either the pkg installer or the bash installer. Again, you want the newest 3.x version. For a pkg install, just double click on the pkg file. For the bash installer, if the installer you downloaded is named "Miniconda3-latest-MacOSX-x86_64.sh", you can start the install by typing

 sh ./Miniconda3-latest-MacOSX-x86_64.sh

For MS Windows, you probably want the newest 64 bit installer. Just double-click on the installer and follow the prompts

(Unix only) Make sure you are using the bash shell

There are several 'shells' available to use under linux/unix. A few popular ones are tcsh and bash. The conda installer sets things up assuming you are running bash, and things will not work properly if you are running tcsh. You can check what shell you are currently using by typing

echo $SHELL

If that comes back and says '/bin/bash' or '/usr/bin/bash', you are probably good to go.

If it somes back and says '/bin/tcsh' or something else, you can start a bash prompt right there, by typing

bash

If you find that you are mostly using python, and having to type 'bash' every time you log in, you can change your default shell to bash by typing

 chsh

and when it asks for what new shell you want to use, type

/bin/bash

From then on, when you log in, your default shell will be bash.

BE CAREFUL though - some of the things that were set up for your old default shell may no longer work when you change to bash. If you aren't sure, check with Pete (poker@aos.wisc.edu) before you make this change.

(Unix/Mac OSX only) Make sure that the correct bash startup file was modified

There are two files that might be used as startup files for bash. In some cases, the miniconda installer will update the file .bashrc in your home directory to enable your miniconda installation, but the bash shell actually uses a file named .bash_profile. If this is the case, even though you are running bash, conda won't be found, and you won't be using the miniconda python you installed. If this is the case, you need to add the contents of .bashrc that miniconda added, to the file .bash_profile.

Because modifying your .bash_profile or .bashrc incorectly can cause issues with you being able to log in or run things properly, at this point, please contact Pete (poker@aos.wisc.edu) for help with this.

Enable conda-forge

The anaconda python distrution uses the tool 'conda' to manage the different packages and modules that you may wish to install. In theory, these packages have all been built to be compatible with one another. Sometimes there are issues, but generally this is true.

The main anaconda distribution includes many popular packages, but there are many more user-supported, open source packages out there that are not included with the main anaconda distrbution. There is another package repostitory called "conda-forge" that includes many, many packages, that are intended to be compatible with the anaconda packages, and use the same 'conda' package manager to install them.

To enable the conda-forge repository, once you've installed miniconda, type the following commands:

 conda config --add channels conda-forge

Now, conda will search the conda-forge repository in addition to the main anaconda repository for packages.

Environments

One of the potential down-sides of using a rapidly developing ecosystem for programming like python, is that sometimes package updates include changes that break compatibility with other packages. For this reason, I generally recommend using the base miniconda environment ONLY for creating and maintaing other environments. It is within these other environments that you will install the custom packages and modules that you need, and actually do your work.
To create your first environment, type something like

conda create --name envname

where envname is the name you want to give the environmenet. It might be something descriptive, like 'climatework' or 'aos575'
You can then activate this environment using the command

conda activate envname

or if that fails, try

 source activate envname

When you do that, you will notice that your command prompt changes to include the name of the environment that you just activated. You can tell which environment you are working in by looking at that beginning part of the command prompt. You can install packages such as matplotlib, numpy, scipy, etc.. by using 'conda install", like:

 conda install matplotlib, numpy, scipy, netcdf4

You can install additional packages later by using the 'conda install' at a later time - you don't have to install them all at once.

You can search for package versions, with 'conda search', for example:

conda search netcdf

If you need to install a specific version of a package, say cartopy 0.18, you can specify the version number like

 conda install cartopy=0.18

Here is a conda cheat sheet that has some other options and info about the conda tool You can deactivate the environment by typing

 conda deactivate

and you will notice that your prompt will change back to show that you are in the base environment again.

You can create many environments, each of which have their own set of installed packages and versions, so they don't conflict with each other. I recommend using a different environment for each project or task you are working on. If you want to test a new version of packages with a project that you already have working, you can create a new/cloned environment to then update and test with the new packages. You can leave your working environment alone, and test updates in the new, updated test environment. That way, if something breaks, so you still have the original working environment available to do your work while you figure out why the update broke something.

A few other tips

List your environments

 conda env list

Run a jupyter notebook on a linux server and connect with a web browser on your local machine through an SSH tunnel

It is possible to connect to a linux machine with ssh, and run a jupyter notebook such that you can connect to it from a web browser on your local machine - this is MUCH faster than running the notebook in a web browser on the remote machine and displaying that on your local machine's Xserver. To do this, make sure you have jupyter installed in your environment on the remote machine - activate the environment first, and then

 conda install jupyter

Instructions for connecting from a mac

Lets pretend you are connecting to cat3.aos.wisc.edu, and want to run a jupyter notebook there. You'll want to change that to the machine that you are actually connecting to.
You'll also want to change the port number to something other than 5900 - only one person on a machine can use each port number. Maybe take 5900 and add the month day of your birthday - mine would be the 3rd, e.g. 5903.
On your local mac, add the following lines to the file ~/.ssh/config (if it does not already exist, just create it. Replace yourusername with your username on the remote machine.

Host cat3 
HostName cat3.aos.wisc.edu 
User yourusername
Compression yes 
ForwardX11Trusted yes

Host cat3_tunnel 
HostName cat3.aos.wisc.edu 
User yourusername
ProxyCommand ssh -W %h:22 cat3 
LocalForward 5900 localhost:5900

To run the notebook, start a new terminal on your Mac OS computer, and log into the remote machine with

$ ssh cat3_tunnel (again, replace that with your machine name)

In this window, if your default shell on the remote machine is not bash, start a bash shell by typing

 $ bash

Then, run the notebook server - the port number needs to match what you used above in the ssh config.

 $ jupyter notebook --no-browser --port=5900

At the end of jupyter’s startup log, it should show something like:

 http://localhost:5900/?token=<long_random_number>

 http://127.0.0.1:5900/?token=<long_random_number>

Paste that URL into whatever web browser you like to use on your local machine – Safari, Chrome, etc., and it should start up the web notebook interface.
Note that first terminal window needs to stay open, since it is running the ssh tunnel.
When you are finished, you can close the browser window/tab that was running the notebook. Then kill the jupyter notebook on the remote machine with -C. It will ask you to confirm by typing "y". Then you can close the ssh connection to the remote machine.

Instructions for connecting from windows with putty

Using putty - download the latest version of Putty
Open putty and enter the server name or IP address as the hostname.
Now, go to the SSH on the bottom left pane, expand the menu, and click on Tunnels
Enter the port number which you want to use to access jupyter on your local machine - I use 5900 plus my birth month day (3rd) so 5903.

Alternate instructions for connecting to an SSEC server through ash.ssec.wisc.edu

Open a terminal window, and in that type:

ssh -L [portnumber]:localhost:7159 [username]@ash.ssec.wisc.edu ssh -L7159:localhost:[portnumber] [username]@[servername].ssec.wisc.edu

[substitute your username in for [username]

After entering your password, you will encounter a message that reads "Pseudo-terminal will not be allocated because stdin is not a terminal." Disregard this message. Without closing out of the first terminal, enter the second one and type the following, again in one line:

ssh -XCAt username@ash.ssec.wisc.edu ssh -XAt username@[servername].ssec.wisc.edu

In that second terminal window, type

jupyter lab --no-browser --port=[portnumber]

At the end of jupyter’s startup log, it should show something like:

http://localhost:5951/?token=[long_random_number]
or
http://127.0.0.1:5951/?token=[long_random_number]

Copy one of those (not the file one) and paste into your local web browser. Assuming all went well, you should see jupyter lab running on the SSEC server but displaying in your browser. More? I'll add more here as I think of more.