Python Environments
Python modules at TACC
TACC uses the Lmod module system to configure user environments.
On Frontera, the default user environment loads a module named TACC, which in turns loads a python module as part of the default action. At the time of this writing, Frontera loads module python3/3.7.0 through the TACC module.
On Frontera, there is an installed Python version 2.7.5 in /usr/bin/python. That is used by the operating system, and you should work with one python3 versions provided by Lmod. When you run python programs, you will want to refer to the executable as python3 instead of python. When in doubt, you can always run which python3 in the shell to figure out which version will be found in your path.)
If you want to run a version of python3 other than the default, run module avail and look for available versions of python within the list. On TACC systems, the environment variable TACC_PYTHON3_DIR is defined when a python3 module is loaded, as are other environment variables pointing to bin, lib, include, etc. subdirectories.
Many of the core python packages for scientific computing (e.g., numpy, scipy, matplotlib) are pre-installed on TACC systems as part of a python3 environment module, but python packages for deep learning, such as TensorFlow/Keras and PyTorch, are not pre-installed. Therefore, those packages must be installed locally by users. Generally this process involves using the pip package manager, which works to pull packages from the Python Package Index (PyPI). We provide below some general information on this process, and in subsequent pages, provide more specific instructions for particular TACC systems to enable you to run the example code discussed. While there are multiple strategies for locally installing packages, we recommend creating Python virtual environments, as described generally below.
The TACC Frontera User Guide provides some instructions on Machine Learning, with information on how to create and configure a Python virtual environment into which you can install the necessary software packages. The material presented here is intended to supplement and complement those basic instructions. The TACC software documentation also includes a page on TensorFlow at TACC containing some information on installing and running TensorFlow, but that material is outdated and suggests installing software into one's home directory using the pip --user option, as opposed to the currently recommended process of creating a Python virtual environment.
Python virtual environments
For python3, the venv module — part of the Python Standard Library — is used to create virtual environments. One or more virtual environments can be created, with packages installed into them, and any one of those environments can be activated within a shell at any time. To create an environment named "myenv" and install a few packages, you would do something like the following: