Lab 1 Python Setup and Install
Aims
The aim of this lab is to help setup and install python and various other tools
- Using PyEnv for installation of python versions
- Using Pip to install packages
- Setup VSCode to edit and run python files
Why do I need PyEnv?
At present the version of python installed in our linux labs are
ls -al /usr/bin/python* /usr/bin/pip*
lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip-2 -> ./pip2
-rwxr-xr-x. 1 root root 206 Aug 19 2020 /usr/bin/pip2
lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip-2.7 -> ./pip2
lrwxrwxrwx. 1 root root 6 Aug 19 2020 /usr/bin/pip2.7 -> ./pip2
lrwxrwxrwx. 1 root root 23 May 26 2021 /usr/bin/pip-3 -> /etc/alternatives/pip-3
lrwxrwxrwx. 1 root root 22 May 26 2021 /usr/bin/pip3 -> /etc/alternatives/pip3
lrwxrwxrwx. 1 root root 8 Jun 18 2021 /usr/bin/pip-3.6 -> ./pip3.6
-rwxr-xr-x. 1 root root 209 Jun 18 2021 /usr/bin/pip3.6
-rwxr-xr-x. 1 root root 25440 Jun 23 2020 /usr/bin/pipewire
-rwxr-xr-x. 1 root root 437160 Jun 23 2020 /usr/bin/pipewire-media-session
lrwxrwxrwx. 1 root root 9 Aug 11 2021 /usr/bin/python2 -> python2.7
-rwxr-xr-x. 1 root root 8016 Aug 11 2021 /usr/bin/python2.7
lrwxrwxrwx. 1 root root 25 May 26 2021 /usr/bin/python3 -> /etc/alternatives/python3
lrwxrwxrwx. 1 root root 31 Aug 11 2021 /usr/bin/python3.6 -> /usr/libexec/platform-python3.6
lrwxrwxrwx. 1 root root 32 Aug 11 2021 /usr/bin/python3.6m -> /usr/libexec/platform-python3.6m
You will notice that some of these are actually symbolic links to other versions. Most of these version of python are essential to the OS and tools that ship with RHEL and as a standard user you don’t have permission to pip install unless you use the -u flag. This can be problematic especially if we want a newer version of python or a different version such as anaconda.
PyEnv
One of the simplest methods I have discovered to manage the different versions of python is the use of PyEnv.
pyenv lets you easily switch between multiple versions of Python. It’s simple, unobtrusive, and follows the UNIX tradition of single-purpose tools that do one thing well.
PyEnv allows you to change both the Global default python version as well as specify specific per folder local versions.
How it works
PyEnv basically uses a system called shims to intercept python calls in your path and select the correct python version. These are added to your path in the .bashrc (or other shells), VSCode will also pick these up.
Installing PyEnv (Linux and Mac)
PyEnv can be installed in numerous ways but as we have all the development tools installed on the lab machines we can use the GitHub checkout method.
git clone https://github.com/pyenv/pyenv.git ~/.pyenv
Next we will build the bash shell extensions which will speed things up.
cd ~/.pyenv && src/configure && make -C src
We now need to setup our shell to point to the installed pyenv. To do this we add the following to our .bashrc file.
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"
Once this has been added we can activate it in the terminal
source ~/.bashrc
We should now be able to run the following to see what version of python we are currently using.
pyenv global
system
PyEnv Commands
The pyenv command has build in help
pyenv --help
Usage: pyenv <command> [<args>]
Some useful pyenv commands are:
commands List all available pyenv commands
exec Run an executable with the selected Python version
global Set or show the global Python version(s)
help Display help for a command
hooks List hook scripts for a given pyenv command
init Configure the shell environment for pyenv
install Install a Python version using python-build
local Set or show the local application-specific Python version(s)
prefix Display prefixes for Python versions
rehash Rehash pyenv shims (run this after installing executables)
root Display the root directory where versions and shims are kept
shell Set or show the shell-specific Python version
shims List existing pyenv shims
uninstall Uninstall Python versions
--version Display the version of pyenv
version Show the current Python version(s) and its origin
version-file Detect the file that sets the current pyenv version
version-name Show the current Python version
version-origin Explain how the current Python version is set
versions List all Python versions available to pyenv
whence List all Python versions that contain the given executable
which Display the full path to an executable
See `pyenv help <command>' for information on a specific command.
For full documentation, see: https://github.com/pyenv/pyenv#readme
We can list all the python version available using
pyenv install --list
As you can see this is a long list, best way to shorten this is to filter the output via grep
pyenv install --list | grep " 3\.1[01]"
3.10.0
3.10-dev
3.10.1
3.10.2
3.10.3
3.10.4
3.10.5
3.10.6
3.11.0rc1
3.11-dev
You will notice that this has some very new dev versions as well as the 10 and 11 point versions. It is usually best to install the non dev versions. For now we will install 3.10.6 but if we are working with DCC tools it is best to install the VFX Reference Platform Version which is 3.10.x for 2023 but 3.9.x for last year (which is what most of our tools use).
pyenv install 3.10.6
This will download, build and install CPython 3.10.6, this may take some time as it is basically building the python and pip executables from source code.
pyenv versions
* system (set by /home/jmacey/.pyenv/version)
3.10.6
This shows at present the default python is the system one to set the new global version we can do the following
pyenv global 3.10.6
pyenv versions
system
* 3.10.6 (set by /home/jmacey/.pyenv/version)
python -V
Python 3.10.6
Local Python
We can set a per folder python version using the pyenv local
command.
mkdir PyEnvTest
cd PyEnvTest
pyenv local system
This will set the version of python to the system one, however in this case it is called python3 not python as before. The following example is still in the directory above.
python3 -V
Python 3.6.8
python -V
pyenv: python: command not found
The `python' command exists in these Python versions:
3.10.6
Note: See 'pyenv help global' for tips on allowing both
python2 and python3 to be found.
cd ..
python3 -V
Python 3.10.6
python -V
Python 3.10.6
python -m test
this will take some time but tests the full installation as well as the standard modules.Using pip
PIP is a package manager for Python packages (modules). In the case of pyenv the version of pip used will work for the current active version.
All the modules and libraries installed with pip will reside in your ~/.pyenv/versions/[current active]
folder. In particular in the ~/.pyenv/versions/[current version]/lib/python[current version]/site-packages
folder.
Some basic packages
We are going to start by installing some basic packages to make our life easier.
black
Black is a tool we can run on our code to auto format it to the correct pep8 style.
pip install black
You may see something like this in your messages
[notice] A new release of pip available: 22.2.1 -> 22.2.2
[notice] To update, run: pip install --upgrade pip
It is best to never upgrade pip as it can cause massive issues.
We will see black in action later, but first we can test it by running black
as the tool is placed in the shims folder.
isort
isort your imports, so you don’t have to. This will just take all the import
statements and sort them alphabetically and automatically separated into sections and by type
pip install isort
isort
MyPy
MyPy is a tool to help check types in your code for mistakes. ( Types in python you ask? More in a later lecture / lab).
pip install mypy
mypy
usage: mypy [-h] [-v] [-V] [more options; see below]
[-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Missing target module, package, files, or command.
A note of pip
All the modules we have installed so far are installed for this version of python. If you install another version of python they will not be available. So for each new install the previous modules, if required, will need to be re-installed.
This could be tedious however pip has a way of managing this for use.
pip freeze
When installing with pip it generates a manifest of the modules installed. We can list them using the following command.
pip freeze
black==22.8.0
click==8.1.3
isort==5.10.1
mypy==0.971
mypy-extensions==0.4.3
pathspec==0.10.0
platformdirs==2.5.2
tomli==2.0.1
typing_extensions==4.3.0
We can pipe the output of this to a file called requirements.txt and then re-use it.
pip freeze > requirments.txt
# in a new version
pip install -r requirements.txt
This will install all of the modules / packages installed for the system which is not always what you would need for a project. So there are tools such as pipreqs which will only use the imports of a project. This is really beyond the scope of what we are doing now but is useful for future projects.
virtual env
Python allows us to create a new clean sandbox to test and run code and manage dependencies. We have already seen how pip can be used to install packages, however pip can also cause conflicts especially when we start to specify version of modules and libraries.
A virtual environment allows us a clean folder with it’s own install so we can mange things better. This is often seen as the preferred way to manage python projects (although docker has become popular for this now as well).
How does this differ from pyenv
We can see pyenv as being a way of installing and testing against different python versions. Virtual Env will allow us to create clean installs for the current installed python version.
creating a venv
python -m venv TestEnv
source bin/activate
python -V
Python 3.10.6
pip list
Package Version
---------- -------
pip 22.2.1
setuptools 63.2.0
Note the packages such as black, isort and mypy are now not installed in this sandbox.
To deactivate the current environment we need to call deactivate
in the terminal.
More Reading
This article has some in depth information about how pyenv works