Python on HAL
Open Cognitive Environment (modules opence/1.0.0 to opence/1.6.1)
The Open-CE project contains everything that is needed to build conda packages for a collection of machine learning and deep learning frameworks.
Open-CE environment packages (click to expand/collapse)
Package |
opence v1.6.1 |
opence v1.5.1 |
opence v1.3.1 |
opence v1.2.2 |
opence v1.1.2 |
opence v1.0.0 |
|---|---|---|---|---|---|---|
python |
3.9.12 |
3.9.12 |
3.8.0 |
3.8.0 |
3.8.12 |
3.8.12 |
cuda |
11.2.2 |
11.2.2 |
11.2.2 |
11.0.221 |
10.2.89 |
10.2.89 |
cudnn |
8.1.1 |
8.1.1 |
8.1.1 |
8.1.1 |
7.6.5 |
7.6.5 |
nccl |
2.12.7 |
2.11.4 |
2.8.3 |
2.7.8 |
2.7.8 |
2.7.8 |
openmpi |
4.1.1 |
4.1.1 |
4.1.1 |
3.1.3 |
3.1.3 |
3.1.3 |
apex |
0.1 |
0.1 |
0.1 |
N/A |
N/A |
N/A |
hdf5 |
1.10.6 |
1.10.6 |
1.10.4 |
1.10.6 |
1.10.4 |
1.10.4 |
horovod |
0.24.2 |
0.23.0 |
0.21.3 |
0.21.0 |
0.21.0 |
0.19.5 |
ipython |
7.32.0 |
7.32.0 |
7.27.0 |
7.29.0 |
7.29.0 |
7.28.0 |
matplotlib |
3.5.2 |
3.4.3 |
3.4.3 |
3.4.3 |
3.4.3 |
3.4.2 |
mpi4py |
3.1.3 |
3.1.3 |
3.1.1 |
N/A |
N/A |
N/A |
numpy |
1.22.3 |
1.20.3 |
1.21.2 |
1.19.5 |
1.19.5 |
1.19.2 |
onnx |
1.11.0 |
1.10.2 |
1.7.0 |
1.6.0 |
1.6.0 |
1.6.0 |
opencv |
4.5.5 |
4.5.3 |
3.4.14 |
4.5.0 |
3.4.10 |
3.4.10 |
pandas |
1.4.2 |
1.4.2 |
1.3.2 |
1.3.4 |
1.3.4 |
1.2.4 |
pyarrow |
7.0.0 |
5.0.0 |
N/A |
N/A |
N/A |
N/A |
pytorch |
1.10.2 |
1.10.1 |
1.8.1 |
1.7.1 |
1.7.1 |
1.6.0 |
scikit-learn |
1.0.2 |
1.0.2 |
0.24.2 |
1.0.1 |
1.0.1 |
1.0.1 |
scipy |
1.7.3 |
1.7.3 |
1.7.1 |
1.7.1 |
1.4.1 |
1.4.1 |
ray |
1.11.1 |
N/A |
N/A |
N/A |
N/A |
N/A |
tensorflow |
2.8.1 |
2.7.0 |
2.5.1 |
2.4.1 |
2.4.1 |
2.3.1 |
tensorboard |
2.8.0 |
2.7.0 |
2.5.0 |
2.4.1 |
2.4.1 |
2.3.0 |
transformers |
4.16.2 |
4.16.2 |
4.4.2 |
2.1.1 |
4.12.2 |
4.12.2 |
IBM Watson Machine Learning Community Edition (modules wmlce/1.7.0 and wmlce/1.6.2)
WMLCE is an enterprise software distribution that combines popular open-source deep learning frameworks, efficient AI development tools, and accelerated IBM Power Systems servers.
WMLCE environment packages (click to expand/collapse)
Package |
wmlce/1.7.0 |
wmlce/1.6.2 |
|---|---|---|
python |
3.7.10 |
3.7.10 |
cuda |
10.2.89 |
10.1.243 |
cudnn |
7.6.5 |
7.6.3 |
nccl |
2.5.6 |
2.4.8 |
apex |
0.1.0 |
0.1.0 |
caffe |
1.0 |
1.0 |
dask |
2.9.2 |
2.3.0 |
hdf5 |
1.10.2 |
1.10.2 |
horovod |
0.19.0 |
N/A |
ipython |
7.26.0 |
7.26.0 |
matplotlib |
3.3.4 |
3.3.4 |
numba |
0.47.0 |
0.45.1 |
numpy |
1.17.4 |
1.16.6 |
onnx |
1.6.0 |
1.5.0 |
pandas |
1.2.4 |
1.2.4 |
pytorch |
1.2.4 |
1.2.0 |
scikit-learn |
0.24.2 |
0.22.1 |
scipy |
1.3.1 |
1.3.1 |
spectrum-mpi |
10.03 |
10.03 |
tensorboard |
2.1.1 |
1.15.0 |
tensorflow |
2.1.3 |
1.15.5 |
Customize a Python Environment
Miniconda is a small, bootstrap version of Anaconda that includes only conda, Python, the packages they depend on, and a small number of other useful packages, including pip and zlib.
Use the conda install command to install 720+ additional conda packages from the Anaconda repository.
Install a Specific Package
You cannot (and should not) install packages in existing python environments.
You need to create your own python environment to install your own packages.
You should search for all the available packages before installation.
You can install a specific python package to avoid conflict or add functionality.
Solve a Dependency Conflict
Conda can solve some dependency conflicts, but it takes time and is not guaranteed to work. Instead, you can:
Create a python environment from scratch.
Install the package that caused the conflict.
Install other packages according to the conflicting package requirement.
Use a Customized Python Environment in HAL OnDemand
Run conda install ipykernel to select and/or switch your conda environment in HAL OnDemand.
Existing Anaconda Environments
Existing Anaconda environments (click to expand/collapse)
Environment Name |
Location |
Description |
|---|---|---|
base |
/opt/miniconda3 |
Default conda env with basic python packages. |
jupyter |
/opt/miniconda3/envs/jupyter |
Default conda env with jupyter server. Used for hal-ondemand service only. |
deepspeed-v0.3.16 |
/opt/miniconda3/envs/deepspeed-v0.3.16 |
DeepSpeed is a deep learning optimization library that makes distributed training easy, efficient, and effective. |
fastai-v0.1.18 |
/opt/miniconda3/envs/fastai-v0.1.18 |
fastai is a deep learning library that provides practitioners with high-level components that can quickly and easily provide state-of-the-art results in standard deep learning domains. |
wmlce-v1.6.2 |
/opt/miniconda3/envs/wmlce-v1.6.2 |
Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. This is the only environment still have tensorflow-v1.15. |
wmlce-v1.7.0 |
/opt/miniconda3/envs/wmlce-v1.7.0 |
Watson Machine Learning Community Edition is an IBM Cognitive Systems offering that is designed for the rapidly growing and quickly evolving AI category of deep learning. This is the last version of WMLCE, the following module is opence. |
opence-v1.0.0 |
/opt/miniconda3/envs/opence-v1.0.0 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.1.2 |
/opt/miniconda3/envs/opence-v1.1.2 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.2.2 |
/opt/miniconda3/envs/opence-v1.2.2 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.3.1 |
/opt/miniconda3/envs/opence-v1.3.1 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.4.1 |
/opt/miniconda3/envs/opence-v1.4.1 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
opence-v1.5.1 opence-v1.5.2 |
/opt/miniconda3/envs/opence-v1.5.1 /opt/miniconda3/envs/opence-v1.5.2 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. This is the default python environment. Use opence-v1.5.1 as base, fixed openblas issue caused by openmp. |
theano-v1.0.4 |
/opt/miniconda3/envs/theano-v1.0.4 |
Open-CE is a community-driven software distribution for machine learning that runs on standard Linux platforms with NVIDIA GPU technologies. |
rapdis-v0.11.0 |
/opt/miniconda3/envs/rapids |
The RAPIDS suite of software libraries, built on CUDA-X AI, gives you the freedom to execute end-to-end data science and analytics pipelines entirely on GPUs. |
Create a New Environment from Existing Environments
It is recommended to create a new environment from one of the existing opence environments. To create a new environment (new_env) from an existing environment (opence-v1.5.1):
conda create --name=new_env --clone=opence-v1.5.1
The new conda environment will be in $HOME/.conda/envs/new_env.
You can now search and/or install python packages via conda in your new environment.
For example, to search for the tensorflow package:
conda search tensorflow
Create a Conda Environment from Scratch
You can also create a new environment from scratch. To create a new environment (new_env_name) from scratch:
conda create --name=new_env_name
Search Packages in All Default Channels
conda search openblas
Search Packages in a Specific Channel
conda search openblas -c conda-forge
Install a Package with a Specific Version and Build Code
Install a package from the result of the search
(new_env_name)[user_id@hal-login2 ~]# conda search openblas
...
openblas 0.3.12 pthreads_hca0ad1f_0 conda-forge
openblas 0.3.12 pthreads_hca0ad1f_1 conda-forge
openblas 0.3.13 h6ffa863_0 pkgs/main
openblas 0.3.13 h6ffa863_1 pkgs/main
openblas 0.3.13 openmp_h25a920f_0 conda-forge
openblas 0.3.13 pthreads_h92053e5_0 conda-forge
...
(new_env_name)[user_id@hal-login2 ~]# conda install openblas=0.3.13=openmp_h25a920f_0