Build Ray on HAL POWER9 Machines

Because PowerPC builds of Ray are not available on conda, we must build Ray ourselves. Ray builds with GCC 8 (the current version available on HAL) have a bug that throws a segfault when trying to initialize the Ray cluster. Therefore, you must use at least GCC 9 for the build. However, because Ray uses Bazel which doesn’t operate well with non-standard compilers, you must build a customized version of Bazel with needed utilities statically linked. Then you can build Ray using GCC 9 properly.

1. Create a Bazel build conda environment.

   You must first create a conda environment for the build environment for Bazel.

   conda create python=3.8 -p ./bazel_venv
   conda activate ./bazel_venv
   

2. Build customized Bazel with statically linked utilities.

   Download distribution version of Bazel 4.2.1 (the version used in Ray version 1.9.2 and current development) and apply patch to allow statically linking utilities (bazel-4.22.1-static-utils.patch)

   wget https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-dist.zip
   mkdir bazel-4.2.1-dist
   cd bazel-4.2.1-dist
   unzip ../bazel-4.2.1-dist.zip
   
   patch -i ../bazel-4.2.1-static-utils.patch
   
   bash compile.sh
   

   Note

   The patch command may not be correct for how I created the patch. This is meant as an approximate guide, so you may have to correct some things to get them to work.

3. Prepare Ray build environment.

   Prepare another conda environment for building Ray as you may need different requirements from building Bazel.

   conda create python=3.8 -p ./ray_build_venv
   cp ./bazel-4.2.1-dist/output/bazel ./ray_build_venv/bin
   

4. Build Ray.

   First, activate GCC 9 and your Ray build environment. Then, checkout Ray and apply the needed patch for the stdc++fs library. (ray-fs.patch)

   conda activate ./ray_build_venv
   
   git clone [email protected]:ray-project/ray
   cd ray
   git checkout ray-1.9.2
   
   patch -i ../ray-fs.patch
   
   cd python
   python setup.py bdist_wheel
   

   The finished wheel will be in the directory python/dist.

   Once completed, use pip to install the ray-1.9.2 wheel in a python environment of your choice, however you must ensure GCC 9 is properly available in your environment.