Get Started with Open Cognitive Environment (Open-CE)

Open Cognitive Environment

The Open-CE project contains everything needed to build conda packages for a collection of machine learning and deep learning frameworks.

See Python on HAL for a detailed list of package versions in each environment version.

Simple Example with TensorFlow

Interactive mode

Get a node for interactive use:
```
swrun -p gpux1
```
Once on the compute node, load the PowerAI module using one of these:
```
module load opence
```
```
module load opence-v1.3.1
```

Copy the following code into file mnist-demo.py:

import tensorflow as tf
mnist = tf.keras.datasets.mnist
(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(512, activation=tf.nn.relu),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation=tf.nn.softmax)
])
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Train on MNIST with keras API:
```
python ./mnist-demo.py
```

Batch mode

The same can be accomplished in batch mode using the following tf_sample.swb script:

wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tf_sample.swb
sbatch tf_sample.swb
squeue

Visualization with TensorBoard

Interactive mode

Get a node for interactive use:
```
swrun -p gpux1
```
Once on the compute node, load PowerAI module using one of these:
```
module load opence
module load opence-v1.3.1
```

Download the code mnist-with-summaries.py to your $HOME folder:

cd ~
wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/mnist-with-summaries.py

Train on MNIST with TensorFlow summary:
```
python ./mnist-with-summaries.py
```

Batch mode

The same can be accomplished in batch mode using the following tfbd_sample.swb script:

wget https://wiki.ncsa.illinois.edu/download/attachments/82510352/tfbd_sample.swb
sbatch tfbd_sample.swb
squeue

Start the TensorBoard session

After a job has completed, the TensorFlow log files can be found in ~/tensorflow/mnist/logs. Start the TensorBoard server on HAL OnDemand, for detail go to HAL OnDemand - TensorBoard.

Simple Example with PyTorch

Interactive mode

Get a node for interactive use:
```
swrun -p gpux1
```
Once on the compute node, load PowerAI module using one of these:
```
module load opence
module load opence-v1.3.1
```

Install samples for Pytorch:

pytorch-install-samples ~/pytorch-samples
cd ~/pytorch-samples

Train on MNIST with Pytorch:
```
python ./examples/mnist/main.py
```