Benchmark generation

Following are the steps to generate the benchmark results used in the paper.

Reproducing complete benchmark paper results:

Run simulation:

  1. Check out to legacy branch.

  2. Download data.gz from git-lfs, and unzip it to data folder. (Or generate it using data/input/generate_data.ipynb)

  3. Go src folder and run python main.py --method=ALL --start=0 --end=38400.

Both main and legacy branches use the same logic (models & algorithms). However, we have refined the organization in the main branch by introducing a unified interface and standardized type notation to enhance maintainability and simplify the efforts to add new methods. The legacy branch houses the source code record used in the paper.

Note: As described in the paper, the benchmark results are generated on Chameleon Cloud with 8 nodes equipped with 2 AMD EPYC® CPUs, 64 cores per CPU. By setting the timeout parameter to 2 hours, it may take more than a week to complete 38400 * 17 experiments with the same environment. Therefore, attempting to replicate these results on a single local machine would be exceedingly time-intensive.

We provide the generated benchmark results in the data folder along with the visualization source code. By running the visualization code, you can reproduce the figures in the paper.

Small scale benchmark demo:

To quickly demonstrate the benchmark results, we provide a small scale benchmark with 256 experiments. Note some algorithms have solver dependencies, see getting started section.

Run the following command to generate the benchmark results for all algorithms:

```
python -m tsnkit.test.benchmark --methods ALL --ins 1-256
```

The estimated time to generate results for one algorithm, such as ls is 30 minutes. When running all the algorithms, results generation may take over 2 days.

The default time limit for each test case for each algorithm is 10 minutes. For faster benchmark results, the time limit can be lowered:

```
python -m tsnkit.test.benchmark --methods ALL --ins 1-256 -t 60
```

Debug tool

The debug module can be used to run and validate methods across 48 test cases. To validate individual modules, the following command can be used:

```
python -m tsnkit.test.debug.at -t 60
```

In the above command, “at” represents one of the scheduling methods and be interchanged with other methods. This tool will take less than 1 hour to validate one method across all 48 cases. To validate multiple modules, the following command can be used, with each desired method name separated by a space:

```
python -m tsnkit.test.debug at cg cp_wa dt -t 60
```

Results for each algorithm are written to a file in the results directory of the debug folder containing information on the runtime, memory usage, error messages (if any), and the simulation log for each test case.