Year: 2021
Type: preprint
Abstract: * Introduction To start, extract the artifact fvi-tacas25.tar.gz in the home directory: $ cd ~/ $ tar xvf fvi-tacas25.tar.bz2 This artifact contains a few scripts and archived versions of: - Oink, by Tom van Dijk, augmented with the algorithms described in the paper: https://github.com/michaelcadilhac/oink/tree/af40e4d These algorithms constitute our main software contribution. - mkplot, by Alexey Ignatiev, with some minor changes: https://github.com/michaelcadilhac/mkplot - A game benchmark library: https://github.com/michaelcadilhac/game-benchmarks/tree/8b7230e ** Dependencies All dependencies are automatically installed from the Internet by running: $ sudo ~/src/scripts/bootstrap.sh For completeness, we list here the dependencies installed by the bootstrapping script. - To compile the test solver, we need: Git, Boost, LibGMP, MPFRC++: $ apt install git libboost-all-dev libgmp-dev libmpfrc++-dev - To run the benchmarking scripts, we need: zsh: $ apt install zsh - To create the plots, we need: Python 3 and Matplotlib: $ apt install python-is-python3 python3-matplotlib * Smoke Test: Early light review In the smoke test, we suggest the reviewer compiles Oink as a parity game solver, tests that Oink can solve a game, runs a handful of our benchmark tests, and creates (uninteresting) plots based on these benchmarks. The script at: $ ~/src/scripts/smoke-test.sh ... does precisely that. In this section, we detail what is executed by this script. The reviewer can elect to simply copy/paste the instructions in the following subsections. ** Compiling the parity game solver This is a straightforward CMake-based project: $ cd ~/src/oink $ mkdir -p build_pg $ cd build_pg $ cmake .. -DCMAKE_BUILD_TYPE=Release -DOINK_BUILD_NATIVE=ON [...] $ make [...] ** Running the parity game solver We first run the parity game solver by hand, using the fast value iteration algorithm presented in the article. The solver is ran on an arbitrary game here. $ cd ~/src/oink/build_pg $ ./oink --fvi --no ~/src/game-benchmarks/parity-games/organic_low_prio/tlsf_based-test2.priomax=5.pg [ 0.00] parsing took 0.002968 sec. [ 0.00] parity game with 42 nodes and 90 edges. [ 0.00] parity game reindexed [ 0.00] solving using fast value iteration (asymmetric) [ 0.00] construct took 0.000097 sec. [ 0.00] run took 0.000028 sec. [ 0.00] preprocessing took 0.000000 sec. [ 0.00] solving took 0.000137 sec. [ 0.00] total solving time: 0.000145 sec. [ 0.00] current memory usage: 5.11 MB [ 0.00] peak memory usage: 5.68 MB To go a bit further, we will rely on a provided script to run a portion of the parity game benchmarks on several algorithms. More explanations on this script are provided in the Full Evaluation section. The following will take a minute. $ cd ~/src/oink/build_pg $ export PROPORTION=10000 # This will tell the script to run 1/10000 of all tests. $ ~/src/scripts/run-games.sh ~/src/game-benchmarks/parity-games/ ~/results-pg/raw fvi fvi-alt ** Plotting the results If the previous script "run-games.sh" was used, the raw logs are stored in ~/results-pg/raw. These can be converted to plots in a two step process: First convert the logs to a format readable by [[https://github.com/alexeyignatiev/mkplot][mkplot]], then run mkplot. Note that the plots obtained with very few tests are not instructive, but running this would allow to check that plots *can* be generated. The two steps correspond to two scripts: $ cd ~/results-pg $ ~/src/scripts/make-mkplottables.sh organic_high_prio organic_low_prio synthetic organic all $ export MKPLOTS_OPTS='-ymin=1e-4' $ ~/src/scripts/make-plots.sh organic_high_prio organic_low_prio synthetic organic all * Full Evaluation In the full evaluation, the reviewer will want to compile two flavors of Oink: One for parity games, and one for energy games. We implemented energy games in Oink by allowing unbounded weights on the states, and foregoing the translation Parity Game -> Energy Game. We also implemented strategy verification for energy games. ** Generating Energy Games This artifact is not shipped with all energy game benchmarks, nor are they available online, as they amount to more than 50GB of data. Note that the TACAS23 VirtualBox image has a disk size that is smaller than that. We provide, for this archive, 10% of our tests, and *we do not recommend* that the reviewer generates more. A script is nevertheless provided to generate all the benchmarks, if desired. These games are randomly generated, but the random generator seed is fixed, so the files produced will be exactly the same as the ones used in the paper. To generate the missing energy games, increase the disk size to about 75GB (here is a [[https://medium.com/@bonguides25/how-to-expand-or-increase-ubuntu-virtual-disk-in-virtualbox-e705e6a70398][tutorial]]) then run: $ cd ~/src/game-benchmarks/tools $ make # This will take about 5 minutes g++ -std=c++20 -Wall -O3 -march=native random-game-generator.cc -lm -lmpfr -lgmp -o random-game-generator $ cd ../energy-games $ ./generate-energy-games.sh # This will take about 30 minutes and generate 50 GB of games ** Compiling the solvers To compile the parity game solver, this is the same as before; to wit: $ cd ~/src/oink $ mkdir -p build_pg $ cd build_pg $ cmake .. -DCMAKE_BUILD_TYPE=Release -DOINK_BUILD_NATIVE=ON [...] $ make [...] To compile the energy game solver, we pass to cmake a different compilation directive: $ cd ~/src/oink $ mkdir -p build_nrg $ cd build_nrg $ cmake .. -DOINK_ENERGY_GAMES=ON -DCMAKE_BUILD_TYPE=Release -DOINK_BUILD_NATIVE=ON [...] $ make [...] ** Running the solvers To run the solvers on many instances,we provide the script ~/src/scripts/run-games.sh As arguments, it takes: - the folder in which all games are, - the folder in which logs should go, - the name of the algorithms to test. Note that the VM has, by default, around 7GB of RAM; the script was adjusted to kill tasks that take more than 5GB. If the *console* is killed, it may be that your VM has less RAM than this, in which case, please adjust the script to kill tasks at a lower RAM usage. To run the script on all tests and on all algorithms, use: $ cd ~/src/oink/build_pg # Go to the parity game solver $ ~/src/scripts/run-games.sh ~/src/game-benchmarks/parity-games/ ~/results-pg/raw fvi fvi-nfvi fvi-alt fvi-nfvi-alt qd zlk rtl tl osi $ cd ~/src/oink/build_nrg # Go to the energy game solver $ ~/src/scripts/run-games.sh ~/src/game-benchmarks/energy-games/ ~/results-nrg/raw fvi fvi-nfvi fvi-alt fvi-nfvi-alt qd osi Running these takes _several_ hours (around 7) on the full 50GB of games, and about 2 hours on the games provided. To only run, for testing purposes, one in every 100 tests, set, *beforehand:* $ export PROPORTION=100 ** Plotting the results This is a two step process: First convert the logs to a format readable by [[https://github.com/alexeyignatiev/mkplot][mkplot]], then run mkplot. These two steps correspond to two scripts: For parity game tests: $ cd ~/results-pg $ ~/src/scripts/make-mkplottables.sh organic_high_prio organic_low_prio synthetic organic all $ export MKPLOTS_OPTS='-ymin=1e-4' $ ~/src/scripts/make-plots.sh organic_high_prio organic_low_prio synthetic organic all For energy game tests: $ cd ~/results-nrg $ ~/src/scripts/make-mkplottables.sh dense-high dense-low sparse-high sparse-low dense sparse all $ export MKPLOTS_OPTS='-ymin=1e-2' $ ~/src/scripts/make-plots.sh dense-high dense-low sparse-high sparse-low dense sparse all All the plots are produced as ~/results-pg/*.pdf and ~/results-nrg/*.pdf
Source: arXiv (Cornell University)
Language: English
Subfield: Computational Theory and Mathematics
Field: Computer Science
Domain: Physical Sciences
Open Access status: green
OpenAlex ID: W4286904021