JavaDL Artifact Introduction JavaDL is a language and toolchain for analyzing Java at the source level. JavaDL is based on MetaDL, a variant of Datalog that adds syntactic pattern matching, here specialized to pattern matching on Java. This Artifact Overview guide describes the VM image that we provide for artifact evaluation: Getting Started Step-by-step instructions for using JavaDL and for reproducing our results Claims from the paper and how our artifact supports them Getting Started Guide For the following steps, we assume a typical modern machine with a recent multi-core CPU and 64 GiB of RAM. The process may take more than 30 minutes on older machines, but most of this will be automated. The necessary steps are: Installing Docker Installing and Starting Docker Image Basic Testing Advanced Testing (optional) Caveats Please note that we have only tested the process below on Ubuntu and Debian Linux. The same process may or may not work for OS X or Windows, though we are hopeful that the Docker website provides sufficient guidance to run the "Install Docker" and "Install and Start Docker Image" steps from these platforms, too. To the best of our understanding of Docker, later steps should be independent of the host platform. Please note also that our terminology in the paper diverges subtly from the terminology in our code. The most important differences are the following: Since JavaDL is an instance of MetaDL, our code base (and VM image) often use "MetaDL" to describe the language that the paper calls "JavaDL". In the paper, we describe the "exhaustive" evaluation strategy (also known as "one-shot" or "non-incremental"). In the code base, we often refer to this strategy as "single evaluation". Parts of the code and documentation also use the term "hybrid evaluation" (referring to hybrid Datalog execution in both Soufflé and our own system). Installing Docker We are providing the JavaDL artifact as a Docker image. To run such an image, make sure to install the relevant tools: For Windows and OS X systems, follow the guidelines on the Docker desktop download site On Linux-based systems, install the docker command line tool. This tool may be provided by the docker.io and/or docker-ce packages. If your distribution does not provide these packages, follow the steps here: For Ubuntu For Debian For CentOS For Fedora Users of other distributions can download pre-compiled binaries or build Docker from source (both "cli" and "engine") Starting Docker Download the Docker image from https://doi.org/10.5281/zenodo.5090140 onto your machine. Open a command shell in the directory into which you downloaded the image. Make sure that you are on a command shell in the directory into which you have downloaded the image Check the integrity of the docker image (this may take a minute or two): sha256sum javadl-oopsla21.tgz should report 49aa9fd6797095f9a33e6013fb582a7cfa1434d58826bcd82dd5df0706f65ad6 md5sum javadl-oopsla21.tgz should report b5892a1bc2579f7be220f08c0064fb3a Install the docker image: docker load -i ./javadl-oopsla21.tgz Start the docker image: docker run -it javadl:oopsla21 This should produce the following prompt (except that the hex number after 'root@' may vary): root@1114245ce565:/# Basic Testing: Bug Checker Output Comparison Once you are logged into the docker image, the fastest way to test our system is to run the following: cd /work ./run_quality.bash This will set up and run JavaDL, SpotBugs, and Error Prone on the benchmarks that we report on in the paper, and print out the bug comparison table (Fig. 13). On a modern machine, this should take around 15 minutes. The output should look something like this (you can ignore the readlink warning): root@1114245ce565:/work# ./run_quality.bash /work/javadl-eval-1 /work readlink: missing operand Try 'readlink --help' for more information. >>> Running the static checkers on the fixed versions from the Defects4j <<< >>> Timing MetaDL-Hybrid >>> Timing SpotBugs >>> Timing ErrorProne >>> Parsing and serializing output from the static checkers <<< #MDL #SB #Common Precision Recall DM_NUMBER_CTOR & 429 & 182 & 162 & 37.76 & 89.01 DM_STRING_CTOR & 5 & 5 & 5 & 100.00 & 100.00 EXPOSE_REP & 3546 & 161 & 108 & 3.05 & 67.08 NM_CLASS_NAMING_CONVENTION & 0 & 0 & 0 & -1.00 & -1.00 NM_FIELD_NAMING_CONVENTION & 874 & 0 & 0 & 0.00 & -1.00 NM_METHOD_NAMING_CONVENTION & 1120 & 38 & 38 & 3.39 & 100.00 SF_SWITCH_NO_DEFAULT & 223 & 87 & 81 & 36.32 & 93.10 UWF_UNWRITTEN_FIELD & 3 & 1 & 0 & 0.00 & 0.00 UWF_UNWRITTEN_PUBLIC_OR_PROTECTED_FIELD & 3 & 0 & 0 & 0.00 & -1.00 MDL vs. SB Total SB: 474 Total MDL: 6206 Common: 395 MDL only: 5808 SB only: 80 #################################################################################################### #MDL #EP #Common Precision Recall BoxedPrimitiveConstructor & 429 & 425 & 423 & 98.60 & 99.53 MissingOverride & 4533 & 5341 & 4393 & 96.91 & 82.25 OperatorPrecedence & 84 & 100 & 82 & 97.62 & 82.00 ReferenceEquality & 1176 & 1235 & 1169 & 99.40 & 94.66 TypeParameterUnusedInFormals & 99 & 95 & 95 & 95.96 & 100.00 UnnecessaryParentheses & 257 & 174 & 134 & 52.14 & 77.01 MDL vs. EP Total EP: 7426 Total MDL: 6578 Common: 6297 MDL only: 281 EP only: 1074 #################################################################################################### /work The output at the end contains the numbers for Figure 13. Advanced Testing Optionally, you can manually build and run JavaDL and its test suite. This is not necessary to reproduce our results, but helpful if you want to have a closer look at the system or extend JavaDL. The following steps will re-run the test suite (all 136 tests should pass): cd /work/metadl ./gradlew test Please refer to /work/metadl/README.org for more details on manually running JavaDL. Step by Step Instructions In this section we describe how to reproduce our performance results and how to manually run JavaDL with custom queries. Re-running our Benchmarks To re-run our benchmarks, run the following in your docker image: cd /work ./run_performance.bash Depending on your machine, it may take over a day to re-run all benchmarks, so we recommend running on a dedicated benchmarking machine (using e.g. tmux or screen to be able to disconnect safely). After running, the output should be similar to the following (concrete numbers may vary greatly depending on the underlying hardware): proj object_file_generate_relations_total object_file_compile_total hybrid_program_input ... sema_and_type_check general_overhead incremental_driver_overhead update_local_relations 0 d4j-Cli 0.293539 0.056372 0.000000 ... 0.517667 0.091731 0.000000 0.000000 1 d4j-Cli 0.138508 0.032728 0.022318 ... 0.390337 0.123824 0.082171 0.184927 2 d4j-Codec 0.388758 0.074782 0.000000 ... 0.400047 0.080707 0.000000 0.000000 3 d4j-Codec 0.135542 0.037518 0.054627 ... 0.407575 0.132363 0.085168 0.113422 4 d4j-Compress 0.460538 0.063568 0.000000 ... 0.297225 0.103416 0.000000 0.000000 5 d4j-Compress 0.202481 0.030771 0.062943 ... 0.309277 0.098028 0.072345 0.186635 6 d4j-Csv 0.279070 0.055226 0.000000 ... 0.533767 0.090738 0.000000 0.000000 7 d4j-Csv 0.185438 0.037808 0.023777 ... 0.397894 0.129165 0.090716 0.105950 8 d4j-JacksonCore 0.490699 0.060332 0.000000 ... 0.289425 0.086140 0.000000 0.000000 9 d4j-JacksonDatabind 0.662087 0.037427 0.000000 ... 0.125948 0.094318 0.000000 0.000000 10 d4j-JacksonXml 0.309379 0.046775 0.000000 ... 0.494052 0.110116 0.000000 0.000000 11 d4j-JacksonXml 0.149556 0.031302 0.024723 ... 0.390042 0.122664 0.097267 0.162492 12 d4j-Jsoup 0.392209 0.063594 0.000000 ... 0.412381 0.080244 0.000000 0.000000 13 d4j-Jsoup 0.267734 0.032205 0.024790 ... 0.268394 0.087463 0.060649 0.218730 14 d4j-Math 0.336936 0.057061 0.000000 ... 0.448329 0.110705 0.000000 0.000000 15 d4j-Math 0.097620 0.032702 0.039766 ... 0.399697 0.129623 0.084926 0.189194 16 d4j-Mockito 0.498430 0.037437 0.000000 ... 0.324475 0.096073 0.000000 0.000000 17 d4j-Mockito 0.371869 0.014380 0.017237 ... 0.165659 0.053631 0.040926 0.318487 18 d4j-Time 0.549664 0.067943 0.000000 ... 0.208878 0.073654 0.000000 0.000000 19 d4j-Time 0.380325 0.024303 0.033737 ... 0.145743 0.047100 0.035855 0.274611 [20 rows x 11 columns] ['proj' 'run_kind' 'object_file_generate_relations_total' 'object_file_compile_total' 'hybrid_program_input' 'hybrid_program_run' 'hybrid_program_output' 'incremental_driver_init' 'sema_and_type_check' 'update_local_relations' 'incremental_driver_overhead' 'general_overhead'] [ 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19] python/src/plot_change_vs_analyzed.py:123: RuntimeWarning: invalid value encountered in long_scalars all_data["n_files_rel"] = all_data.apply(lambda r : r["n_files"] / max_n_files[(r["proj"], r["run_kind"])], axis=1) python/src/plot_change_vs_analyzed.py:131: RuntimeWarning: invalid value encountered in long_scalars max_n_touched_files[(r["proj"], r["run_kind"])], axis=1) [0 1 2 3 4 5 6 7] [0 1 2] python/src/plot_change_vs_analyzed.py:54: RuntimeWarning: More than 20 figures have been opened. Figures created through the pyplot interface (`matplotlib.pyplot.figure`) are retained until explicitly closed and may consume too much memory. (To control this warning, see the rcParam `figure.max_open_warning`). _, axs = plt.subplots(1, 2, figsize=(8, 4), gr