Skip to content
BranchesTags

Repository files navigation

TaintCC - Testing Static Taint Analyzers with Equivalence Modulo Taint

This repository contains the implementation of TaintCC, a framework for testing taint analyzers by generating programs equivalent with respect to source-sink flows. The methodology is described in the paper:

M. Christakis, A. Isychev, S. Pilz, F. Tesarek and V. Wüstholz, Testing Static Taint Analyzers with Equivalence Modulo Taint, ISSTA'26, Oct. 03, 2026.

This README describes the setup for running TaintCC, for instructions on how to reproduce results from the paper, got to the Experiments section at the bottom.

This implementation includes support for testing the following analyzers:

  • FlowDroid
  • MarianaTrench
  • Pysa
  • Semgrep for Python and Java. Requires a Semgrep user account and a token to log in with the CLI (see helper scripts for example usage).

Run using Podman (similar for Docker)

  • Build image: podman build -t taintcc .
  • Start container: 
    podman container run -d --rm \
    --name taintcc  \
    --mount type=bind,source="${LOCAL-LOGS-FOLDER}",target=/taintcc/logs  \
    -e ROUNDS=10000 \
    -e ITERATIONS=10  \
    -e INITIAL_STATEMENTS_TO_GENERATE=8 \
    -e TESTING_MODE='metamorphic' \
    -e TAT_TO_TEST="FD" \
    taintcc
    The following parameters can be adapted:
    • LOCAL-LOGS-FOLDER add the path of the folder on your local machine to write logs and the source code of found issues to
    • ITERATIONS Number of metamorphic or differential testing iterations to be carried out in one round, i.e. before resetting the original program T_og
    • ROUNDS the total number of tests is bounded by ROUNDS x ITERATIONS
    • INITIAL_STATEMENTS_TO_GENERATE Number of statements generated in for the "original" program at the beginning of each round
    • STATEMENT_GENERATION_STEP Increase of generated statements each round
    • TAT_TO_TEST Static Taint Analysis tool(s) that are tested, only one for metamorphic and time-to-no-bug modes (specify multiple for the differential mode,separated by comma, no spaces)
    • TESTING_MODE one of 3 available testing modes: differential, metamorphic, time-to-no-bug, gen-seeds
    • BUGGY_SEED ID of the program that revealed the bug, required for time-to-no-bug experiment (refinds existing issue). Possible values (for more details see Detected issues below): - for FlowDroid: FD763, FD764, FD767, FD779, FD781 - for Pysa: PY923, PY964, PY977 - for MarianaTrench: MT172, MT173, MT176, MT174, MT179
    • RSEED random seed for deterministic results, by default set to 777

Manual setup on Linux/WSL

If you're using Windows and WSL isn't yet setup:

On Linux/WSL:

  • Install Python
  • Install Java sudo apt install openjdk-17-jdk
  • Install Android SDK 
    sudo apt update
sudo apt install android-sdk -y
sudo apt install sdkmanager -y
sdkmanager --licenses
sdkmanager "platforms;android-34" "build-tools;34.0.0"

Copy .env.example to .env and provide the needed config.

Setup for MarianaTrench

The source/sink definitions for FlowDroid and Pysa are generated on the go whilst running the tests. For Mariana Trench they need to be preconfigured with the following steps:

  • Copy the files custom.models and propagation.models (located in fpc-files/MT-Files) to mariana-trench config folder (in venv, normally located at \home\{username}\.venvs\{venv-name}\share\mariana-trench\configuration\model-generators, without venv at \usr\local\share\mariana-trench\configuration\model-generators) and rename it to {filename}.models
  • Add the model generator name (i.e, the file name) in the JSON configuration file (in venv, normally located at \home\{username}\.venvs\{venv-name}\share\mariana-trench\configuration\default_generator_config.json).
  • Make sure the file contains all possibly used sources and sinks

Troubleshooting

Android SDK not found

Make sure the sdk.dir is set in local.properties file in ./Android-App/Testapp/. Usually the android SDK can be found at /usr/lib/android-sdk/, therefore set sdk.dir=/usr/lib/android-sdk/.

Licences missing

Somehow, sometimes the accepted licenses are stored at \home\{USERNAM}\android\licenses but need to be present at \usr\lib\android-sdk\licenses. Just copy them using sudo cp -a /home/{USERNAME}/android/licenses/. /usr/lib/android-sdk/licenses/.

Failed to install SDK components

Same as with licenses, just copy the platforms and platform-tools folders.

Gradlew build not working

Error message: ./gradlew: 52: Syntax error: word unexpected (expecting "in") Solution: it's simply a difference between UNIX and windows handling end of line character, so easy to solve just type sed -i.bak 's/\r$//' gradlew (see https://stackoverflow.com/questions/55342922/gradle-gradlew-expecting-in)

Adding a new programing language

  • Add the name of the new language to the enum CodeLanguage at src/enums.
  • Add a new python class to src/CodeTranslators inheriting GenericCodeTranslator
    • Implement all abstract methods
    • Correctly set all fields from GenericCodeTranslator
  • Expand the methods safe_run_files and build_binaries in src/TaintAnalysisTools/GenericTaintAnalysisTool to correctly handle the new language

Adding a new static analysis tool

  • Add the name of the new tool to the enum FlowPathChecker at src/enums.
  • Add a new python class to src/TaintAnalysisTools inheriting GenericTaintAnalysisTool
    • Implement all abstract methods
    • Correctly set the fields language and used_tool
  • Add a new case to main.read_tools_to_test() instantiating the new tool

Experiments

All experiments are set up to be executed in Podman containers, each script will start up to 150 containers (each bounded to 2 CPUs by default). Make sure you have enough computing power and disk space (>12G) before starting the experiments. You can change the number of CPUs by adjusting or removing --cpus 2 from the command that starts containers (in each script). Note, however, that if containers compete for computing power, time to finish the experiment might be unpredictable.

We used python 3.13.7 in all containers, we observed that different versions of Python lead to different initialization of random number generators. To reproduce the generated programs and applied transformations as in the paper, use the provided scripts.

Important! Before running the experiments with Semgrep, obtain a free registration and get a token for logging in to use with their Command Line Interface. Put the token into the SEMGREP_APP_TOKEN environment variable, it will be automatically picked up by the corresponding scripts (run_differential.sh, time-to-no-bug.sh, statfier-start-containers.zsh, run_mt_semgrep.sh, gen-seed-programs.sh, run_mt_reset_1.sh) when building podman images using Containerfile.semgrep, Containerfile.differential, and Containerfile.Statfier.

Start all shell scripts from the root directory of this repository.

RQ3: Time to no bug experiment (Table 2)

Run the command ./helper-scripts/time-to-no-bug.sh to start 16 containers (1 per bug). Requirements:

  • 32 CPUs (assuming 2 per container)
  • (roughly) 2G disk space to save all logs and intermediate files (buggy programs, analysis results, pickled AST object)
  • Time on our machine one bug (FD764) reaches 24h time out, all others terminate within 4 hours. Please refer to the Table 2 in the paper for individual times per bug.

To run the container for the fastest bug FD763, change FOLDER_NAME to a path to local directory where logs will be stored, then execute (build the image then create a container):

  podman build -t taintcc-flowdroid . -f Containerfile.fd-oldest-buggy

  podman container run --rm -d --name taintcc-fd-FD763-eq --cpus 2 \
    --mount type=bind,source="$FOLDER_NAME",target=/taintcc/logs  \
    -e ITERATIONS='100000'  \
    -e INITIAL_STATEMENTS_TO_GENERATE=1 \
    -e MAX_STATEMENTS_TO_GENERATE=-1  \
    -e STATEMENT_GENERATION_STEP='0' \
    -e TAT_TO_TEST="FD" \
    -e ALLOW_TRIGGERING_ALL='True' \
    -e TESTING_MODE='time-to-no-bug' \
    -e BUGGY_SEED="FD763" \
    -e MAX_FINDINGS=10 \
    -e MAX_RUNTIME=86400 \
    -e METAMORPH_FLOWDROID_SETTINGS='True' \
    -e INCLUDE_NONEQUIVALENT_TS='False' \
    -e INCLUDE_TAINTAWARE_TS='True' \
    -e INCLUDE_TAINTOBLIVIOUS_TS='True' \
    -e INCLUDE_SEMEQUIV='True' \
    localhost/taintcc-flowdroid

RQ4 and RQ5: Average number of bugs and test throughput

Run the commands ./helper-scripts/run_differential.sh(5 containers, one per integer random seed) to start differential testing. For metamorphic setting, we recommend to start batches of experiments separately, as the total required number of containers to start is 375 (for reset after 10 iterations) + 45 (for reset after every iteration). The scripts to run metamorphic testing experiments:

  • ./helper-scripts/run_mt_flowdroid.sh - to obtain values for FlowDroid in Table 3, Figure 11, and row 8 of Table 4 (75 containers, 5 seeds x 1 analyzer x 3 program sizes x 5 configurations)
  • ./helper-scripts/run_mt_mtrench.sh - to obtain values for Mariana Trench in Table 3, Figure 11, and row 9 of Table 4 (75 containers, 5 seeds x 1 analyzer x 3 program sizes x 5 configurations)
  • ./helper-scripts/run_mt_pysa.sh - to obtain values for Pysa in Table 3, Figure 11, and row 10 of Table 4 (75 containers, 5 seeds x 1 analyzer x 3 program sizes x 5 configurations)
  • ./helper-scripts/run_mt_semgrep.sh - to obtain values for Semgrep in Table 3, Figure 11, and rows 11-12 of Table 4 (150 containers, 5 seeds x 2 analyzers x 3 program sizes x 5 configurations)
  • ./helper-scripts/run_mt_reset_1.sh (45 containers, 5 seeds x 5 analyzers x 3 program sizes) - to obtain values for rows 3-7 of Table 4.

Requirements:

  • for all in parallel: 840 CPUs (scripts allocate 2 CPUs per container)
  • (roughly) 10G disk space for metamorphic testing logs
  • (roughly) 12G disk space for differential testing logs
  • Time on our machine all experiments (containers running in parallel) took multiple days. More detailed breakdown:
    • metamorphic testing per analyzer:
      • FlowDroid terminated within 4.5 hours
      • MarianaTrench took ~4 hours (default reset after 10) and ~40 hours (reset after 1)
      • Pysa: approximately 20 hours for TaintCC default setting (reset the AST after 10 iterations), and 19 hours for reset after 1 iteration
    • differential testing took at most 4.5 days (~109 hours)

You can adjust the timeout when staring the containers by configuring the environmental variables. Set time limit in seconds with -e MAX_RUNTIME=your-limit, set number of iterations with -e ROUNDS=your-limit and TaintCC will execute your-limit x ITERATIONS total iterations. Note that ITERATIONS variable controls after how many iterations a fresh AST is generated. Keep -e ITERATIONS=1 for differential testing, for metamorphic the values in our experiments are 1 and 10. If you set both time and rounds limits, the experiment will terminate when the first of these limits will be reached.

Note that the number of bug reports may differ depending on the total number of the iterations performed. You can view the summary of each run after it terminates in the taintcc_stats.csv file in your selected log directory.

Processing the results

Generated results will be saved into the <root-of-TaintCC>/data directory by default, they can be processed with python helper-scripts/process_logs.py in the terminal (from the root directory of this repo). The script requires packages matplotlib, statistics, copy, csv, os, copy, datetime. It will generate Tables 2,3 and Figure 11 Please wait until the experiments terminate and containers are deleted before processing the results! The script relies on the files with statistics that are generated at the end of each experiment.

If you change the log directories from the defaults, adjust them in the process_logs.py script. Note that the current script uses subdirectory names for mapping the results to the correct experiment, e.g. data/metamorphic/reset-after-10/seed_777/fd/size_32_semeq/ stores results after running TaintCC on FlowDroid with random seed 777, resetting T_og after 10 iterations, starting from the medium-sized programs with 32 statements, and using only semantically-equivalent transformations. We do not recommend renaming the subdirectories.

Statfier Experiments

The complete artifact including experiments with Statfier are available on Zenodo.

Detected Issues

About

A metamorphic-testing framework for static taint analyzers

Resources

Readme
Apache-2.0 license
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages