tasks: Conducting dynamic analysis

How do you conduct dynamic analysis of your research software?

Description

Dynamic analysis means running your software and observing it at runtime to find problems that static checks and unit tests miss: memory errors, undefined behaviour, performance bottlenecks, or inputs that crash it. When your software depends on numerical correctness or runs on high-performance computing systems, these defects are hard to detect because they may not surface under normal test conditions. It complements static analysis by exercising the code under real or adversarial inputs and resource limits. It reports only on the paths and inputs you actually run, so a clean run is not proof of correctness.

Considerations

Memory-unsafe languages (C, C++, Fortran) are most exposed to the errors dynamic analysis catches; in memory-safe languages the highest-value checks are performance and concurrency.
Sanitizers and memory checkers add significant runtime overhead, so run them in dedicated test or CI builds, not in release builds.
Fuzzing (feeding your program many automatically generated, often malformed, inputs to find ones that make it crash or misbehave) reaches input-handling paths that hand-written tests rarely cover.
Profiling results depend on input data and system state, so one dataset’s profile may not predict performance on another, particularly for data-dependent scientific algorithms.
In research software, numerical correctness often matters more than crash detection, so match your checks to the failure modes that would invalidate your results.
The tools named here reflect the time of writing; check for current alternatives before adopting one.

Solutions

For memory-unsafe code, run your test suite under Valgrind, or under AddressSanitizer and UndefinedBehaviorSanitizer for lower overhead by compiling with -fsanitize=address,undefined -g on GCC or Clang.
Run the sanitizer build and tests in CI on every pull request, with a separate ThreadSanitizer job for multithreaded code, as it cannot be combined with AddressSanitizer.
Profile representative workloads with perf, gprof, or Valgrind’s Callgrind for compiled code, Python’s cProfile module, or Score-P for parallel HPC applications.
Measure coverage in CI — Gcov with lcov for C and C++, coverage.py or pytest-cov for Python, covr for R — so you know which code your sanitizer and test runs exercised.
Fuzz code that processes external input with AFL++ for C and C++ or Atheris for Python, seeded with valid inputs and run in a sandbox for a fixed time budget.
For web-facing software, add a dynamic scanner such as ZAP to your pipeline to find vulnerabilities that static scanners miss.
Compare numerical results with stored reference outputs under sanitizer and optimised builds, and use Verrou or Verificarlo to find results sensitive to floating-point rounding.
Record every finding as an issue with the triggering input, execution environment, and tool output, and add that input to your regression tests so the bug stays fixed.

AI Disclosure

This work was produced with the assistance of Qwen 3.8(27B), under the strict editorial control and factual verification of the human author. Claude Fable 5.1 was then used as a second step to review and reformulate the draft and then checked again by the human author.

tasks

Testing software

How to test your software?

tasks

Continuous Integration and Continuous Delivery/Deployment

How can you use CI/CD in software development?

tasks

Using static analysis

How to use static analysis tools to detect bugs, enforce coding standards, check types, and improve the quality of research software before it runs.

tasks

Improving research software security

How to improve the security of research software by managing repository risks, secrets, dependencies, static analysis, CI/CD checks, and vulnerability reporting.

Training

EVERSE TeSS search results:

Tools and resources on this page

Tool or resource	Description	Related pages
AddressSanitizer	AddressSanitizer (ASan) is a memory error detector built into GCC and Clang, enabled with -fsanitize=address, which finds out-of-bounds accesses, use-after-free, and memory leaks at runtime.
AFL++	AFL++ is a coverage-guided fuzzer and the maintained successor to American Fuzzy Lop (AFL). It supports persistent-mode fuzzing, fork-server mode, and integration with libFuzzer, honggfuzz, and other harnesses.
Atheris	Atheris is a coverage-guided fuzzing engine for Python code and native CPython extensions, based on libFuzzer, which can be combined with AddressSanitizer or UndefinedBehaviorSanitizer when fuzzing native code.
coverage.py	coverage.py is a Python code-coverage tool that measures which statements and branches execute during a test run, producing line- and branch-coverage reports in text, HTML, or XML format.
covr	covr is an R package for measuring test coverage of R code, reporting which lines and functions are exercised by a test suite and integrating with continuous integration services to track coverage over time.	Maintaining research s...
Gcov View on TechRadar	Code coverage analysis tool
gprof	gprof is the GNU profiler, part of GNU Binutils, which produces flat profiles and call graphs for programs compiled with the -pg option.
lcov	lcov is a graphical front end for gcov that collects coverage data from multiple source files and produces HTML reports showing line, function, and branch coverage.
perf	perf is a performance analysis tool for Linux built on top of the kernel's perf_events interface, used for CPU profiling, hardware counter sampling, and tracing of compiled programs.
pytest-cov	pytest-cov is a pytest plugin that produces coverage reports for Python code, integrating with the coverage.py library to show which lines are exercised by tests.	Maintaining research s...
Score-P View on TechRadar	The Score-P (Scalable Performance Measurement Infrastructure for Parallel Codes) measurement infrastructure is a highly scalable and easy-to-use tool suite for profiling and event trace recording of HPC applications.
ThreadSanitizer	ThreadSanitizer (TSan) is a data race detector built into GCC and Clang, enabled with -fsanitize=thread, for multithreaded C and C++ code.
UndefinedBehaviorSanitizer	UndefinedBehaviorSanitizer (UBSan) is a runtime checker built into GCC and Clang, enabled with -fsanitize=undefined, which detects undefined behaviour such as signed integer overflow, misaligned access, and null dereferences.
Valgrind View on TechRadar	Detect memory management and threading bugs, run profiling to see how to speed up your programs	Phoenix2
Verificarlo	Verificarlo is an LLVM-based compiler that instruments floating-point operations with Monte Carlo Arithmetic and other backends to estimate the numerical accuracy and stability of a program.
Verrou	Verrou is a Valgrind tool that finds floating-point round-off errors by perturbing rounding modes at runtime, so that results which are sensitive to rounding become visible without recompiling the program.
ZAP	ZAP (Zed Attack Proxy, formerly OWASP ZAP) is a dynamic application security testing tool that scans running web applications for vulnerabilities and can run in CI pipelines.

Shoaib Sufi, "Conducting dynamic analysis". everse.software. http://everse.software/RSQKit/dynamic_analysis .

tasks: Conducting dynamic analysis

How do you conduct dynamic analysis of your research software?

Description

Considerations

Solutions

Further Reading

AI Disclosure

Related pages

Training

Tools and resources on this page