Physics Validation Testing

(this file was partially generated with an LLM but carefully checked by me, Jorge)

This document describes the physics validation test suite for metalquicha, which ensures that code changes don’t introduce numerical regressions.

Overview 

The validation suite runs quantum chemistry calculations and compares the computed total energies against reference values. All tests must pass before merging code changes.

Key features:

Automated testing of fragmented and unfragmented calculations
Energy comparison with configurable tolerance (default: 1.0e-6 Hartree)
Support for multi-molecule calculations (conformers, isomers)
CI/CD integration ready
Detailed logging for debugging failures

Directory Structure 

validation/
└── inputs/
    ├── *.json          # Input JSON files (geometry, fragments, parameters)
    └── *.mqc           # Generated .mqc files (created by mqc_prep.py)

validation_tests.json   # Test manifest with expected energies
run_validation.py       # Validation test runner script
validation_logs/        # Generated: stdout/stderr from each calculation

Output Files 

The validation script generates:

output_*.json - Calculation results for each test
validation_logs/*.log - Stdout/stderr from each calculation (for debugging)

Workflow 

Adding a New Test 

1. Create Input File

Place JSON input files in validation/inputs/:

cat > validation/inputs/my_test.json << EOF
{
  "schema": {"name": "mqc-frag", "version": "1.0"},
  "molecules": [{
    "xyz": "my_molecule.xyz",
    "molecular_charge": 0,
    "molecular_multiplicity": 1
  }],
  "model": {"method": "XTB-GFN1"},
  "driver": "Energy"
}
EOF

2. Generate Reference Energy

Run the calculation to get the reference energy:

# Convert JSON to .mqc
python3 mqc_prep.py validation/inputs/my_test.json

# Run calculation
./build/mqc validation/inputs/my_test.mqc

# Extract energy from output JSON
cat output_my_test.json | grep '"total_energy"' | head -1

3. Update Test Manifest

Edit validation_tests.json to add your test with expected energy:

{
  "description": "Physics validation tests for metalquicha",
  "tolerance": 1.0e-6,
  "tests": [
    {
      "name": "My test case",
      "input": "validation/inputs/my_test.mqc",
      "expected_energy": -123.456789,
      "type": "unfragmented"
    }
  ]
}

Test types:

unfragmented: Single molecule, no fragmentation
fragmented: Many-body expansion calculation
multi_molecule: Multiple independent molecules (conformers, isomers)

4. Run Validation

# Run all tests (automatically converts JSON to .mqc first)
python3 run_validation.py

# Verbose output
python3 run_validation.py -v

# Skip .mqc preparation (if already done)
python3 run_validation.py --skip-prep

# Use custom executable
python3 run_validation.py --exe ./my_build/mqc

Running the Validation Suite 

Basic Usage 

# Run all validation tests
python3 run_validation.py

Output:

============================
Metalquicha Validation Suite
============================

Converting JSON inputs to .mqc format...
✓ Converted validation/inputs/h3o.json
✓ Converted validation/inputs/prism.json
...

Running 10 validation tests...

[ 1/10] Unfragmented glycine tripeptide ... ✓ PASSED
[ 2/10] Charged cluster MBE ............... ✓ PASSED
[ 3/10] Glycine 10-mer MBE ................ ✓ PASSED
...

============================
Summary: 10/10 tests passed
============================

Advanced Options 

# Verbose mode (shows all output)
python3 run_validation.py -v

# Skip JSON → .mqc conversion
python3 run_validation.py --skip-prep

# Use custom mqc_prep.py location
python3 run_validation.py --prep-script /path/to/mqc_prep.py

# Use custom executable
python3 run_validation.py --exe /path/to/mqc

# Specify custom test manifest
python3 run_validation.py --tests my_tests.json

Test Types 

Unfragmented Tests 

Single molecule calculations without fragmentation.

Example: validation/inputs/h3o.mqc

{
  "name": "Hydronium ion unfragmented",
  "input": "validation/inputs/h3o.mqc",
  "expected_energy": -5.773131213617977,
  "type": "unfragmented"
}

JSON output structure:

{
  "h3o": {
    "total_energy": -5.773131213617977
  }
}

Fragmented Tests (MBE/GMBE)

Many-body expansion calculations with fragment-based energies.

Example: validation/inputs/prism.mqc (MBE(2) on water prism)

{
  "name": "Water prism MBE",
  "input": "validation/inputs/prism.mqc",
  "expected_energy": -34.6736678571,
  "type": "fragmented"
}

JSON output structure (abbreviated):

{
  "prism": {
    "total_energy": -34.6736678571,
    "levels": [
      {
        "level": 1,
        "n_fragments": 6,
        "energies": [...]
      },
      {
        "level": 2,
        "n_fragments": 15,
        "energies": [...]
      }
    ]
  }
}

Multi-Molecule Tests 

Multiple independent molecules (conformers, isomers) in one input.

Example: validation/inputs/multi_frag.mqc

{
  "name": "Multi-fragment calculation",
  "input": "validation/inputs/multi_frag.mqc",
  "expected_energies": {
    "molecule_1": -34.6736678571,
    "molecule_2": -34.6736678571
  },
  "type": "multi_molecule"
}

JSON output structure:

{
  "multi_frag": {
    "molecule_1": {
      "total_energy": -34.6736678571
    },
    "molecule_2": {
      "total_energy": -34.6736678571
    }
  }
}

Energy Extraction 

The validation script extracts the top-level total_energy field for each calculation type.

Unfragmented Calculations 

{
  "basename": {
    "total_energy": -123.456789
  }
}

Extracted value: -123.456789

Fragmented Calculations (MBE/GMBE)

{
  "basename": {
    "total_energy": -123.456789,
    "levels": [...]
  }
}

Extracted value: -123.456789 (intermediate level energies are not validated)

Multi-Molecule Calculations 

{
  "basename": {
    "molecule_1": {"total_energy": -123.45},
    "molecule_2": {"total_energy": -124.56}
  }
}

Extracted values: molecule_1: -123.45, molecule_2: -124.56

Tolerance Configuration 

Default energy tolerance is 1.0e-6 Hartree, configurable in validation_tests.json:

{
  "description": "Physics validation tests for metalquicha",
  "tolerance": 1.0e-6,
  "tests": [...]
}

The test passes if:

abs(computed_energy - expected_energy) < tolerance

Validation Test Examples 

The validation suite includes several representative test cases:

GMBE(1) Test (Overlapping Fragments)

Test: overlapping_gly3.mqc

System: Glycine tripeptide with overlapping fragments
Method: GMBE(1) with inclusion-exclusion principle
Expected energy: -47.0192718920 Ha
Purpose: Validates PIE coefficient enumeration and overlapping fragment logic

GMBE(3) Test (Higher-Order)

Test: nlevel_3_ov_decane.mqc

System: Decane molecule with overlapping fragments
Method: GMBE(3) (trimers)
Expected energy: -33.0506139740 Ha
Purpose: Validates higher-order GMBE calculations

Standard MBE(2) Test 

Test: prism.mqc

System: Water prism (6 water molecules)
Method: MBE(2) (dimers)
Expected energy: -34.6736678571 Ha
Purpose: Validates standard many-body expansion

Large System Test 

Test: w20_isomer.mqc

System: Water 20-mer isomer
Method: MBE(2)
Expected energy: -115.6850246841 Ha
Purpose: Validates performance on larger systems

Charged Cluster Test 

Test: charged_cluster.mqc

System: Charged molecular cluster
Method: MBE(2)
Expected energy: -45.7161383790 Ha
Purpose: Validates fragment charge handling

CI Integration 

For GitHub Actions or similar:

name: Validation Tests

on: [push, pull_request]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2

      - name: Build metalquicha
        run: |
          mkdir build && cd build
          cmake ..
          make -j

      - name: Run validation tests
        run: |
          python3 run_validation.py --exe ./build/mqc

The script exits with code 1 on any test failure, causing CI to fail.

Troubleshooting 

Common Issues 

“No JSON files found”

Create validation/inputs/ directory and add JSON files

“mqc_prep.py not found”

Specify path with --prep-script /path/to/mqc_prep.py

Energy mismatch

Check for:

Different compiler optimization levels (-O2 vs -O3)
Different dependency versions (tblite, LAPACK)
Numerical instabilities in the method
Incorrect reference energy (re-run and update)

Calculation timeout

Default is 300s per test. Edit run_validation.py if needed:

timeout = 600  # 10 minutes

Test fails intermittently

May indicate:

A bug in xtb caused by a compiler version, math library, etc. (this has been observed in MacOS)
Numerical instabilities in SCF convergence
Compiler-specific floating-point behavior

Debugging Failed Tests 

Check the log file:
```
cat validation_logs/my_test.log
```

Run test manually with verbose output:

./build/mqc validation/inputs/my_test.mqc

Check JSON output:

cat output_my_test.json | python3 -m json.tool

Compare energies:

python3 -c "import json; print(json.load(open('output_my_test.json')))"

Run with different compilers to check portability:

# gfortran
cmake -DCMAKE_Fortran_COMPILER=gfortran ..
make
python3 run_validation.py

# ifort
cmake -DCMAKE_Fortran_COMPILER=ifort ..
make
python3 run_validation.py

Best Practices 

Add tests for new features: Every new feature should have at least one validation test
Use representative systems: Include small (fast) and large (realistic) test cases
Document expected behavior: Add comments in test manifest explaining what each test validates
Update reference energies carefully: Only update when intentional physics changes occur
Check all test types: Include unfragmented, fragmented, and multi-molecule examples
Run before committing: Always run validation suite before pushing changes

Generating Reference Energies 

Best Practices 

Use release build for consistency:

cmake -DCMAKE_BUILD_TYPE=Release ..
make

Run multiple times to ensure reproducibility:

for i in {1..5}; do
  ./build/mqc validation/inputs/my_test.mqc
  grep total_energy output_my_test.json
done

Document the compiler and flags used:

gfortran --version
cmake .. -LA | grep CMAKE_Fortran_FLAGS

Cross-validate with external codes if possible (e.g., run XTB standalone)

Validation Test Manifest Schema 

The validation_tests.json file has the following structure:

{
  "description": "Physics validation tests for metalquicha",
  "tolerance": 1.0e-6,
  "tests": [
    {
      "name": "Test name (required)",
      "input": "path/to/input.mqc (required)",
      "expected_energy": -123.456,  // for single molecule
      "expected_energies": {...},   // for multi-molecule
      "type": "unfragmented|fragmented|multi_molecule (required)"
    }
  ]
}

Fields:

description: Human-readable description of test suite
tolerance: Energy comparison tolerance (Hartree)
tests: Array of test objects

Test object:

name: Descriptive test name
input: Path to .mqc input file
expected_energy: Expected total energy (for single molecule tests)
expected_energies: Dictionary of expected energies (for multi-molecule tests)
type: Test type (unfragmented, fragmented, or multi_molecule)

Contributing New Tests 

When contributing new test cases:

Use descriptive names: "GMBE(2) water dimer with H-caps" not "Test 1"
Add comments in JSON: Explain what the test validates
Include .xyz files: Commit geometry files to validation/inputs/
Document the system: Add a comment describing the molecular system
Test edge cases: Include unusual charges, multiplicities, or geometries
Verify portability: Run on different compilers before submitting

Example contribution:

{
  "name": "Charged peptide with broken bonds",
  "input": "validation/inputs/charged_peptide.mqc",
  "expected_energy": -78.1234567890,
  "type": "fragmented",
  "comment": "Tests H-capping on charged system with multiple broken C-C bonds"
}