Objeck Language Regression Test Suite

← All docsprograms/regression/README.md on GitHub

This regression test suite validates the compiler, virtual machine, JIT and libraries after every change. It has about 780 tests; TESTS.md lists every one, by category, with what it checks. It covers:

Purpose

TESTS.md is generated by gen_manifest.py in this directory; it also lists the 14 debugger tests that run_debugger_tests.sh drives. Do not commit it in a pull request: the Regression test manifest workflow regenerates it on master after each merge that changes a test. When every pull request carried its own copy, each merge made every other open pull request conflict on it.

Running Tests

Build a deploy tree first (core/release/deploy_windows.cmd x64|arm64, core/release/deploy_posix.sh x64|arm64 or core/release/deploy_macos_arm64.sh). The runners use core/release/deploy-<arch> when it exists and core/release/deploy otherwise.

Linux/macOS

cd programs/regression
chmod +x run_regression.sh
./run_regression.sh x64          # or arm64

Windows

cd programs\regression
run_regression.cmd x64      REM For x64 build
run_regression.cmd arm64    REM For ARM64 build

Each test is compiled with obc -opt s3 -lib cipher,collect,xml,json (plus any libraries its # EXTRA_LIBS: line names) and run with obr. Compiler logs and program output go to results/. Apart from the negative tests (see the directives below), the runners judge a test by its exit code: 0 passes, anything else fails. A test that prints a line starting with SKIP: and exits 0 counts as skipped, not passed.

Environment variables both runners honor:

OBJECK_JIT_THRESHOLD=1 makes obr compile every method on its first call; CI runs the suite a second time with it.

Other runners in this directory:

Test Organization

The file-name prefix decides a test's category in TESTS.md (core_, jit_, arm64_, bad_, collect_, ml_ and so on; the mapping is CATS in gen_manifest.py, and a name that matches none is filed under Other). Directives are comment lines starting at column 0:

DirectiveEffect
# EXTRA_LIBS: net,gen_collectLinks these libraries as well (comma-separated, no spaces)
# EXPECT_COMPILE_ERROR: <message>Passes only if compilation fails and the compiler output contains <message>
# EXPECT_RUNTIME_ERRORPasses only if obr exits non-zero with output
# JIT_DISABLERuns the test with the JIT off; the next line must be # reason: ... (tools/cicd/check_jit_optouts.py)
# VERIFY_SKIP, # GC_STRESS_SKIPSkipped in the nightly heap-verifier and forced-GC-threshold runs; the next line must be # reason: ... naming the run that failed

Data a test reads lives under data/, found relative to this directory (every runner runs tests from here). data/bert/ holds three tiny BERT-shaped ONNX models, their vocabulary and gen_tiny_bert.py, which regenerates them byte-for-byte; docs/ML_BERT_DESIGN.md describes them.

Adding New Tests

  1. Create a new .obs file in this directory
  2. Name it descriptively: category_feature.obs
  3. Include a comment header explaining what it tests; gen_manifest.py takes the test's TESTS.md description from it
  4. Output PASS or FAIL messages for automation
  5. Exit with code 0 on success, non-zero on failure (Runtime->Exit(1)). Printing FAIL is not enough: a test that returns normally exits 0 and is scored a pass, and CI's tools/cicd/check_test_exit_codes.py rejects a test that prints FAIL with no exit path
  6. Leave TESTS.md alone: it is regenerated on master after the merge (run python gen_manifest.py locally to preview, but do not commit the result)

Test Template

#~
# Test Name
# Brief description of what this tests
# Related commit: <commit-hash>
~#

class MyTest {
    function : Main(args : String[]) ~ Nil {
        "Testing my feature..."->PrintLine();

        pass := true;

        # Your test logic here
        if(test_condition) {
            # Test passed
        } else {
            pass := false;
            "  FAIL: Description of failure"->PrintLine();
        };

        if(pass) {
            "PASS: My feature"->PrintLine();
        } else {
            "FAIL: My feature"->PrintLine();
            Runtime->Exit(1);
        };
    }
}

CI/CD Integration

This test suite runs automatically in GitHub Actions on:

It runs natively on all five platforms (Windows x64 and ARM64, Linux x64 and ARM64, macOS ARM64), twice on each: once as is and once with OBJECK_JIT_THRESHOLD=1. The same job runs the debugger, DAP and VM-flag tests from this directory, and CI's tools job lints the tests themselves (check_test_exit_codes.py, check_jit_optouts.py, check_diff_markers.py in tools/cicd). .github/workflows/nightly-hardening.yml (dispatched by hand for now) adds run_differential.py and runs the suite under the heap verifier and a forced GC threshold.

See .github/workflows/ci-build.yml for CI configuration.

Expected Output

From run_regression.sh; run_regression.cmd prints the same summary with slightly different per-test lines.

========================================
  Objeck Regression Test Suite
  Platform: x64
========================================

Running: ai_game_test...
  [PASS] (1s)
...
Running: api_gemini_test...
  [SKIP] SKIP: No Gemini API key
...
Running: bad_call_undefined...
  [PASS] (compile error as expected)
...

========================================
  Results: N passed, M skipped, 0 failed
========================================

Maintenance