refactor: rewrite binary-builder in Go, remove Ruby implementation by ramonskie · Pull Request #97 · cloudfoundry/binary-builder

ramonskie · 2026-03-23T11:11:16Z

Summary

This PR replaces the Ruby implementation of binary-builder with a full Go rewrite that achieves parity with the original for all cflinuxfs4 dependencies.

Motivation

The Ruby binary-builder was tightly coupled to the cflinuxfs3/cflinuxfs4 stack lifecycle and increasingly difficult to maintain. A Go rewrite provides:

A single statically compiled binary with no Ruby runtime dependency inside the build container
Type-safe, testable recipe logic with a clear Runner/Fetcher interface boundary
Stack configuration as data (stacks/*.yaml) rather than hardcoded Ruby conditionals
A consistent recipe abstraction (Recipe interface) that makes adding new deps a one-file change

What Changed

Removed

Entire Ruby implementation (bin/, lib/, recipe/, spec/, Gemfile, .rubocop.yml)
Duplicate cflinuxfs4/ subtree (was a copy of the root Ruby builder)

Added

cmd/binary-builder/main.go — CLI entry point with dual-mode input (direct flags or source/data.json)
internal/ — Go packages for every concern: recipe, runner, fetch, stack, archive, artifact, output, source, apt, portile, autoconf, compiler, gpg, php, fileutil
stacks/cflinuxfs4.yaml + stacks/cflinuxfs5.yaml — all stack-specific values (apt packages, compiler paths, bootstrap URLs) as data
test/parity/ — shell-based parity test harness comparing Go builder output against the Ruby builder for each dep
test/exerciser/ — Docker-based exerciser tests that stage and run a built artifact

Recipe architecture highlights

Runner interface — all subprocess execution goes through runner.Runner; FakeRunner is used in all unit tests
Fetcher interface — all HTTP calls go through fetch.Fetcher; FakeFetcher is used in all unit tests
autoconf.Recipe — shared configure/make/install abstraction used by libunwind, libgdiplus, openresty, nginx
RepackRecipe — download-and-optionally-strip abstraction used by bower, yarn, setuptools, flit-core, rubygems
PassthroughRecipe — download-only abstraction used by tomcat, composer, all JVM deps, and PyPI sdist deps
PyPISourceRecipe — generic PyPI sdist recipe; new deps (e.g. flit-core) are a single line in passthrough.go
Stack config is data, not code — no if stack == "cflinuxfs4" guards anywhere in recipe logic

Testing

# Unit tests (no Docker)
go test -race ./...

# Parity tests (requires Docker + network)
make parity-test DEP=ruby STACK=cflinuxfs4
make parity-test-all STACK=cflinuxfs4

All unit tests pass with -race. Parity has been verified for the full cflinuxfs4 dependency set.

for backwards compatibility we have create a backup brach https://github.com/cloudfoundry/binary-builder/tree/backup/main-before-go-refactor

- Add complete Go binary-builder CLI (cmd/binary-builder/main.go) - Add Go recipes for all 33 in-scope deps: ruby, python, node, go, php, r, jruby, nginx, openresty, nginx-static, httpd, dotnet-*, libunwind, libgdiplus, hwc, composer, bundler, rubygems, yarn, bower, pip, pipenv, setuptools, tomcat, openjdk, zulu, sapmachine, skywalking-agent, appdynamics, jprofiler-profiler, your-kit-profiler, miniconda3-py39 - Add stack config as data (stacks/cflinuxfs4.yaml) - Add parity test infrastructure (test/parity/compare-builds.sh) - Add PHP extension support (php_extensions/, internal/php/) - Fix jruby: ArtifactVersion field keeps dep-metadata version as raw JRuby version; tar czf -C packDir . produces ./-prefixed entry list - All parity tests pass on cflinuxfs4; go test ./... passes - Update AGENTS.md with parity project context and results - Pin Gemfile to ruby 3.3.6 and update Gemfile.lock

…soft - Delete all Ruby source: recipe/, lib/, bin/, spec/, cflinuxfs4/ Ruby files - Remove Ruby project files: Gemfile, Gemfile.lock, .rubocop.yml, .ruby-version, .rspec, go-version.yml - Remove Ruby-era docs: PHP-Geoip.md, PHP-Oracle.md - Remove compiled binary artifact - Update .gitignore: add cflinuxfs4/, /binary-builder; remove .rspec - Rewrite README.md and AGENTS.md for Go-only builder - Fix stacks/cflinuxfs4.yaml: replace bionic openjdk-jdk-1.8.0_242 URL with jammy Bellsoft JDK 8u442 (correct stack, latest available) - Fix stacks/cflinuxfs5.yaml: replace XXX placeholder with same jammy Bellsoft JDK 8u442 (no noble bucket exists; Bellsoft JDK 8 is binary-compatible with Ubuntu 24.04)

- (#1) output: fix Commit() using shell || operator with Runner; now runs git diff --cached --quiet first and only commits when changes are staged - (#2) fetch: add 10-minute timeout to HTTP client (was http.DefaultClient) - (#3) archive: remove unused globs parameter from Pack - (#4) simple: fix YarnRecipe mutating src.Version; use local var and set outData.Version explicitly so findIntermediateArtifact can locate the file - (#5) nginx: fix misleading comment claiming custom args are prepended - (#6) nginx: use slices.Concat instead of append onto package-level slice - (#7) passthrough: extract moveFile into internal/fileutil.MoveFile with cross-device (EXDEV) fallback; use it in both passthrough.go and main.go - (#8) go_recipe: move bootstrap URL from hardcoded constant into stack YAML (go.bootstrap_url) following the same pattern as jruby.jdk_url

- stack_test: update TestJRubyConfigCflinuxfs4 to assert 'jammy' (was stale 'bionic'); update TestJRubyConfigCflinuxfs5 to assert 'jammy' with a comment explaining there is no noble JDK bucket yet so the jammy build is used as a binary-compatible fallback - recipe_compiled_test: change newCompiledStack JDKInstallDir from /opt/java to t.TempDir()+"/java" so FakeFetcher's os.MkdirAll succeeds without root

Remove omitempty from OutDataSource.SHA256 and OutDataSource.URL so they are present even when empty, matching Ruby builder output format where these fields are always included in builds JSON.

…ages - Bump bellsoft JDK URL/sha256 from 8u442+7 to 8u452+11 (URL-encodes '+' as '%2B' to avoid HTTP 404 from java-buildpack.cloudfoundry.org) - Add libunwind_build apt group (autoconf, automake, libtool) needed by the libunwind recipe to regenerate ./configure from autotools sources

GitHub source archives only contain autotools sources (configure.ac), not the generated ./configure script, so we must run autoreconf -i first. - Install autoconf/automake/libtool from stack's libunwind_build apt group - Run autoreconf -i before ./configure - Fix dirName derivation: GitHub archives extract to libunwind-X.Y.Z (repo name + version, stripping the 'v' prefix from the tag)

Maven's polyglot plugin invokes bin/jruby as an external subprocess to build gem native extensions. That subprocess runs a shell script which needs JAVACMD to find java — but env vars set via RunWithEnv were not visible to Maven's ProcessBuilder child processes. Fix by embedding 'export JAVA_HOME=... export JAVACMD=... export PATH=...' directly in the shell command string so all descendent processes inherit them without relying on process environment inheritance through JVM. Also add jdkSubdir() helper: bellsoft JDK tarballs extract into a subdirectory (e.g. jdk8u452/) inside the install dir, so JAVA_HOME must point to that subdir rather than the install dir itself.

strings.ToLower() was lowercasing the map key to 'rserve' but the Ruby builder uses the original package name 'Rserve' (capital R) as the key in the dep-metadata sub_dependencies object. Use pkg.name directly.

…ILTER_DEP Data fixes: - jruby: upgrade 9.4.5.0 → 9.4.14.0 (src.zip, correct sha256); 9.4.5.0 fails during JRuby Lib Setup due to resolv native ext build - r: upgrade 4.3.2 → 4.4.2 with correct URL and sha256 - go 1.22.0: correct sha256 - httpd 2.4.58: sha256 is for .tar.bz2 (which the Go recipe downloads) - hwc: upgrade 2.0.0 → 106.0.0 (current release) - jprofiler-profiler: upgrade 13.0.14 → 15.0.4 (current release) New helpers: - run_dep_r: generates the 4 sub-dep data.json dirs (forecast, plumber, Rserve, shiny) that the Go R recipe reads from source-*-latest/ dirs, then calls compare-builds.sh with --sub-deps-dir - FILTER_DEP support in run_dep, run_dep_r, skip_dep so that 'DEP=httpd make parity-test' only runs that one dep Makefile: fix parity-test target to use DEP=$(DEP) ./test/parity/run-all.sh instead of calling compare-builds.sh directly (so FILTER_DEP works and all dep metadata is consistent with the test matrix)

…ENTS.md README: add a full Parity Tests section documenting how compare-builds.sh works (source pre-download, Ruby/Go container layout, output comparison table, exit outcomes, input format, and running instructions). AGENTS.md: remove the Parity Results table — it was a point-in-time snapshot that became stale and misleading as fixes were applied. Current pass/fail status is always in /tmp/parity-logs/ from actual test runs.

… behaviour FakeRunner does not execute commands so the jdk*/ subdir was never created on disk by the tar extraction step, causing jdkSubdir() to fail. Pre-create the directory in test setup so the recipe can proceed. Add hasDownload / hasDownloadContaining helpers and replace the inline download-loop assertions. Remove the JAVA_HOME env-map assertion (env vars are now embedded in the shell command string, not passed via RunWithEnv). Re-add TestJRubyRecipeUnknownVersion which was accidentally dropped.

- Move php_extensions/*.yml into internal/php/assets/ (co-locate with owning package) - Replace per-file //go:embed + hardcoded maps with a single embed.FS glob and init()-time auto-discovery keyed by filename convention - Adding a new PHP minor/major version now only requires dropping a YAML file in assets/ — no Go code changes needed - Remove --php-extensions-dir CLI flag and ExtensionsDir struct field - Update tests, README, and parity script to reflect new design

…ding test

… downloads - Wire Fetcher.Download (with checksum verification) into all recipes that previously used raw wget/curl: openresty, php, bundler, go, httpd, libunwind, and all PHP extension types (pecl, pkgconfig, native, special) - Add BootstrapSHA256 and HTTPDSubDepsConfig to stack structs; populate real SHA256s for all bootstrap and sub-dep URLs in cflinuxfs4.yaml / cflinuxfs5.yaml - Replace httpd's runtime git ls-remote APR version lookup with pinned values from stack config - Add ExtractFlag helper to portile to fix tar zstd mis-detection on cflinuxfs4 - Fix libunwind dirName logic to handle both URL styles without double-prefix - Update all unit tests to wire FakeFetcher and assert on Download calls

… URL) - Use Ruby 3.4.6 and mount local buildpacks-ci working tree (writable) to fix build.rb argument mismatch with master buildpacks-ci - Remove silent-skip hatch so Ruby builder failures are real failures - Add zstd to Go builder container apt-get to fix mise Go bootstrap extraction - Filter PHP snmp mibs paths from file-list diff (Ruby builder bug workaround) - Switch libunwind parity test to release download URL with correct SHA256 - Fix your-kit-profiler dispatch in builder.rb (sub -> gsub for method name)

…e helpers - Move fileSHA256 and mustCwd into helpers.go (phase 0) - Extract GoToolRecipe in dep.go; dep/glide/godep delegate to it (phase 1) - Add RepackRecipe in repack.go; bower/yarn/setuptools/rubygems delegate to it (phase 2) - Add BundleRecipe in bundle.go; pip/pipenv delegate to it (phase 3)

…diplus/openresty/nginx - Add internal/autoconf package with Recipe build engine and Hooks (10 hook fields including BeforeDownload and AfterPack); full unit test coverage (phase 5.1–5.2) - Extract removeNginxRuntimeDirs helper in nginx.go (phase 4.1) - Migrate libunwind, libgdiplus, openresty, nginx/nginx-static to delegate to autoconf.Recipe via thin wrapper structs; remove buildNginxVariant (phase 5.3–5.6)

- Add shared recipe abstractions table (GoToolRecipe, RepackRecipe, BundleRecipe, autoconf.Recipe, PassthroughRecipe) - Add full autoconf.Recipe usage example with thin wrapper pattern - Add hook reference table with defaults and typical overrides

…tack YAML Replace the three scattered bootstrap sections (ruby_bootstrap, go.bootstrap_*, jruby.jdk_*) with a single bootstrap: key containing go/jruby/ruby sub-entries, all using consistent url/sha256/install_dir field names. Sort top-level YAML sections alphabetically. Update stack.go (BootstrapBinary + BootstrapConfig), all three recipe consumers, and tests to match.

Replace `git clone --depth=1` (which tracked the default branch and would break once this Go-builder branch merges to main) with `git clone --depth=1 --branch ruby-builder-final` so the parity test always uses the last known-good Ruby builder tree regardless of what main contains.

…ect parity test flags

…nd parity test fixes - cmd/binary-builder/main.go: replace --artifacts-dir/--builds-dir/--dep-metadata-dir/--skip-commit flags with --output-file; add Mode 1 (--name/--version) and Mode 2 (--source-file) input modes; write JSON summary to file instead of stdout; route all build log output to stderr - internal/recipe/r.go: use devtools::install_version (matching ruby-builder-final tag, not remotes) - test/parity/compare-builds.sh: update Go builder invocation to --output-file flag; write dep-metadata and builds-artifacts JSON from summary in the test script (mirroring build.sh); treat R artifact file list diff as WARN-only (non-deterministic transitive R packages)

ivanovac

lgtm

…ble)

ramonskie added 28 commits February 25, 2026 15:25

point to log location in overview for our parity test

c4366e2

[test] remove deps that we do not have and add deps that where missing

507b885

output: always serialize source sha256 and url fields

4884af1

Remove omitempty from OutDataSource.SHA256 and OutDataSource.URL so they are present even when empty, matching Ruby builder output format where these fields are always included in builds JSON.

recipe/r: preserve Rserve key casing in sub-dependencies map

d78934c

strings.ToLower() was lowercasing the map key to 'rserve' but the Ruby builder uses the original package name 'Rserve' (capital R) as the key in the dep-metadata sub_dependencies object. Use pkg.name directly.

🔍 fix: clarify patchFileRE intent, drop duplicate test, rename mislea…

dbe227a

…ding test

remove temp test files

a0a5160

♻️ refactor: move hardcoded apt packages into stacks/*.yaml

f4a481e

docs: update README with dual-mode CLI, JSON summary format, and corr…

c9f2911

…ect parity test flags

✨ feat: add flit-core and generalize setuptools into PyPISourceRecipe

c5c8ff8

tnikolova82 previously approved these changes Mar 23, 2026

View reviewed changes

ivanovac previously approved these changes Mar 23, 2026

View reviewed changes

🐛 fix: switch R recipe from devtools to remotes for install_version

3f84107

ramonskie dismissed stale reviews from ivanovac and tnikolova82 via 3f84107 March 23, 2026 17:27

ramonskie added 2 commits March 23, 2026 21:16

🐛 fix: correct gfortran version to 13 for cflinuxfs5 (Ubuntu 24.04 no…

e8c21b0

…ble)

✅ test: fix cflinuxfs5 gfortran tests to use version 13 and libexec_path

a6734fe

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: rewrite binary-builder in Go, remove Ruby implementation#97

refactor: rewrite binary-builder in Go, remove Ruby implementation#97
ramonskie wants to merge 31 commits intomainfrom
go-binary-builder-cflinuxfs4-parity

ramonskie commented Mar 23, 2026 •

edited

Loading

Uh oh!

ivanovac left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ramonskie commented Mar 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Related

Motivation

What Changed

Removed

Added

Recipe architecture highlights

Testing

Uh oh!

ivanovac left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ramonskie commented Mar 23, 2026 •

edited

Loading