Provide support for traceability to original sources

[gberche-orange]

Offering the ability for paas-user or paas-operator to trace back the origin of the buildpack certain helps adoption of the buildpack by teams that are used to rely on certificated distros for handling their application stack binaries.

I'm thinking for instance to archive the verbose output of the binaries, outputting md5 for downloaded sources, and md5 of resulting binaries, in order to provide traceability to original sources.

Those verbobe traces could for instande be gziped and included in the buildpack as meta-data that paas-user can consult on the app droplet (using cf files) or ideally in the future complete a normalized [https://docs.google.com/document/d/1y0KoHCZ5r1kCShjipnqOjcpooxPmdbaq1p9cFxi6wGA/edit#heading=h.iym9agqqf9qs](buildpack dependencies) meta-data made available to the CC, and accessible through an API to Cf users.

The https://github.com/cloudfoundry-incubator/buildpack-packager#manifest seems close to partially fullfilling this traceability goal (i.e. traeability from offlline buildpack cached dependency to remote binaries repos). I wonder whether the buildpack manifest.yml is currently available at staging or runtime to cf users ?

[flavorjones]

Hi @gberche-orange,

Thanks for bringing this subject up again. I'd love to chat more about your use case, so that I can understand a bit better what we'd need to build.

The buildpack manifest is indeed present during staging, and we even verify the checksums of any downloaded dependencies when the buildpack is operating in uncached mode.

It sounds like you're specifically asking for a way to verify a checksum on the original source tarball (or a hash on a git repository)?

[gberche-orange]

thanks @flavorjones for your response and sorry for lack of clarity/structure in this exchange.

Yes, to rephrase the need I'm expressing: Once a app has been staged through the buildpack process, how can an operator be confident that the following potential issues did not happen, or alternatively can perform forensicks to trace back to root-cause of an issue, such as

1. for a buildpack fetching online internet dependencies (say the S3 bucket pivotal is exposing):
   1. the S3 bucket can compromised, and someone injected trojans
   2. the CF/pivotal pipelines were compromised and injected trojans into the S3 buckets
   3. the source repo DNS were temporary hijacked and infected sources were served from an attacker repo
   4. the source repos (or one of its replica) were compromised and trojans were injected into source code
2. for a buildpack fetching dependencies replicated on-prem:
   1. the replica was corrupted and is serving infected binaries

So I'm suggesting to record the evidences of successive steps in the process, so that someone can have elements to trace back and identify potential issues.

So, I'm suggesting an operator could inspect a staged app. By downloading one available log file, the operator could trace back manually up to the original source tarball URL and MD5/SHA, and most important steps in the chains.

In #4 I was suggesting to prevent 1iii and 1iv

To prevent 1i and 2i, it would be good to have binaries provided by CF/Pivotal be digitally signed, and publish a CF/Pivotal public key to verify them. Ideally, embed the public key at staging time so the signatures can be checked at staging time.

[flavorjones]

I've created a track of work around this request, which is outlined here:

https://docs.google.com/document/d/1vKHD0ulIq3Vw3J6-wI7AZ3peu2ZnPdXLYh0F1Rl-w8s/edit#

and the Tracker epic is here:

https://www.pivotaltracker.com/epic/show/2077742

[jvshahid]

It has been a while since the issue was opened and lot has changed since then. I think we currently address 1i and 2i by hashing the downloaded artifacts and making sure the hash matches the value found in the manifest. 1iii & 1iv are also covered by the pipeline checking the downloaded asset hash, and if available the gpg signature of the downloaded asset. This leave 1ii which I'm not sure how to tackle. thoughts ?

[jvshahid]

bump

[gberche-orange]

Sorry for late response on this. Thanks for the great improvements brought as part of this "chain of custody" epic!

One possibility for 1ii the CF/pivotal pipelines were compromised and injected trojans into the S3 buckets could be to publish the Cf/pivotal pipeline outputs including intermediate checksums for inputs (any download, concourse docker resource...) and outputs, so that a independent party is able to scrutinize it, reproduce it and detect from suspicious deltas. This may require to sanitize the public CI so that logs get made publicly available without leaking secrets.

For a 3rd party to be able to detect intrusions, this may require behavior close to reproductible builds in the buildpacks CI, i.e. tracking and removing non reproductible parts (such as timestamps).

[RochesterinNYC]

I believe that the work in this story is related to your issue @gberche-orange: https://www.pivotaltracker.com/story/show/120319485

This story is essentially focused on making our Concourse logs public.

When you refer to "intermediate checksums for inputs and outputs" in regards to the pipeline, are you referring to specific Concourse resources used in tasks in the pipelines?

[RochesterinNYC]

@gberche-orange Still looking for some clarification on your comments regarding "intermediate checksums for inputs and outputs". Thanks.

[gberche-orange]

Sorry @RochesterinNYC for the late response on this (it took time us to set up our instance of buildpacks-ci in order to try to provide relevant comments).

/CC @o-orand

Having concourse logs public as described in https://www.pivotaltracker.com/story/show/120319485 indeed contributes to providing ways for the community to have more visibility and be more protected against 1ii the CF/pivotal pipelines were compromised and injected trojans into the S3 buckets. It's great to see progresses being made on this story, and the cloudfoundry/docs-buildpacks#71 to ease understanding buildpack ci.

Still looking for some clarification on your comments regarding "intermediate checksums for inputs and outputs".

I was thinking that if some more fingerprints were added into the buildpack-ci output logs, it would be easier for an independent party to scrutinize them and potentially identify a compromise, or instead feel confident that the pipeline is safe and hard to break into.

Taking a specific example of the httpd binary building with the following output in our pipeline (likely to correspond to https://buildpacks.ci.cf-app.com/pipelines/binary-builder/jobs/build-httpd/builds/1)

This would mean adding MD5 at the end of the steps

Downloading apr-1.5.2.tar.gz (100%)
Extracting apr-1.5.2.tar.gz into /tmp/x86_64-linux-gnu/ports/apr/1.5.2... OK
Running 'configure' for apr 1.5.2... OK
Running 'compile' for apr 1.5.2... OK
Running 'install' for apr 1.5.2... OK
Downloading apr-iconv-1.2.1.tar.gz (100%)
Extracting apr-iconv-1.2.1.tar.gz into /tmp/x86_64-linux-gnu/ports/apr-iconv/1.2.1... OK
Running 'configure' for apr-iconv 1.2.1... OK
Running 'compile' for apr-iconv 1.2.1... OK

TODO: display APR binary hash here

Extracting httpd-2.4.20.tar.bz2 into /tmp/x86_64-linux-gnu/ports/httpd/2.4.20... OK
Running 'configure' for httpd 2.4.20... OK
Running 'compile' for httpd 2.4.20... OK
Running 'install' for httpd 2.4.20... OK
Running 'archive' for httpd 2.4.20... OK

TODO: display httpd binary hash here

Overvall, as a follow-up of #5 (comment), I tried the exercise of:

• 1. printing the traceability chain for httpd of 2.4.20 which is the current version used in the buildpack, with the corresponding story and try to follow the documented chain of custody
• 2. evaluating the reproductibility of builds: i.e. is it possible for a 3rd party to reliably reproduce the binary builds from available data.

Results below:

---

1 binary traceability chain:

• sources hash (md5: 3ff4f00ee051cf5d5304b47b1bc418b8 corresponding to https://archive.apache.org/dist/httpd/httpd-2.4.20.tar.bz2 URL built in the binary-builder httpd recipe) are recorded into cloudfoundry/buildpacks-ci@cee826d (not cross-linked with the story in this case)
• source-to-binary logs are into https://buildpacks.ci.cf-app.com/pipelines/binary-builder/jobs/build-httpd/builds/80 (the URL is currently broken. Is it due to a concourse update/redeploy ?). Our local ci instance displays the following traces

  Source YAML:
---
- url: https://archive.apache.org/dist/httpd/httpd-2.4.20.tar.bz2
  sha256: 0e76a375ed3dbac636f50ac39de966ece443751fe4d62392f9a360a19d94d0da

• with httpd binary hash (binary-builder/httpd-2.4.20-linux-x64.tgz, md5: 22dcae98467c318a856589d171670a01 ) recorded as a buildpacks-ci repo git commit log
  • httpd binary hash (md5: 22dcae98467c318a856589d171670a01) are added into php-buildpack/manifest.yml commit (lacking traceability with the story)
  • AWS binary are at https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/httpd/httpd-2.4.20-linux-x64.tgz with expected content matching the md5 hash

---

2 versionning of the CI used to produced the binary, from a given build output referenced in the story:

• concourse output logs display the buidpacks-ci git hash (see screenshot197 so
  • with docker image resources hashes:
    • currently pointing to docker hub head, so it is hard to precisely know which version was used
    • docker resources pull traces in concourse logs are not archived within the builds
  • with concourse version number used: 1.2
    • concourse version is only displayed on the homepage but not archived in logs, so could have changed)

---

I'll try formalizing improvement suggestions soon as a result of this. I also vonlunteered to the buildpack interview next week, if ever this could be an opportunity to further exchange on this topic.

Thanks again for the great improvement work on buildpacks audit trace,

Guillaume & Olivier.

[gberche-orange]

Improvement suggestions on the traceability of source to binary:

• make it easier to navigate from pivotal tracker story to invidivual commits and hashes of source and binaries:
  • manually pushed commits to include the tracker reference
  • automatically pushed commit such as cloudfoundry/buildpacks-ci@c44c664 to include tracker reference
• archive build logs associated to released artifacts (e.g. on S3 along with binaries) to ensure logs are available independently of buildpack-ci instance refactoring (similarly to a long term archival of ubuntu packages such as php 5.5 its list of builds and individual build archive)
• make it easier for the community to reproduce the buildpack-ci, in whole or in parts, and in particular and independently verify the source to binary process (likely to depend on reproductible builds on prem:
  • record in the archived log the environment details: the concourse ci version used, the tasks' docker images hash. This may be potentially be a concourse improvement that could apply to other teams ci.
  • parametrize the buildpack-ci repo so that independent parties are able to run them

[geramirez]

@gberche-orange thanks for the feedback and improvements suggestions. I've added a story for us to review comments more closely.

[jchesterpivotal]

It turns out (unsurprisingly) that we're not the first to grapple with this.

See this long discussion by the Cargo team, including references to one design in the wild -- TUF ("The Update Framework") and Toto, which was apparently renamed Diplomat.

Of note, Steve Klabnik is a friend of Labs and lives in Brooklyn. The authors of the Diplomat paper are at NYU.

[keaty]

@gberche-orange we've made a lot of changes over the past year that address a lot of the original issues that you brought up in regards to traceability and auditing, ex:

• public CI builds
• tracker story IDs in automated commits
• build history for automated builds
• Non hard-coded parameterization for a lot of our CI configuration (courtesy of @o-orand)

What other concerns or missing functionalities are there in terms of traceability and auditing?