reproducable build fails although there is no diff

added bug reproducible-builds labels

I thought there was already an issue for this problem, but I couldn't find it. The open question is, which I'm sure I've asked before is:

What output should the buildserver give in this case?

Sounds to me like we should somehow publish the failed APK, then people can use whatever tools they want to check the diff. There is no existing publishing path for this kind of thing, but I don't think it would be hard to include. Another option would be to use diffoscope to dump the diff to the build log, with @uniqx's !515 (merged) to publish the complete build log.

@grote @Bubu @uniqx @sysrqb

changed milestone to %2.0

Didn't we already publish diffoscope output to the wiki? Or did we not install diffoscope on the new buildserver?

diffoscope isn't installed on the buildserver, I think its running Ubuntu/xenial, so its not easy to install it there. But I see in !505 (merged) that @grote added support for diffoscope, so if I remember correctly, the verify process runs in the buildserver VM at the end of a successful call to fdroid build. So if a build recipe includes this, then diffoscope should run:

  sudo: apt-get -qy install diffoscope/stretch-backports

Support for diffoscope was there long before !505 (merged) where I just fixed reproducibility bugs.

Can't we just do a pip install diffoscope on the buildserver? Since our goal is to build all apps reproducible, I don't see a point to install it only for individual apps.

Because changes to the default buildserver need a lot more testing than changes to a single job. pip install diffoscope won't work since diffoscope relies on lots of helper tools like apktool, enjarify, jar, xxd, etc. in order to show useful output. Which reminds me, a safer bet for the sudo line would be:

  sudo: apt-get -qy install --install-recommends -t stretch-backports diffoscope

mentioned in commit fdroiddata@0be513bf

I'm testing this with info.guardianproject.locationprivacy, the first attempt failed, the install got stuck. Here's the second attempt: fdroiddata@8c619a77

I also met the issue of empty "Unexpected diff output" when built locally.

WARNING: Ignoring META-INF/MANIFEST.MF from unsigned/xyz.jienan.xkcd_57.apk
DOES NOT VERIFY
ERROR: JAR signer CERT.RSA: JAR signature META-INF/CERT.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?
ERROR:
/var/folders/c7/66kjtx_x37j8qbk9crhy7yy00000gp/T/tmptr7t_y3t/sigcp_xyz.jienan.xkcd_57.apk:
INFO: ...NOT verified - /var/folders/c7/66kjtx_x37j8qbk9crhy7yy00000gp/T/tmptr7t_y3t/sigcp_xyz.jienan.xkcd_57.apk
DEBUG: > diff -r /var/folders/c7/66kjtx_x37j8qbk9crhy7yy00000gp/T/tmptr7t_y3t/unsigned_binaries_xyz.jienan.xkcd_57.binary /var/folders/c7/66kjtx_x37j8qbk9crhy7yy00000gp/T/tmptr7t_y3t/_var_folders_c7_66kjtx_x37j8qbk9crhy7yy00000gp_T_tmptr7t_y3t_sigcp_xyz.jienan.xkcd_57
DEBUG: removing unsigned/xyz.jienan.xkcd_57.apk
DEBUG: removing unsigned/binaries/xyz.jienan.xkcd_57.binary.apk
ERROR: Could not build app xyz.jienan.xkcd: compared built binary to supplied reference binary but failed
==== detail begin ====
Unexpected diff output:
==== detail end ====
INFO: Finished
INFO: 1 build failed

I suspect the way to create apk with ZipFile.writestr will not apply to the integrity check of v2 signature scheme?

Now I disabled v2 signature, then fdroid build succeed.

mentioned in merge request fdroiddata!5690 (closed)

@zjn0505 thanks for the hint! You might have found the issue.

AFAIK, we still don't support v2 sig verification, right @eighthave? But still this used to work. I wonder if something was added to CERT.SF that somehow forces v2 sig verification?

Did we maybe move from jarsigner to apksigner for this check? The latter has what they call rollback protection which prevents verification to succeed if there was a v2 or v3 signature before.

Hmm indeed the APK downloaded from oeffi.schildbach.de contains X-Android-APK-Signed: 2 in its META-INF/CERT.SF. Shall I provide a V1-only signed APK instead?

Just in case you think that I need to provide V1-only sigs, here is a patch that adapts the binaries URL:

fdroiddata!5710 (merged)

mentioned in merge request fdroiddata!5710 (merged)

Looks like the MR, which was merged tonight, fixed the build. The missing 10.5.5 and 10.5.6 were built and the build log indicates success. The versions still have to appear in the main repo though.

Good to know, we have not yet supported anything about v1-only signatures for reproducible builds. #634 (closed) It shouldn't be too hard to support the newer signatures, for me it hasn't been a priority since it won't really improve the security in F-Droid since the signed index already provides a signature that covers the whole APK.

Also, FYI, the reproducible issue with Location Privacy is related to diffs from building with the Android SDK from Google vs. from Debian packages.

Honestly I think the security of v1 sigs is so bad that we could also simply remove them entirely.

What location privacy issue are you referring to? I don't get this part.

This? https://f-droid.org/wiki/page/info.guardianproject.locationprivacy/lastbuild_31

@eighthave until we have v2 sig support, should we maybe go back to using jarsigner for verification of repro builds, since it doesn't seem to support this rollback protection and thus should continue to work with APKs that also have v2/v3 sigs?

As far as I know, apksigner is not currently in use on the f-droid.org infrastructure. Also, apksigner will verify v1-only APKs just fine. The problem is that APKs with v2+ signatures cannot be reproduced yet, therefore always fail verification with either jarisgner or apksigner.

Also, I think the best way forward for fdroidserver is to do signature verification using pure python #94, only use apksigner for signing, then drop jarsigner support entirely. That will also mean fdroidserver will no longer depend on the JDK, only the JRE (for keytool and apksigner).

As far as I know, apksigner is not currently in use on the f-droid.org infrastructure.

That's interesting. Then I have no idea why things stopped working.

The problem is that APKs with v2+ signatures cannot be reproduced yet, therefore always fail verification with either jarisgner or apksigner.

That used to work though. We have v2 signed APKs in the index that used to verify just fine. So something must have changed in the meantime.

I think the best way forward for fdroidserver is to do signature verification using pure python #94

That sounds slightly more long-term that what I had in mind. Also, because there doesn't seem to be a maintained library doing it, just some code snippets here and there.

Because apksigner includes an extensive collection of test APKs, turning one of those snippets of code into usable verifier should not be hard. Also, consider that fdroid only verifies APK signatures as a smoke check and usability enhancement. Verifying APK signatures on the server is not part of the security model.

Verifying APK signatures on the server is part of the security model when it comes to verifying reproducible builds, right? I mean that's how it is done. We transplant the signature from the reference APK to the one we built ourselves and then see if it still verifies. We could also take other approaches like just relying on empty diff output which are probably less secure.

However, for fixing reproducible builds in the short term, it might be useful to know what changed on the server as things used to work fine with v2 sigs. My guess would have been that we moved to apksigner, but you say we don't use it. So maybe jarsigner was upgraded to do v2 checking with rollback protection?

According to the rollback protection docs only Android 7.0 and later are checking the v2 signature, so maybe we can fix this issue short-term, by setting the --max-sdk-version to one level before Android 7.0.

@zjn0505 in your tests, did you use the official buildserver VM or build this locally? If locally, did you have apksigner installed? If so, can you maybe try what happens with only jarsigner available?

In the aftermath of this issue, I noticed that this problem started to appear after I switched from Gradle 3.4.1 to 4.4.1. It might be conincidence, but I thought it's worth mentioning. The Android gradle plugin version didn't change though, it has been at 2.3.3 for a long time.

Generally, I think we should do away with the "Binaries" method of downloading an APK from an URL while building reproducibly. It's unreliable anyway, and we shouldn't rely on an Internet connection. Rather, all reproducible builds (be it developer signed or both developer and F-Droid signed) should be migrated to putting the signature files into the metadata. And that method was always sensitive to the version of signature used.

I'd like to try reproducing this issue locally. @eighthave where can I find out about the OS and packages installed on the buildserver so I can reproduce the environment as closely as possible?

@akwizgran https://gitlab.com/fdroid/fdroidserver/-/blob/master/buildserver/provision-apt-get-install#L50

@akwizgran if you really want to go into that rabbit hole my recommendation is to use the actual build server VM instead of trying to install the packages locally. https://gitlab.com/fdroid/fdroidserver/-/blob/master/buildserver/Vagrantfile

I guess https://gitlab.com/fdroid/fdroid-bootstrap-buildserver is not used yet in production, right?

changed milestone to %2.1

what is best way to reproduce issue?

i will do https://gitlab.com/fdroid/fdroid-bootstrap-buildserver later but also am interested in knowledge is this in use or not

You could try building de.schildbach.oeffi (latest version). It should succeed. Then, edit metadata/de.schildbach.oeffi.yml and strip the -v1sig from the Binaries: https://oeffi.schildbach.de/downloads/oeffi-%v-v1sig.apk line. If I re-read the above ticket correctly, it should exhibit the problem with the empty diff.

We're seeing this issue again with the latest release of Briar (1.2.9).

https://f-droid.org/wiki/page/org.briarproject.briar.android/lastbuild_10209

I believe the issue re-emerged as a result of this commit:

06766ba4

When the build server signs an APK, the new find_apksigner() method finds the latest version of apksigner and adds it to the config. If the same build server subsequently tries to verify a reproducibly built APK, it finds apksigner in the config and uses it instead of jarsigner. If the reference APK has a v2 signature, this is not carried across to the reproduced APK, so apksigner rejects the reproduced APK with the following error:

2020-09-17 10:59:22,495 DEBUG: buildserver > 2020-09-17 09:59:22,490 INFO: Using /home/vagrant/android-sdk/build-tools/29.0.3/apksigner
2020-09-17 10:59:23,356 DEBUG: buildserver > DOES NOT VERIFY
2020-09-17 10:59:23,380 DEBUG: buildserver > ERROR: JAR signer CERTIFIC.RSA: JAR signature META-INF/CERTIFIC.SF indicates the APK is signed using APK Signature Scheme v2 but no such signature was found. Signature stripped?

The only difference between the reference and reproduced APKs is the v2 signature. The zip entries of the APKs are identical, so the diff output is empty.

2020-09-17 10:59:24,373 DEBUG: buildserver > 2020-09-17 09:59:24,368 ERROR: Could not build app org.briarproject.briar.android: compared built binary to supplied reference binary but failed
2020-09-17 10:59:24,373 DEBUG: buildserver > ==== detail begin ====
2020-09-17 10:59:24,374 DEBUG: buildserver > Unexpected diff output:
2020-09-17 10:59:24,374 DEBUG: buildserver > ==== detail end ====

However, if the build server tries to verify the same APK before find_apksigner() has been called, it doesn't find apksigner in the config and therefore uses jarsigner for the verification. This checks the v1 signature, which is included in the reproduced APK, so the verification succeeds.

2020-09-16 10:19:39,458 DEBUG: buildserver > 2020-09-16 09:19:39,458 WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
2020-09-16 10:19:40,810 DEBUG: buildserver > 2020-09-16 09:19:40,806 DEBUG: JAR signature verified: /tmp/tmp8wym_9z9/sigcp_org.briarproject.briar.android_10209.apk
2020-09-16 10:19:40,810 DEBUG: buildserver > 2020-09-16 09:19:40,807 INFO: ...successfully verified

A possible workaround for this regression would be to modify find_apksigner() so it doesn't save the location of apksigner in the config. This would avoid unwanted interactions between signing and verification, but it would make subsequent calls to find_apksigner() more expensive.

I can confirm that @grote's suggested workaround of setting --max-sdk-version=23 for apksigner in verify_apk_signature() also works.

2020-09-17 11:44:59,783 DEBUG: buildserver > 2020-09-17 10:44:59,778 INFO: Using /home/vagrant/android-sdk/build-tools/29.0.3/apksigner
2020-09-17 11:45:00,968 DEBUG: buildserver > 2020-09-17 10:45:00,962 DEBUG: /tmp/tmp05wavkjd/sigcp_org.briarproject.briar.android_10209.apk: Verifies
2020-09-17 11:45:00,969 DEBUG: buildserver > Verified using v1 scheme (JAR signing): true
2020-09-17 11:45:00,969 DEBUG: buildserver > Verified using v2 scheme (APK Signature Scheme v2): false
2020-09-17 11:45:00,969 DEBUG: buildserver > Verified using v3 scheme (APK Signature Scheme v3): false
2020-09-17 11:45:00,969 DEBUG: buildserver > Number of signers: 1

[Snipped: a bunch of apksigner warnings about files in META-INF not being protected by the v1 signature]

2020-09-17 11:45:00,971 DEBUG: buildserver > 
2020-09-17 11:45:00,971 DEBUG: buildserver > 2020-09-17 10:45:00,962 INFO: ...successfully verified

I'm happy to submit a patch for either of these workarounds.

@akwizgran that seems like a correct analysis.

I believe the cleaner solution here is to explicitly force verification of reproducibly built apps to jarsigner, they can never work with apksigner right now anyway because the v2 signature isn't copied to the new build (yet, hopefully).

This would then still use v2 verification for other apks (which your solution #2 (closed) wouldn't) and also avoids this breaking when apksigner is globally installed on the signing machine where it would be used for verification anyway.

fdroiddata#1630 (comment 413767392) has a patch, could you confirm this makes sense? conceptually and in practice?

Or indeed we could set the maxsdk verification option for apksigner for the codepath coming from verify_apks() only. That should have the same effect as switching to jarsigner there and might be the cleaner solution by only depending on apksigner.

@Bubu your solution from fdroiddata#1630 (closed) works:

2020-09-17 12:12:58,105 DEBUG: buildserver > 2020-09-17 11:12:58,104 WARNING: Ignoring META-INF/MANIFEST.MF from unsigned/org.briarproject.briar.android_10209.apk
2020-09-17 12:13:00,549 DEBUG: buildserver > 2020-09-17 11:13:00,547 DEBUG: JAR signature verified: /tmp/tmp9gr1yfeq/sigcp_org.briarproject.briar.android_10209.apk
2020-09-17 12:13:00,549 DEBUG: buildserver > 2020-09-17 11:13:00,548 INFO: ...successfully verified
2020-09-17 12:13:00,550 DEBUG: buildserver > 2020-09-17 11:13:00,549 INFO: compared built binary to supplied reference binary successfully
2020-09-17 12:13:00,555 DEBUG: buildserver > 2020-09-17 11:13:00,555 INFO: success: org.briarproject.briar.android

I think this is better than setting max-sdk-version because it ensures we always use jarsigner regardless of whether a previous build has called find_apksigner().

@akwizgran I cannot currently submit this as a MR myself, feel free to take my patch, preferably add a regression test and MR that.

mentioned in merge request !804 (merged)

closed with merge request !804 (merged)

mentioned in commit b5ce285f

I'm afraid the issue is not fixed. See https://f-droid.org/wiki/page/de.schildbach.oeffi/lastbuild_682

reopened

I should add that the latest occurence was probably triggered by fdroiddata@ba9ec2a7 – anyhow the output still doesn't make any sense. Either there is a diff, then it should be listed. Or there is no diff, then the build should not fail.

I'm afraid the issue is not fixed. See https://f-droid.org/wiki/page/de.schildbach.oeffi/lastbuild_682

Check the fdroidserver commit hash at the top of that page. After merging to master, deploys to production usually take some extra time.

Hmm, two weeks later it's still using fdroidserverid 1f7228d5. Is there something I need to do to use the recent version?

@schildbach add your voice here: admin#128 (closed)

Hasn't Briar already been published using a v1+v2 signature thanks to this patch?

Not recently, no: https://f-droid.org/wiki/page/org.briarproject.briar.android

The reason for the empty diff, by the way, is that if diffoscope isn't installed on the build server, compare_apks() compares the zip contents of the APKs. If the only difference between the APKs is that one has a v2 signature and the other doesn't, then the diff will be empty as the v2 signature is stored in zip metadata.

So an empty diff indicates that a v1 signature check would have succeeded but a v2 signature check was used instead, which in turn indicates that a v2 signature was present and apksigner was used for verification (rather than jarsigner, which would have ignored the v2 signature and accepted the v1).

mentioned in commit fdroiddata@f10b2c59

Is this still an issue now that we support v2/v3 signatures?

reproducable build fails although there is no diff

Child items ...

Activity