Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build fails due to missing resources #84

Open
IzzySoft opened this issue Nov 12, 2024 · 14 comments
Open

build fails due to missing resources #84

IzzySoft opened this issue Nov 12, 2024 · 14 comments
Assignees

Comments

@IzzySoft
Copy link

Trying to build the app using ./gradlew assembleRelease on Debian bookworm with JDK 17 fails, as some resources seem to be missing:

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:processReleaseResources'.
> A failure occurred while executing com.android.build.gradle.internal.res.LinkApplicationAndroidResourcesTask$TaskAction
   > Android resource linking failed
     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:43: AAPT: error: resource drawable/pkg_not_wet (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_not_wet) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:55: AAPT: error: resource drawable/pkg_wet (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_wet) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:65: AAPT: error: resource drawable/pkg_no_poop (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_no_poop) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:77: AAPT: error: resource drawable/pkg_poop (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_poop) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:89: AAPT: error: resource drawable/pkg_notes (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_notes) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/baby_manager.xml:100: AAPT: error: resource drawable/pkg_no_notes (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_no_notes) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/quick_timer_entry.xml:79: AAPT: error: resource drawable/pkg_no_notes (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_no_notes) not found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/quick_timer_entry.xml:91: AAPT: error: resource drawable/pkg_notes (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_notes) not
found.

     ERROR: /build/repo/app/build/intermediates/incremental/release/mergeReleaseResources/stripped.dir/layout/timeline_item.xml:66: AAPT: error: resource drawable/pkg_notes (aka eu.pkgsoftware.babybuddywidgets:drawable/pkg_notes) not found.

Could it be you forgot to check those in to this repo?

@MrApplejuice
Copy link
Collaborator

Likely the buildsystem is somewhat borked. Very likely in fact. I did not do a "from scratch" build in ages. I really really should get my build system in order ... 👀

Anyway, one problem is are the "non-free images" I am using from https://www.flaticon.com/ . I cannot add them to this repo according to their license. Therefore, one integral part of getting everything up and running is to get an API key from them so that the builder can download the resources required.

I kind of find this important to address and I am feeling down about the Google app store rejecting my breast icons all of a sudden, so you are in luck: I will try to fix this now!

@MrApplejuice MrApplejuice self-assigned this Nov 12, 2024
@IzzySoft
Copy link
Author

to get an API key from them so that the builder can download the resources required.

That will make reproducible builds (see Reproducible Builds, special client support and more at IzzyOnDroid – your app is listed there) impossible then, as builders wouldn't sign up for API keys. Maybe a better approach would be finding free replacements, e.g. at svgrepo.com (those are mostly licensed CC0 as far as I could see)?

so you are in luck: I will try to fix this now!

I didn't think I'd say "thank you, Google" one day and not meaning it as sarcasm 🤣 So I'll add a "Thank you, Paul" if you don't mind 😉

@MrApplejuice
Copy link
Collaborator

That will make reproducible builds (see Reproducible Builds, special client support and more at IzzyOnDroid – your app is listed there) impossible then, as builders wouldn't sign up for API keys.

That is very much true. When I created the app, I kind of realized that this was the only non-free aspect of it. On the other hand, I had to realize that creating my own icons or scraping open sources was very time intensive and just did not yield the same quality of results. Flaticon, even though it is non-free has a solid and appealing selection of icons and therefore saved me a lot of time making this app exist in the first place.

That said, for the builds, I am planning on creating a separate "source" for images based on a filler-texture. Then I can do full builds but I do not need to hook flaticon/freepik in the build system. This then opens up the possibility for alternative icon packs.

Alternative icon packs, however, have their own rats tail attached to them. svgrepo.com seems to be a pretty nice source, but requires CC attributions. Not a problem per-se, just a lot of work. So, getting alternative icon packs, or replacing the current one to get rid of the freepik-license would probably need to become its own issue.

There is also a completely different solution on the horizon that I discovered yesterday: The freepik api has changed once again, and it seems that the icon-downloads themselves are no longer hidden behind a api-key. This would be great: The images themselves are still protected by the freepik license itself (meh for an open source project, I know), but at the same time no API-key is needed for downloading them. I can actually investigate that solution before replacing the entire pictogram library.

@IzzySoft
Copy link
Author

That said, for the builds, I am planning on creating a separate "source" for images based on a filler-texture. Then I can do full builds but I do not need to hook flaticon/freepik in the build system. This then opens up the possibility for alternative icon packs.

And that way, also opens a way to get rid of the NonFreeAssets anti-feature: the app itself could include a libre icon pack – which then can optionally replaced via manual download (on-demand from within the app, or simply by placing them into a specified/configurable location). Feeding three birds with one basket, so to say: no problematic build dependency, the AF dropped, and more flexibility/customization. I'd like that approach 😉

svgrepo.com seems to be a pretty nice source, but requires CC attributions.

I was just browsing it yesterday as we need some icons "refreshed" here. All the icons I took a closer look at were licensed CC0 (aka "public domain"). Why would those require attribution? Of course YMMV, as other icons might use other licenses – I didn't check all 500k of their collection.

freepik

That sounds like a possibility as well – if a "central attribution" (like "icons are ⓒ xxx by freepik") suffices. Though that might still require NonFreeAssets then. For dropping that AF, the "multiple icon packs" approach would be the better fit. But yeah, both would work.

@MrApplejuice
Copy link
Collaborator

MrApplejuice commented Nov 14, 2024

Hey @IzzySoft - would you mind testing my branch #85 ? I updated the build-docs, added (working!) GitHub CI, and dropped the need of acquiring a freepik API key, but I am still relying on their resources.

However, I think that this should fully fix your issue?

@IzzySoft
Copy link
Author

"Install Android SDK Platform 33 (revision 3)" finished.

> Task :app:checkKotlinGradlePluginConfigurationErrors
> Task :app:generateAboutStrings UP-TO-DATE
> Task :app:invokeHelpMakefile FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:invokeHelpMakefile'.
> A problem occurred starting process 'command 'make''

The process installed NDK v.26.1.10909125 "Side by side". Any dependency I miss? OK, make, and obviously imagemagick – but doing this during build is a bad idea:

convert resources/help_images/screenshots/help_diaper_logging.png -crop "+0+170" -crop "-0-135" app/src/main/res/drawable/help_diaper_logging.png

I doubt that is deterministic (converting images rarely is, so better use the "final ones" directly). And if it's not, it will make Reproducible Builds impossible (which is the reason I try to build the app myself, see Reproducible Builds, special client support and more at IzzyOnDroid).

Ah, and then python3. Looking at your github workflow (which I thought was for the desktop app) I guess the next thing it will fail to find is pandoc then… aw, no, first pipenv, python3-urllib3… Might be a good idea to list the build requirements somewhere 😉

So at this point, build crashes with

cd resources/help_images \
&& python3 -m pipenv install --skip-lock -r requirements.txt \
&& python3 -m pipenv run python process_markdown.py /build/repo/resources/help_images/help.md /build/repo/app/src/main/res/values/help_strings.xml
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/usr/lib/python3/dist-packages/pipenv/__main__.py", line 4, in <module>
    cli()
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 696, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 621, in make_context
    self.parse_args(ctx, args)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 1018, in parse_args
    rest = Command.parse_args(self, ctx, args)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 875, in parse_args
    parser = self.make_parser(ctx)
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 821, in make_parser
    for param in self.get_params(ctx):
  File "/usr/lib/python3/dist-packages/pipenv/vendor/click/core.py", line 774, in get_params
    help_option = self.get_help_option(ctx)
  File "/usr/lib/python3/dist-packages/pipenv/cli.py", line 26, in get_help_option
    from .import core
  File "/usr/lib/python3/dist-packages/pipenv/core.py", line 21, in <module>
    import requests
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/__init__.py", line 65, in <module>
    from . import utils
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/utils.py", line 27, in <module>
    from .cookies import RequestsCookieJar, cookiejar_from_dict
  File "/usr/lib/python3/dist-packages/pipenv/vendor/requests/cookies.py", line 172, in <module>
    class RequestsCookieJar(cookielib.CookieJar, collections.MutableMapping):
AttributeError: module 'collections' has no attribute 'MutableMapping'
make: *** [help.makefile:23: app/src/main/res/values/help_strings.xml] Error 1
make: Leaving directory '/build/repo'

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:invokeHelpMakefile'.
> Process 'command 'make'' finished with non-zero exit value 2

And I haven't yet installed what I dug up from the requirement.txt files I found so far: panflute pandoc PyYAML (for the latter, I don't know which Debian package would be the matching one).

So simply running ./gradlew assembleRelease does not seem to do the job here, despite of having installed additional Debian packages:

extra_packages:
  - make
  - imagemagick
  - python3
  - pipenv
  - python3-urllib3

What build process would you advise then? In case it matters: the pod for building your app is using ubuntu:jammy here, with OpenJDK-21.

@MrApplejuice
Copy link
Collaborator

Hey! Sorry, I got very ill since yesterday.

One very quick pointer: https://github.com/babybuddy/babybuddy-for-android/blob/6588135e445aaf8fb3a4b516089b7030c43c1d83/README.md

I updated the readme with the required packages list, had to determine the list myself again. I am still running all of this on ubuntu 22.04.5, but annoyingly, some of the Android features rely on JDK 18, not 21, so a custom JDK is required.

Rest needs to wait until I am healthy again.

@IzzySoft
Copy link
Author

some of the Android features rely on JDK 18, not 21

Oof. That will make it unfit for RB then. We only support long-term JDKs (11, 17, 21 currently – with 17 being the default for now). Features being dropped with later releases are one of the reasons to not use such interim-JDKs for "production".

Do you see any chance getting rid of that "unstable dependency", making the app build fine with 21 directly (i.e. not requiring JDK-18)?

@IzzySoft
Copy link
Author

IzzySoft commented Dec 7, 2024

some of the Android features rely on JDK 18, not 21

which btw also tells you why you should not rely on such in-between versions but rather use LTS releases (currently 11, 17, 21) as reference for what features to use. Features introduced by those "experimental" JDKs might be dropped again once the next LTS is released…

@MrApplejuice
Copy link
Collaborator

Quick update from my side: I have recovered already and I did not forget about this. I tried to update to JDK 21, but I cannot get the builds to work. There seems to be "some weirdness" with how Android handles the JDK versions: https://developer.android.com/build/jdks . Took me some time to find this, I dug myself to that page via the gradle-docs, which was very convoluted.

As you correctly point out, Version 11 or 17 are the ones I should be going for. I hope that this will work, will test it over the next few days!

@IzzySoft
Copy link
Author

Best luck! 🤞

Btw, JDK-11 is quite old. Please try 17 first if 21 doesn't work for you.

@MrApplejuice
Copy link
Collaborator

JDK 17 it is, #85 is merged. I think at least parts (huge parts) of your issues should have been addressed.

Please check the software list on https://github.com/babybuddy/babybuddy-for-android?tab=readme-ov-file#build-instructions, or alternatively the ci.yml file https://github.com/babybuddy/babybuddy-for-android/blob/master/.github/workflows/ci.yml. It looks quite promising that I was able to string things up so that they can be built here in GitHub now. I think this should mostly work.

I am not 100% sure if I can support 100% reproducible builds, I do not know the android toolchain well enough for that.

@IzzySoft
Copy link
Author

Thanks, on it now. Seems your check script needs an update:

Check if python3, cython, and pipenv are installed...
cython is not installed

it's cython3 here (debian:bookworm as well as ubuntu:jammy). I see ubuntu:noble has a symlink for it; as that's what your build uses, I can switch to that instead. Getting past that point: your project is the first one (out of 370+) requiring me to explicitly set JAVA_HOME (as per check-build-prerequisites.sh) – and then fails with Android SDK for targetSdk 35 not found in /opt/sdk (which is AFAIK set up by gradle at a later point – that's why it's defined in app/build.gradle – same for NDK, which you did not define in build.gradle, nor anywhere else, so which NDK is needed?)

OK, so thanks to that check script I was able to complete most dependencies. Going by my assumption that gradle takes care for the rest, I now comment it out to see if the build works.

Checking the license for package NDK (Side by side) 27.0.12077973 in /opt/sdk/licenses
License for package NDK (Side by side) 27.0.12077973 accepted.
Preparing "Install NDK (Side by side) 27.0.12077973 v.27.0.12077973".

So yes, it does (also Android SDK for targetSdk 35) – but build fails:

convert-im6.q16: delegate failed `'rsvg-convert' -o '%o' '%i'' @ error/delegate.c/InvokeDelegate/1997.
convert-im6.q16: unable to open file `/tmp/magick-vK4ak8z-bAIU88W4lVpxPnmPvuelwgW2': No such file or directory @ error/constitute.c/ReadImage/620.
convert-im6.q16: no images defined `resources/tmp/_round_icon_cutout.png' @ error/convert.c/ConvertImageCommand/3234.
make: *** [resources.makefile:167: resources/tmp/_round_icon_cutout.png] Error 1

> Task :app:invokeResourcesMakefile FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:invokeResourcesMakefile'.
> Process 'command 'make'' finished with non-zero exit value 2

What might be missing there? Here's my recipe:

---
repository: https://github.com/babybuddy/babybuddy-for-android.git
updates: releases
notes:
  - using ubuntu:noble due to the cython3 version required
versions:
  - tag: v2.4.3
    apks:
      - apk_pattern: babybuddy-for-android-.*\.apk
        apk_url: https://github.com/babybuddy/babybuddy-for-android/releases/download/$$TAG$$/babybuddy-for-android-$$TAG$$.apk
        build:
          - export JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
          - echo bash check-build-prerequisites.sh
          - ./gradlew assembleRelease -PskipIntegrationTests
          - find . -name '*.apk'
          - mv app/build/outputs/apk/release/*unsigned.apk /outputs/unsigned.apk
        build_home_dir: /build
        build_repo_dir: /build/repo
        build_user: build
        provisioning:
          android_home: /opt/sdk
          build_tools:
          cmake:
          cmdline_tools:
            version: '12.0'
            url: https://dl.google.com/android/repository/commandlinetools-linux-11076708_latest.zip
            sha256: 2d2d50857e4eb553af5a6dc3ad507a17adf43d115264b1afc116f95c92e5e258
          extra_packages:
            - python3
            - cython3
            - pipenv
            - pandoc
            - make
            - gcc
            - g++
            - curl
            - imagemagick
            - jq
          image: ubuntu:noble
          jdk: openjdk-17-jdk-headless
          ndk:
          platform:
          platform_tools:
          tools:
          verify_gradle_wrapper: true

Besides: it's a bad idea to convert images during build, that's almost sure to make RB fail (image conversion/compression usually is not deterministic), so better provide the "ready-to-use" images in the repo directly.

@IzzySoft
Copy link
Author

I've just tried v2.5.0 now. Build failed with:

convert-im6.q16: delegate failed `'rsvg-convert' -o '%o' '%i'' @ error/delegate.c/InvokeDelegate/1997.
convert-im6.q16: unable to open file `/tmp/magick--vwyV1j3IcqpibEKGtxs75xjzbG0xAVm': No such file or directory @ error/constitute.c/ReadImage/620.
convert-im6.q16: no images defined `resources/tmp/_round_icon_cutout.png' @ error/convert.c/ConvertImageCommand/3234.
make: *** [resources.makefile:167: resources/tmp/_round_icon_cutout.png] Error 1
make: Leaving directory '/build/repo'

> Task :app:invokeResourcesMakefile FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:invokeResourcesMakefile'.
> Process 'command 'make'' finished with non-zero exit value 2

As it requires a lot of believe to think a build involving images created on-the-fly with convert & co could be reproducible, I give up at this point, and remove babybuddy from my queue here. I leave the issue open in the hope you might one day have a build recipe with a chance to be RB; be welcome to ring my bell then.

All the best for your project nevertheless!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants