Provide and publish s390x images

[wohali]

A couple of requests have come in for s390x images, see #66 for instance.

This is blocked until:

• outstanding issues with CouchDB on s390x are resolved, such as JIRA COUCHDB_3261.
• Jenkins CI is updated to have an s390x build platform (currently, unavailable), see Add multi-platform support to CI configuration couchdb#1121.
• CI workflow is "all-green" on a regular basis
• PR Support multi-arch image building for couchdb docker image #60 is merged

[ivandov]

Any updates to this? An official image for s390x would be great to have.

[wohali]

@ivandov Unfortunately there are serious errors in compiling CouchDB correctly on S390X, as some unit tests fail. The problem is related to endianness in snappy, see https://issues.apache.org/jira/browse/COUCHDB-3261 for details.

We refuse to release a broken image on principle. Until an S390-supplied engineer is ready to step up and fix these issues properly we are stalled on this. No one on the CouchDB team is working this issue.

[ivandov]

@wohali thank you for that information. Let me forward this along to the Linux on Z "ecosystem" team, to see if they can look into a fix for this.

I am personally planning on creating an image for s390x following this guide: https://github.com/linux-on-ibm-z/docs/wiki/Building-CouchDB

It seems that the build process for s390x has already been verified - at least directly on a VM - I just need to make it into a portable image. Would you like to me to retry some test suites during the image build process?

[wohali]

@ivandov The modifications in that walkthrough are unacceptable. You can't just change the "right answer" in a test case without delving into why things are the way they are. For instance, your solution would not allow people to take .couch database files from an amd64 machine and transfer them to an s390x` machine; they would not be readable.

We will not release an s390x build until this problem is resolved.

Further, note #118 which now builds CouchDB in Docker using our .deb binary packages. We won't be building these on qemu with s390x until the build process is successful on an unmodified source set. The work for that happens over in the apache/couchdb-pkg repository.

[wohali]

To make it clear - don't waste your time building a Dockerfile for s390x that doesn't install from the .deb, we won't host it here or at the official couchdb location.

Fix the snappy code correctly, get CouchDB building and passing all tests without modification to our source code.

Then, review and help in the apache/couchdb-ci repo for installing dependencies for a Debian stretch-slim s390x system, since that is the selected base image for our Docker image. (We are in good company here, just about every other official Docker image for a database is based on this, not CentOS or Ubuntu.)

Then, we can update the couchdb-pkg repo with the modifications necessary for SpiderMonkey 1.8.5, any specific compile-time changes for CouchDB, and build the s390x Debian binaries in a qemu docker container.

At that point, this Dockerfile will just "work," we'll just enable manifest multi-arch building here which is entirely a build-time step.

[ivandov]

To be clear, I did not write the documentation that I linked, I was just planning on attempting to follow it to see if a Docker Image could successfully be built. I understood that would be a private Docker Image and not one that would contribute back to the multi-arch effort for the official couchdb Docker library image.

As I said, I will make sure the Linux on Z ecosystem team is aware of the blocking bug, and hopefully they can work on creating a fix to allow the unit tests to pass as they are currently written.

I'll follow-up soon.

[ivandov]

Also, I just read through the the JIRA issue you linked earlier: https://issues.apache.org/jira/browse/COUCHDB-3261

The final comment by Paul may be the best we can do for snappy support on s390x, because of the endian-ness issue, and as mentioned in that thread, this is a consistent problem across libraries that need cross-architecture support.

The requirement for a .couch file to be portable between architectures may never be able to be satisfied for binary files that have different byte ordering. However, this should not gate the ability to run CouchDB on a specific architecture as needed, and at least in this case, you could isolate s390x CouchDB deployments and still have something running there. That's already a step above where we stand today.

That JIRA thread sounds like it needs some updating / re-running for CouchDB 2.x, I will attempt to do that and open a new Github issue for tracking if needed.

[wohali]

@ivandov It really depends how far you s390x people want to go. snappy could be extended to handle reverse-endianness (with a performance hit) if desired. I could see it being an upselling point that you could "outgrow" your amd64/ppc64le Couch instances and want to move to s390x without having to replicate the data over, but instead just mounting your IBM Cloud storage device over to the new hardware node. But that's really your call.

For reference @davisp is also IBM so you can bug him if you want about what he thinks might be acceptable now - though that JIRA ticket you linked was at least 2.0.0 already, so there's not much different architecturally between then and the imminent 2.3.0 release.

Sorry for coming across angrily, I just don't want a slapped-together public image. I hope you understand and appreciate my attention to detail on this point :) Help in improving the apache/couchdb-ci and apache/couchdb-pkg repos to support s390x/debian stretch would still be the next steps after the snappy work is done. Thanks again!

[adpopescu]

Hi @wohali,

Any update on this?

It looks like most of the issues listed initially have been taken care of?

This is blocked until:

* outstanding issues with CouchDB on `s390x` are resolved, such as [JIRA COUCHDB_3261](https://issues.apache.org/jira/browse/COUCHDB-3261).

Looks like it was fixed with commit: 4dbd7c17e6e7e7894b64595b2b7fdde7ead823f6

* Jenkins CI is updated to have an `s390x` build platform (currently, unavailable), see [apache/couchdb#1121](https://github.com/apache/couchdb/issues/1121).

Issue 1121 seems to have been closed as well.

* CI workflow is "all-green" on a regular basis

I’ve built and run all the tests, on Ubuntu 16.04( as well as others, but not Debian), they all pass.

* PR #60 is merged

The PR was merged here?:

I think it's been a while since anyone has looked at this...so I'm wondering what would be needed to get version 3.x images for s390x?

Thank you

[wohali]

@adpopescu update:

I had missed the fix to the couch compress tests, that's good.

We don't have an s390x build target in Jenkins today. apache/couchdb#1121 and was tracking overall support for multi-platform building in our Jenkins environment - and was closed because that landed.

That's the next step. @davisp is in charge of the IBM/SoftLayer machines for us, and has helped us set up ppc64le and x86_64 targets there. You'll need to liaise with him (internal to IBM is fine) to get us an s390x run environment.

Parallel to that, I can build an s390x Docker image that contains our build chain. I'll start work on that this week.

Next step after that lands is to add s390x to our main platform matrix build here: https://github.com/apache/couchdb/blob/master/build-aux/Jenkinsfile.full Specifically, it'd make sense to clone this section.

Once that is passing and is green for a few commits, we can add s390x in this repo, and push support downstream to the main Docker repo.

Thanks for offering to lend a hand!

[wohali]

Our CI image, couchdbdev/s390x-debian-buster-erlang-20.3.8.25-1, is now available.

[SriramaSharma]

@wohali I believe late last year, Paul Davis from IBM had shared access to an Ubuntu Community cloud VM for s390x architecture, to be included as build node into the Jenkins CI/CD for community s390x builds. I gather that the VM is still up and active(online). Could you please confirm if the s390x builds are still running without any issues ? Please let know if any support is required in getting the s390x docker image published through CI.