Simplify packit custom create archive command #249

psss · 2020-05-21T05:10:24Z

No description provided.

psss · 2020-05-21T05:16:10Z

@lachmanfrantisek, is there anything wrong with the config?

The create-archive action did not output a path to the generated archive. Please make sure that you have valid path in the single line of the output.

The full path to the archive is printed as the last line:

> make tarball
...
rst2man /home/psss/git/tmt/tmp/man.rst > /home/psss/git/tmt/tmp/tmt-0.16/tmt.1
cd /home/psss/git/tmt/tmp && tar cfz SOURCES/tmt-0.16.tar.gz tmt-0.16
/home/psss/git/tmt/tmp/SOURCES/tmt-0.16.tar.gz

lachmanfrantisek · 2020-05-21T07:00:49Z

It works nicely locally. So there is some problem with the service. I'll create an issue.

Sorry for this occurring problem.

lachmanfrantisek · 2020-05-21T07:07:37Z

Potential problem:

you run action in the sandbox and print absolute path
we check the output outside of the sandbox and the absolute path can differ

(You can try using relative paths, but I still create an issue for this.)

psss · 2020-05-21T10:35:57Z

@lachmanfrantisek, thanks for looking into this. The relative path does not work:

error: File ./tmt-0.16.tar.gz: No such file or directory

I've tried both tmp/SOURCES/... and ./tmp/SOURCES/.... Seems the only currently working way is to move the created archive to the top directory.

lachmanfrantisek · 2020-05-22T06:50:12Z

Thank you for testing all variants. We need to fix that on our side. There might a problem with the subdirectory.

psss · 2020-05-22T13:50:13Z

@lachmanfrantisek, thanks for looking into this. Let me know when I can test it again.

psss · 2020-07-24T13:58:55Z

/packit build

psss · 2020-07-24T14:01:59Z

@lachmanfrantisek, has there been any progress to make this working?

lachmanfrantisek · 2020-07-28T09:34:10Z

@lachmanfrantisek, has there been any progress to make this working?

Unfortunately no, sorry. But @TomasTomecek is now working on the shell expansion for commands and is having some problems that can be related to this.

TomasTomecek · 2020-07-29T06:32:35Z

/packit build

TomasTomecek · 2020-07-29T06:37:00Z

It seems that the archive is in ./tmp/SOURCES/tmt-0.18.tar.gz and not ./tmt-0.18.tar.gz.

psss · 2020-07-29T07:37:06Z

@TomasTomecek, yes, that was the intention: To simplify Makefile so that moving the archive is not necessary. Providing the full/relative path to the archive in the output should be sufficient for Packit to find it.

TomasTomecek · 2020-07-29T07:49:39Z

@TomasTomecek, yes, that was the intention: To simplify Makefile so that moving the archive is not necessary. Providing the full/relative path to the archive in the output should be sufficient for Packit to find it.

So you're saying that the build should pass with the current changes and the fact the SRPM can't be created is wrong?

psss · 2020-07-29T08:52:55Z

@TomasTomecek, that's what I've understood from our discussion with @lachmanfrantisek: That if the custom archive command prints out a valid path the to archive it should just work. Which makes sense to me. Otherwise users are forced to move the archive to proper location. Fixing this in one place (packit) seems to me to be a better option than duplicating this unnecessary action again and again in individual Makefiles.

lachmanfrantisek · 2020-07-29T09:28:43Z

@TomasTomecek We are creating the symlink, but it does not work for some reason in the service:

Using user-defined script for ActionName.create_archive: [['make', 'tarball']]
Running command: make tarball
06:33:42.377 base_git.py       DEBUG  Action command output: ['rm -rf /sand...
Created archive: /sandcastle/tmp/SOURCES/tmt-0.18.tar.gz
Linking to the specfile directory: /sandcastle/tmt-0.18.tar.gz
:
Running command: rpmbuild -bs --define _sourcedir . --define _srcdir . --define _specdir . --define _srcrpmdir . --define _topdir . --define _builddir . --define _rpmdir . --define _buildrootdir . tmt.spec

Message: An unexpected error occurred when creating the SRPM: Command failed (rc=1, reason={"metadata":{},"status":"Failure","message":"command terminated with non-zero exit code: Error executing in Docker Container: 1","reason":"NonZeroExitCode","details":{"causes":[{"reason":"ExitCode","message":"1"}]}})
error: File ./tmt-0.18.tar.gz: No such file or directory

TomasTomecek · 2020-07-29T14:59:13Z

I suspect the symlinks are not being propagated well b/w sandbox and worker.

Anyway, I'm really confused here. I can see the tarball path being printed:

cd /sandcastle/docker-io-usercont-sandcastle-prod-20200729-063317269798/tmp && tar cfz SOURCES/tmt-0.18.tar.gz tmt-0.18
./tmp/SOURCES/tmt-0.18.tar.gz\n']

So how come that packit goes for error: File ./tmt-0.18.tar.gz: No such file or directory and not for ./tmp/SOURCES/....

We should probably add some debug prints in packit codebase whether it's able to find the archive. We'd also know if the symlinking works.

mfocko · 2020-07-30T18:52:25Z

I've looked around it a bit; enforced same handling of paths as in service.

So how come that packit goes for error: File ./tmt-0.18.tar.gz: No such file or directory and not for ./tmp/SOURCES/....

@TomasTomecek That is because the build command is called with every directory set as . (when ran locally produces absolute path, when ran in service uses .).

Tried to break it in multiple ways:

the archive must be accessible (as in: was accessible when it was built) at the location in packit script, otherwise it fails (e.g. built archive name != printed path)
from that I conclude the symlink is created (since the path to archive is correct)
- not excluding possibility of the symlink being in wrong directory (thus making the relative path incorrect)
I also noticed that Makefile contains TMP variable that is used when creating tarball, but the echo with path to archive doesn't use it. (The version before ~~changes~~ this PR uses mv from specifically $(TMP)/SOURCES/... to .)
@psss Have you tried it this way? If not, could you please try?
```
- @echo ./tmp/SOURCES/$(PACKAGE).tar.gz
+ @echo $(TMP)/SOURCES/$(PACKAGE).tar.gz
```

psss · 2020-08-03T12:50:44Z

Trying with @echo $(TMP)/SOURCES/$(PACKAGE).tar.gz to print the path.

psss · 2020-08-03T12:55:14Z

Seems this did not help either. The resulting path was:

/sandcastle/docker-io-usercont-sandcastle-prod-20200803-125030982828/tmp/SOURCES/tmt-0.20.tar.gz

But:

Message: Preparing of the upstream to the SRPM build failed:
The create-archive action did not output a path to the generated archive.
Please make sure that you have valid path in the single line of the output.

mfocko · 2020-08-04T11:39:08Z

Sorry for the confusion, packit requires relative path and the $(TMP) got expanded into absolute path.

I think it would be best to continue with ./tmp/SOURCES/$(PACKAGE).tar.gz, since it passed the file-existence check in packit and focus on the symlinking on our side (I have packit/packit#923 ready to see what's going on with the relative/absolute paths while linking and checking existence).

Also would you like to try stage version (more info: packit/packit-service#649) of packit? It would help with catching problems early and speed up debugging of problems like this. :)

psss · 2020-08-04T14:49:13Z

Ok, I'll revert the change to use relative path. I've also enabled the staging instance. Let's see...

mfocko · 2020-08-04T14:53:48Z

Thanks. PR with extended logging is not merged yet, I'll trigger rebuild when it gets merged.

mfocko · 2020-08-05T11:18:31Z

/packit build

packit-as-a-service-stg · 2020-08-05T11:18:35Z

Only users with write or admin permissions to the repository can trigger Packit-as-a-Service

packit-as-a-service · 2020-08-05T11:18:36Z

Only users with write or admin permissions to the repository can trigger Packit-as-a-Service

lachmanfrantisek · 2020-08-05T11:21:46Z

/packit build

TomasTomecek · 2020-08-05T12:19:15Z

we should make @mfocko packit admin :)

it seems the change you did in packit is not deployed to stg yet, let me retrigger stg worker image build

mfocko · 2020-08-06T08:33:57Z

/packit build

packit-as-a-service · 2020-08-06T08:34:02Z

Only users with write or admin permissions to the repository can trigger Packit-as-a-Service

packit-as-a-service-stg · 2020-08-06T08:34:02Z

Only users with write or admin permissions to the repository can trigger Packit-as-a-Service

psss · 2020-08-06T08:41:36Z

/packit build

TomasTomecek · 2020-08-06T09:37:14Z

Wow, passed! are we good here? I just can't believe it that the fix was so "easy"

psss · 2020-08-06T11:29:24Z

Nice! Looks good :)

psss · 2020-08-10T08:55:49Z

@lachmanfrantisek, @TomasTomecek, @mfocko, works nice in production as well. Thanks for fixing this!

TomasTomecek · 2020-08-11T07:35:02Z

@psss thank you for your patience and working with us!

psss added the tests label May 21, 2020

psss mentioned this pull request May 21, 2020

Make it easier to supply custom archive to packit srpm packit/packit#566

Closed

lachmanfrantisek mentioned this pull request May 21, 2020

Getting archive path from user action does not work in service packit/packit-service#621

Closed

psss force-pushed the simplify-packit branch 2 times, most recently from 4151f72 to 9c820af Compare May 21, 2020 10:27

psss force-pushed the simplify-packit branch from 9c820af to d04bb9b Compare June 8, 2020 14:25

psss force-pushed the simplify-packit branch from d04bb9b to 923d921 Compare August 3, 2020 12:49

psss mentioned this pull request Aug 3, 2020

Enable copr builds and add packit config psss/edd#10

Open

psss force-pushed the simplify-packit branch from 923d921 to 37f8f09 Compare August 4, 2020 14:50

Simplify packit custom create archive command

c9089f5

psss force-pushed the simplify-packit branch from 37f8f09 to c9089f5 Compare August 10, 2020 08:29

psss self-assigned this Aug 10, 2020

psss merged commit c9089f5 into master Aug 10, 2020

psss deleted the simplify-packit branch August 10, 2020 08:56

mfocko mentioned this pull request Aug 18, 2020

[create-archive] Symlinking absolute path packit/packit#942

Closed

psss added the tests label Dec 9, 2020

Simplify packit custom create archive command #249

Simplify packit custom create archive command #249

Conversation

psss commented May 21, 2020

psss commented May 21, 2020

lachmanfrantisek commented May 21, 2020

lachmanfrantisek commented May 21, 2020 • edited Loading

psss commented May 21, 2020

lachmanfrantisek commented May 22, 2020

psss commented May 22, 2020

psss commented Jul 24, 2020

psss commented Jul 24, 2020

lachmanfrantisek commented Jul 28, 2020 • edited Loading

TomasTomecek commented Jul 29, 2020

TomasTomecek commented Jul 29, 2020

psss commented Jul 29, 2020

TomasTomecek commented Jul 29, 2020

psss commented Jul 29, 2020

lachmanfrantisek commented Jul 29, 2020

TomasTomecek commented Jul 29, 2020

mfocko commented Jul 30, 2020 • edited Loading

psss commented Aug 3, 2020

psss commented Aug 3, 2020

mfocko commented Aug 4, 2020

psss commented Aug 4, 2020

mfocko commented Aug 4, 2020

mfocko commented Aug 5, 2020

packit-as-a-service-stg bot commented Aug 5, 2020

packit-as-a-service bot commented Aug 5, 2020

lachmanfrantisek commented Aug 5, 2020

TomasTomecek commented Aug 5, 2020 • edited Loading

mfocko commented Aug 6, 2020

packit-as-a-service bot commented Aug 6, 2020

packit-as-a-service-stg bot commented Aug 6, 2020

psss commented Aug 6, 2020

TomasTomecek commented Aug 6, 2020

psss commented Aug 6, 2020

psss commented Aug 10, 2020

TomasTomecek commented Aug 11, 2020

lachmanfrantisek commented May 21, 2020 •

edited

Loading

lachmanfrantisek commented Jul 28, 2020 •

edited

Loading

mfocko commented Jul 30, 2020 •

edited

Loading

TomasTomecek commented Aug 5, 2020 •

edited

Loading