Investigate vanilla tweaks file input mode

[itzg]

vanillatweaks_file test failed with a 403

Disabled test in #3142

https://github.com/itzg/docker-minecraft-server/actions/runs/11758152976/job/32756197269?pr=3142

[mc-image-helper] 17:33:43.066 ERROR : 'vanillatweaks' command failed. Version is 1.40.4
me.itzg.helpers.http.FailedRequestException: HTTP request of https://vanillatweaks.net/assets/server/zipcraftingtweaks.php failed with 403 Forbidden: Fetching object content
	at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ Resolving download link
Original Stack Trace:
		at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132)
		at reactor.core.publisher.FluxDefaultIfEmpty$DefaultIfEmptySubscriber.onNext(FluxDefaultIfEmpty.java:122)
		at reactor.core.publisher.FluxHandle$HandleSubscriber.onNext(FluxHandle.java:129)
		at reactor.core.publisher.FluxMap$MapConditionalSubscriber.onNext(FluxMap.java:224)
		at reactor.core.publisher.FluxDoFinally$DoFinallySubscriber.onNext(FluxDoFinally.java:113)
		at reactor.core.publisher.FluxHandleFuseable$HandleFuseableSubscriber.onNext(FluxHandleFuseable.java:194)
		at reactor.core.publisher.FluxContextWrite$ContextWriteSubscriber.onNext(FluxContextWrite.java:107)
		at reactor.core.publisher.Operators$BaseFluxToMonoOperator.completePossiblyEmpty(Operators.java:2071)
		at reactor.core.publisher.MonoCollectList$MonoCollectListSubscriber.onComplete(MonoCollectList.java:118)
		at reactor.core.publisher.FluxPeek$PeekSubscriber.onComplete(FluxPeek.java:260)
		at reactor.core.publisher.FluxMap$MapSubscriber.onComplete(FluxMap.java:144)
		at reactor.netty.channel.FluxReceive.onInboundComplete(FluxReceive.java:415)
		at reactor.netty.channel.ChannelOperations.onInboundComplete(ChannelOperations.java:445)
		at reactor.netty.channel.ChannelOperations.terminate(ChannelOperations.java:499)
		at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:800)
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:115)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:289)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1503)
		at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1366)
		at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1415)
		at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:530)
		at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:469)
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1357)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:868)
		at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:799)
		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:501)
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:399)
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:[74](https://github.com/itzg/docker-minecraft-server/actions/runs/11751871102/job/32745504295?pr=3142#step:6:75))
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Unknown Source)
	Suppressed: java.lang.Exception: #block terminated with an error
		at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:103)
		at reactor.core.publisher.Mono.block(Mono.java:1712)
		at me.itzg.helpers.vanillatweaks.VanillaTweaksCommand.call(VanillaTweaksCommand.java:127)
		at me.itzg.helpers.vanillatweaks.VanillaTweaksCommand.call(VanillaTweaksCommand.java:52)
		at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
		at picocli.CommandLine.access$1500(CommandLine.java:148)
		at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
		at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
		at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
		at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:22[77](https://github.com/itzg/docker-minecraft-server/actions/runs/11751871102/job/32745504295?pr=3142#step:6:78))
		at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
		at picocli.CommandLine.execute(CommandLine.java:2174)
		at me.itzg.helpers.McImageHelper.main(McImageHelper.java:161)

[litetex]

Same error here.
Looks like this was blocked by cloudflare (maybe "under attack mode" active?): I'm getting a challenge as HTTP answer on my server...

[calvinli723]

Similar error also happens to me when using share code.

[itzg]

I'm starting to wonder if they're disallowing automated access and instead requiring usage via their website...

[itzg]

FWIW using a sharecode worked fine for me using:

services:
  mc:
    image: itzg/minecraft-server
    environment:
      EULA: true
      VANILLATWEAKS_SHARECODE: KcuEN2

[litetex]

Still having the same problem 2 weeks later:

ERROR : 'vanillatweaks' command failed. Version is 1.40.2
me.itzg.helpers.http.FailedRequestException: HTTP request of https://vanillatweaks.net/assets/server/sharecode.php?code=TqqyFI failed with 403 Forbidden: Fetching object content
       at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
       Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
       *__checkpoint ⇢ Resolving share code
Original Stack Trace:
               at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
...

Notes: I'm trying to get the content via IPv6 on a server that is hosted by Hetzner.

Running curl yields a cloudflare challenge response:

curl -L https://vanillatweaks.net/assets/server/sharecode.php\?code\=TqqyFI
<!DOCTYPE html><html lang="en-US"><head><title>Just a moment...</title>
....
<span id="challenge-error-text">Enable JavaScript and cookies to continue</span></div>
...

When trying to connect to other websites that are hosted on Cloudflare I don't see any such errors.
This particular server also didn't connect to vanillatweaks for 2 weeks, so it's extermely likely that there is some kind of block in place.

Also had a look at there Discord but found no related information there either :(

Anyway I'm opening an issue on there feedback page now. If I don't forget I will post some updates here later ^^

[andre-paulo98]

VanillaTweaks dev here (got the link from @litetex feedback)

We didn't change anything in our side, I guess Cloudflare flagged certain those IPs as Bots and are enforcing the challenge.
We don't have an official API currently, and this project (and some others) are using the methods that the website uses to generate the packs.
We don't do anything to prevent this, but you might run into issues like this.

In the past we've allowed a hosting provider access to our website without having to deal with Cloudflare. Depending on the occasion, we might give access to it to other people (best if you ask about it on Discord)

Eventually we'll have an official API, but soonTM.

[itzg]

Thanks for the follow up @andre-paulo98 . That would explain why it's fairly sporadic when it succeeds or not. It worked fine with my recent, manual tests.

An official API would be fantastic!

[litetex]

@andre-paulo98

In the past we've allowed a hosting provider access to our website without having to deal with Cloudflare. Depending on the occasion, we might give access to it to other people (best if you ask about it on Discord)

I don't think that this is a good general solution as it would involve a lot of communication from various users that want to get whitelisted or wonder what's going on...

I guess Cloudflare flagged certain those IPs as Bots and are enforcing the challenge

I don't think so as other websites that are also hosted on cloudflare work completely fine. But it's cloudflare so who knows how it works ¯\_(ツ)_/¯

Spontaneous idea as I also have some experience with cloudflare:
You could maybe try adding a custom rule for the /assets/server/*.php path (looks like only this one is used for downloading) with a lower "security" level. If you use that functionallity in the first place...

[andre-paulo98]

I don't think that this is a good general solution as it would involve a lot of communication from various users that want to get whitelisted or wonder what's going on...

That's fair, but as I said, this and other projects are using undocumented and unofficial APIs, it might work for some people, it might not.

I don't think so as other websites that are also hosted on cloudflare work completely fine. But it's cloudflare so who knows how it works ¯\_(ツ)_/¯

It might be flagging the request to that specific URL.

Spontaneous idea as I also have some experience with cloudflare: You could maybe try adding a custom rule for the /assets/server/*.php path (looks like only this one is used for downloading) with a lower "security" level. If you use that functionallity in the first place...

I understand, but we're not wiling to decrease the security (and implement some other mechanism for protection) at this moment.

[itzg]

@andre-paulo98 I can locate when exactly this started failing, if that helps. It had been passing CI/CD tests for several years and then just suddenly started failing.

At the same time I certainly understand that being an unofficial API it's a best effort kind of solution. Maybe I'll add a note to the docs to set people's expectations.

[boldt]

I still have this error. I also tried saving the config as JSON and using VANILLATWEAKS_FILE instead of VANILLATWEAKS_SHARECODE. I get the same error message:

minecraft-1  | me.itzg.helpers.http.FailedRequestException: HTTP request of https://vanillatweaks.net/assets/server/zipdatapacks.php failed with 403 Forbidden: Fetching object content
minecraft-1  | 	at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)
minecraft-1  | 	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
minecraft-1  | Error has been observed at the following site(s):
minecraft-1  | 	*__checkpoint ⇢ Resolving download link
minecraft-1  | Original Stack Trace:
minecraft-1  | 		at me.itzg.helpers.http.FetchBuilderBase.lambda$failedRequestMono$2(FetchBuilderBase.java:220)