The future of kancolle-auto

[mrmin123]

I've been meaning to write this status update a few weeks back but work and life in general have been taking priority.

With kancolle-auto being close to (what I consider) feature-complete, I've been working on re-factoring the entire app so that it's faster, more stable, more flexible, and hopefully more extensible. Please do note that this is not a timeline or a roadmap, but a list of planned features, although I do hope to complete this in time for the next event (not a guarantee!). This also means that kancolle-auto as it is now will be more or less in a feature freeze.

Some of the planned features and changes:

• Hot-reloadable config - change config and have it apply on the next cycle, instead of restarting kancolle-auto, which will also allow things like pausing of the script
• Multithreaded image searches
  • Damage states
  • Morale
  • Combat module, in between nodes
  • Quests
  • Maybe submarine/shiplist
• Alternate sortie config mode - node location-based, instead of node count-based configuration
• LBAS morale checking - [Feature Request] LBAS Morale checking #373
• Revise all modules - key part of rework will involve even further separation of responsibilities and accessing information via referencing the relevant instances
  • Expedition module - revise so that hot-swapping expeditions are possible (taking out the last remnants of the original kancolle-auto's code)
  • Sortie module - refactor so that methods are less monolithic
  • PvP module - separate out from Combat
  • Repair module - separate out from Combat
  • Submarine Switch - separate out from Combat
  • Fleetcomp module
  • Quests - optimize quest checking and tracking
  • Navigation - separate out from Util
  • Util - more helper functions, harden existing functions
• Revised assets

What will not be changing:

• No UI - base kancolle-auto will remain command-line only
• No access to API - while API access would streamline many things compared to image matching, this is not currently within the scope of this project. kancolle-auto will never make API calls

Whether or not this refactor will continue to live as kancolle-auto, or if I'll give it a new home as kancolle-auto-kai or kancolle-auto-zwei or whatever is something I haven't decided yet. More on this in the future.

[waicool20]

I will also be giving Kaga a update if needed, if any of you guys have ideas for this feel free to post over here: waicool20/KAGA#34

Seeing there will be a massive change, a redesign may be due for Kaga too

[waicool20]

@mrmin123 btw since kancolle-auto is at EOL, you should create a final release. Been getting an influx of people who have been having problems with Kaga/kancolle-auto because they havn't updated the script to master.

[mrmin123]

@waicool20 I'll do that now. Forgot to do it after pushing the last set of updates.

Done: Release 10

[waicool20]

Thanks, it's nice that kancolle-auto ended on a round number :)

[stackhanovets]

@mrmin123,
Did you test a jdk memory leakages during multiprocessing?
Why not add autoscrap of lvl 1 'blacklisted' common drops / quest-dependent autocraft module? A bunch of Ryuuseis,+10 radars, 460 mm guns or Type 9 guns after month of daily autocraft is a good feature. Well, at most I do not talk about LSC...

[waicool20]

@stackhanovets the JVM has its own garbage collection, there should be no memory leakages by nature.

[stackhanovets]

@waicool20,
OK then. Here is the size of my special Mint VM after about a month of usage (relevant discussion), 96 GB:

_{^{Did you try to launch Eclipse with some heavy plugins enabled?}}

And I have been talking about a python strategy based on the multiprocessing.Pool() feature. A nice way to parallelize wrappers, but pretty consumptive at long-time processes. Did @mrmin123 mean it as the "multithreading"?

[waicool20]

@stackhanovets Don't know what that pic (and the eclipse text) has to do with memory leaks? And he will most likely use that to multithread as it's python based, I reckon he won't be using native JVM Thread objects.

[mrmin123]

By 'multithreading' I meant the threading library in python, not multiprocessing.

As for auto-development, auto-building, and auto-scrapping, I considered these a while back but it was kind of a pain to fully automate due to ship requirements, asset generation, and quest workflow. It's definitely a possibility, however, maybe as a one-time triggerable event, especially since hot-reloading configs are a thing now.

[stackhanovets]

@waicool20,
How could Chrome, libflashplayer, python, even apt eat 90 GB of disc space? I did not use this VM in any purpose except farming. Is there all right with Sikuli native code? Maybe JVM GC? There is only fact - the disc overuse. There are memory leaks. The old things are still happening in JVM, aren't they? Just like that:

http://lmgtfy.com/?q=java+memory+leaks

[waicool20]

@stackhanovets You are clearly doing something wrong if your install size reaches 90GB with only farming in mind. You can install all of those stuff and still have plenty of room to spare with only a 30GB disk. It could be you installed something that's generating logs to no end/chromium is caching to no end and not being cleared/something else you didn't realize. Use du and analyze your filetree and see what's consuming all that disk space, or just start a new VM with a max disk allocation of 30GB, I don't understand why you would give 90GB in the first place if you're just gonna use it for farming purposes.

2nd of all, memory leaks even if they do exists in the current iterations of kancolle-auto/kaga would not be the cause of your absurd disk usage. Memory leaks consume heap space (aka RAM) and not disk. The JVM by nature has no memory leaks, the only reason why it would consume lots of RAM is because of bad and dumb programming practices (eg. using global objects as temporary storage) IF you really are having memory leak problems....feel free to post a heap dump that shows excessive RAM usage but then again it probably will be related to Sikuli native code in which case post the report over there (though i think that's unlikely)

[mrmin123]

Unfortunately the new version of kancolle-auto won't be ready in time for this new event which is apparently tomorrow?? The new version is well under way but not ready for full automation yet.

The current version of kancolle-auto will be on life support to support this event as before.

[waicool20]

It might be a good thing that you haven't finished, it was mentioned in comtiq that some UI overhauls are coming. Apparently they are phasing out flash?

[mrmin123]

Interesting! That's good to know. Did they ever hint at a timeline?

[waicool20]

The said something about wanting to do the overhaul for their anniversary, nothing concrete though

[mrmin123]

Updates will be at mrmin123/kancolle-auto-kai#1.

[Kiras756]

btw, I resolved my old problem with Tesseract
#343 (comment)

[mrmin123]

Updates in #418.