Proposal for new energy system (as a common API with other mods)

[gdelazzari]

I have created a separated repository for this. I'll probably start from scratch but first I want to have a specifications sheet that defines everything, from API interface with nodes to performance tips&tricks and, obviously, overall system architecture. Please contribute there with your ideas, suggestions and requirements. Just create issues or, even better, pull requests are greatly accepted to help me write the specifications document. If you plan on contributing constantly I can give you access to the repository.

https://github.com/gdelazzari/tech_api

Thank you!

---

Hi everyone, I've been experimenting and sketching out a new energy distribution API with the aim to create something that, in the future, may become a common API (i.e. a mod that other mods will depend on) that will make energy distribution and item & fluid management uniform and easier for anyone willing to write mods with such requirements. For the moment I'm focusing just on power. Also, I'm not using metadata to pass energy through networks so the Technic mod would benefit from this in terms of performance (given the current implementation). I already talked with nora on IRC #minetest-technic and he told me to write down here my progress and my current results so everyone can possibly contribute ideas or, maybe, help with the code.

This is what I have so far.
The system uses a table (under the API namespace) to store the energy networks. A network is a collection of devices connected by a transporter (i.e. a wire - a transporter is just a more general concept of something that can move energy around). A device can be a user, a provider or a storage. I think the names are self-explanatory, but if you read the comment at the beginning of the Pastebin below you'll get more information.
This means that every time a player places something that is power-capable, the table must be updated. This is not so easy to manage (especially when you need to merge two networks because a wire connected them together or, vice-versa, you need to split them etc...) but the advantage is that in one minetest globalstep handler we can manage the power distribution for all the networks in the world. Also, I'm using callbacks to pass energy in/out from devices, so no metadata is used in the process (except if the machines have an internal buffer and need to save the energy they contain, or in the case of storage devices like batteries. Maybe there's another way of doing that, but I'm not aware of any at the moment).

From a "mod-that-uses-the-API" perspective, I want to keep the usage really simple. This is what I have so far (please note that everything is experimental and thrown there, there are a few things that could be implemented differently so that it's simpler to use the API, but I need feedback regarding this).
This is a Pastebin with an example API usage and some comments to explain what's going on: https://pastebin.com/smDzZ6Nq

As you may notice there are two callbacks for each device every tick of the server, which is not the best thing in terms of CPU usage. However I plan to find a better solution. nora (I still don't know your username on GitHub, sorry :D) on #minetest-technic has made a proposal for this, maybe he can report it below so it's here for reference.

If you have spare time you can look at the current API implementation (which, by the way, is a mess). It is also incomplete (still no wire removal thus no network split) and has a ton of bugs, especially on the power distribution algorithm that was written in something like 15 minutes just to test out if the power was somehow getting from a point to the other (however I promise your PC won't explode). If you want to try it on a test world sure you can, just /give yourself some "unitech:user", "unitech:provider", "unitech:storage", "unitech:wire" and, if you want to get fancy, some "unitech:solar_panels" (featuring textures stolen from Technic - don't hate me). They're an example of a solar panel implementation that uses the API. Every freaking thing is "logged" on the chat, so you should be able to understand what's going on internally.

Just download the ZIP file attached and install the "mod".
unitech.zip

The nodes are simply demo-devices represented by colored cubes with some text on them, just to let you know what you're doing. Please note that the API, ideally, would not define any node: other mods will do that. The advantage is that we can have a mod just providing solar panels, a mod providing fancy wires, another one providing super-useful machines etc... all working together seamlessly. The nodes I have implemented are just to test out the system.
All the implementation is in the script unitech/power/power.lua

If you notice strange power distribution (e.g. storages not charging correctly, power losses etc..) don't worry, it's just because of the sketchy algorithm - which is not the priority at the moment. The priorities at the moment, IMHO, are the following:

• defining whether or not this will be useful if I develop it completely
• in the case of positive response to the point above, defining the API (I mean how nodes are going to interface with the power system in terms of functions/callbacks/properties/etc)
• given that I'm writing this from scratch, define all the best practices to make this blazing fast and not overload the network or the servers CPU in the case of multiplayer worlds

I hope this is somehow going to be appreciated, I'll be glad to have some feedback along with ideas. If anyone wants to help me write this, they will be totally welcome :)

Thank you for your attention!

[Ekdohibs]

So, let me explain what I had in mind for the API (btw, nore on IRC is me):

The first part is the design goals for the API we need. The main objective of that API would be to focus on something that can have enough performance, because we some servers are really suffering from the slowness of the current implementation of technic. However, networks need to function in an "all-or-nothing" fashion, even if only some parts of them are loaded.

In particular, that means the following:

• Running the users or producers should not need an ABM, as these don't work for unloaded nodes.
• There should be a way to avoid unneeded callbacks - the main use case being that while a machine is empty, it should not impact the network efficiency at all. Ideally, devices would be able to specify the time they would like to be supplied at the current power rate until a further callback to avoid more callbacks.

Now, for the implementation, keeping efficiency in mind:

• For now, we can just use a basic data structure to keep track of the different networks. If this proves to inefficient, I can rewrite it to use algorithms for dynamic connectivity.
• Likewise, I would start with a naïve algorithm for the callbacks -- I am not sure this one will be enough to start with, but it is simple enough to code and would be a starting point for more advanced algorithms using a heap.
• Last but not least, not running on every globalstep but having a configurable step time would be helpful to mitigate the load on the largest servers.

I feel we should however really have a polished API before implementing it, so that we know what we want. This rewrite could also be a good occasion to shift to a more proper energy system inside of the non-sensical "EUs" we currently use, which are both an unit of power and energy (!). I'd like some input from other technic developers or from players/server owners if they have a specific request.

Anyway, all in all, this is a rewrite I've longed to do, and an unified energy system is definitely something we need to have around, so thank you for wanting to help out!

[raymoo]

For the configurable step time thing, it might be a good idea to stagger network processing instead of doing it all at once. For example, assuming a Minetest step time of 0.05s, instead of ticking all the networks every second, you put the networks into 20 buckets and process one of the buckets every 0.05s step. This would avoid having lag spikes at the network update interval.

I haven't checked what the test implementation does, but since all connectivity info is in Lua-space, I feel like it might be cheap enough to just do a graph traversal on a network every time a node is removed from it (addition is easier to handle). Maybe that is what @Ekdohibs means by a basic data structure.

Also there might be crashes or unreported changes to the map that cause inconsistency between the table and the map. To make the system more robust, in case of unhandled deletions, it would be a good idea to pick a few network nodes every step and make sure they are consistent with the table data. This should be chosen to have good (map) cache performance. You could keep a mapping from mapblock -> set of network nodes to choose nodes that are in the same block. Unhandled additions could be mitigated by LBMs on the network nodes.

[gdelazzari]

First off thank you for the interest. Regarding the network structure handling/storing, I should say that I actually have little experience with graphs in computer science, so things like dynamic connectivity and heaps (which are different from the RAM heaps I know :D) are new to me. This is to say that I'm probably already using something similar to handle the networks (at least looking at the wiki suggests so). Anyway, @Ekdohibs may eventually help me out understand this if needed, I hope :D

Another quick thing before diving into the main one. You arised the "EUs" question. I was expecting someone to do that. I totally agree with you that they are non-sensical, like EU/RF/Tesla in Minecraft for instance (electronics guy here) but the thing is... I've been following a ton of youtubers playing tech modpacks in the past (I'm talking about Minecraft) and I really noticed that the majority of them (around 80%) didn't like approaches using something more realistic like Volts and Amps. EU/RF/etc is just simpler to understand and somewhat more straightforward to use, carry around and play with. Also, I think we can consider "energy/power units" somehow like Watts, with magical wires and machines that can transport them without worrying about Volts and Amps. Anyway I'm totally open regarding this, I think we should understand mainly what the players want (this is obviously a point of discussion/voting). In addition, units like that are probably easier to handle in code.

Now into the main point: the callbacks thing. I like the idea that @Ekdohibs proposed (using a callback when a machine needs of energy for ticks, etc...) but I was thinking... that will require more code on the distribution algorithm for sure (I don't know how many lines will need to be added) so I'm not sure how much that will improve performance. Remember that LuaJIT is actually quite fast, sometimes faster than Java. I think I should do some benchmarks (after you approved the two approaches under test), and let you know. However if the difference is minimal I feel like we should just go with the approach that leads to a simpler API usage for someone writing a mod using this. Regarding a couple of other things you said, I'm not using ABM at the moment, since everything is done in the globalstep handler which, eventually, calls-back functions of the devices.

For @raymoo: I like your idea for the network processing split. Also, yes I'm going graph traversals when nodes are removed and yes, doing some timed checks for consistency is mandatory. With the current implementation it is also quite easy to do that.

So, in conclusion, I think we should first off define what energy unit we want to implement and then, once we have that, talk about the interface that the API exposes. At this point, I would say that we could start from that i.e. we build up an example of how the API should be used (what methods to call and when etc...) and then we define and code the underlying system, collecting suggestions for performance.

[VanessaE]

One thing to consider about splitting up the workload: don't just go by network, be willing to split any given network into small steps also. I say that because my Dreambuilder survival server (daconcepts.com 30001) has a server-wide public HV electric grid.

It's currently one massive grid, kinda haphazardly-laid-out, but it's a single contiguous network with about 13 MEU of capacity (when all power sources are operating).

Adding a cable junction/spur to the HV grid, or in some cases adding a machine to it, triggers a "recalculation" of the grid that can run for around 30 seconds, during which time the server is basically unresponsive.

You'll need to take that sort of use-case into account.

[VanessaE]

As for what energy unit, I can't think of anything that's quantifiably better than an EU in a game setting, except perhaps megajoules or kilowatt-hours. The "voltage" can still be a unitless LV/MV/HV.

Of course that would surely prompt someone to want to measure continuous draw in kW for machines and generators and such, and use kW/h only for stored charge e.g. battery boxes.

(or whatever the equivalent measures are for joules)

[Desour]

a common API […] that will make energy distribution and item & fluid management uniform and easier for anyone willing to write mods with such requirements.

I wonder how you want to transport liquids like this since they don't move nearly instantly.

kW/h […] (or whatever the equivalent measures are for joules)

J=W*s

Btw, @HybridDog has already done something on a separated branch.

[gdelazzari]

@VanessaE with the current implementation, adding a device to a networks always takes the same time, which is equal to "instant". When you place a machine/generator/battery/something, it simply looks for an adjacent wire, looks at the wire metadata to figure out what network id it belongs to, and then the device gets registered to the network by adding it to the network table. So problems like the one you wrote wouldn't be there.

Also, I would like to report some limitations that the system I have implemented has (that I forgot to mention before):

• Having different types of wires (in term of max transport rate) in the same network is really really hard to handle with the approach I took, hence we should just say that we can't have "mixed" networks.
• No way to specify the sides of a node that can be connected to a wire, at the moment. In fact I totally forgot to think about that while I was designing and coding the thing, so it will take some efforts to snap this feature in
• Connecting a device to multiple networks is currently not feasible. It could be implemented, of course, but it wouldn't be fun. With storages and providers I could just make them conductive (which would be quite easy) so that they simply join together the two networks you place them in between but, with users, that may be a problem. Maybe you want a machine to take energy from two different networks and handling that is currently not so straightforward in terms of code

Replying to the other comments that appeared while I was writing this...

@VanessaE regarding the LV/MV/HV classification I don't know how you would implement that, probably in the mod space (i.e. not in the API) by making 3 different types of cables and programming them not to connect to each other, and then having converters once the second point above is implemented (which is, I think, how you are already doing this). But the API will know nothing about LV/MV/HV. When I started thinking and writing the system I had in mind something like RF in Minecraft, where if a machine is really big and needs a lot of power, it just asks for a lot (100000+ units/tick). The wire tiers would be the amount of energy they can carry. The progression in the game wouldn't be broken I think.

@DS-Minetest as far as liquid and items are concerned, as I said at the beginning, I am currently not interested in this. I'm focusing on the power delivery at the moment. What I meant is that, once we have a common API/system (if we'll ever have :D) throwing items and fluid management would be nice so that other mods would be able to connect to wires, pipes and tubes of any other mod for Minetest. My idea is that it would be nice to have something somehow similar to the Minecraft Technic experience, where all the mods interact with each other. If someone has already done some work regarding fluids it's nice because we can just integrate this and make this available to every mod through the common package :)

[gdelazzari]

@Ekdohibs I ran a quick benchmark by copy-pasting the distribution algorithm and the few lines strictly necessary (function to register devices etc..) on a separate file and assigned 1000 users, 1000 providers and 500 storages to the network. To run 1 million cycles, the program takes 78 seconds using LuaJIT, which translates to almost 13k cycles per second (AMD FX-8350@4GHz on Linux). Seems pretty good. Obviously there was no metadata access on the storage devices (I wasn't even running inside minetest) but even if there was, I don't think the performance would be lower than 2/3k per second. Considering we need 20 cycles per seconds.. I don't see any problem here. And this is using the callbacks-every-tick approach you see in the Pastebin. Now I'm going to try implementing something like what you have described and see how much the performance increases. If it's more than 25/50 times faster I might go with that :)

[numberZero]

Finally.

So, I have nothing against EUs (Energy Units) as long as they are accompanied by PUs (Power Units, EUs per second). Joules and watts are too realistic, you would have to define ingot mass in kilograms, for example. And on volts and amps... as long as you consider conductors and insulation ideal, these values don’t matter except of that voltage must match (also remember that IRL there are DC, 1-phase AC and 3-phase AC systems, and some others; also, at least 2 wires are needed virtually always).

What I am against of is the implementation. While the API, being suboptimal, may be polished, the implementation must be dropped. It will have catastrophic problems on real servers, for various reasons. I do understand it is here just as an example; I might do the same. Just don’t build real implementation on that.

Also, how would you join 2 networks temporarily (like in a switch)? That’s often handy with technic, but my be useful with any such mod.

P.S. @DS-Minetest J=W⋅s, not W⋅h

[gdelazzari]

While the API, being suboptimal, may be polished, the implementation must be dropped. It will have catastrophic problems on real servers, for various reasons.

Why are you saying that? I mean, what are the reasons? Help me understand :D if we can come up with something better, that's great. What I posted is just an experiment, I am ready to write everything from scratch :)

Also, how would you join 2 networks temporarily (like in a switch)?

I can come up with a couple of ways to accomplish this:

• If you want a switch that acts as a sort of "diode", passing EUs just from one side to another but not vice-versa, you could just make a node that is both a user and both a provider, but on 2 different networks. And in the node implementation, you just give the other network whatever you get in from the other one (only when the "switch" is active)
• If you want to really join the 2 networks, like having a wire connecting them, but just "on command", I can easily implement a "linked_networks" field for each network, so when you want to temporarily join two of them you just add the other network id to the "main" one (with the "main" network being one of the two, IDK what the criteria will be and even if it was random it wouldn't probably matter). When you need to disconnect them you just remove the id from the "linked_networks" table. The main loop, when going through a network that has linked networks, will treat everything as the same network, and skip the following loop cycles for the linked networks.

Regarding the energy unit, I think the simplest thing for the players that will make total sense to them is to define a unit of measurement (like EU, PU, RF, T, J, whatever) and to measure in/out/exchange/transfer rates (like the max rates of the cables, the power used by a machine or the power generated by something) with the same unit followed by "/t", which stands for "per tick". What this unit will be, I have no idea. EU is fine for me. I wouldn't bother with more realistic concepts for the sake of simplicity of the game.

[Ekdohibs]

About the units: EU for energy and EU/t for power are fine with me, as long as we do something consistent :).

Concerning joining two networks temporarily: this is the why of dynamic connectivity algorithms. But I expect that if we keep all the data needed for the networks in Lua, this will not be even needed as a graph traversal on ~1M nodes in Lua only with a correct implementation should run in <10ms. If this is still not enough, I will have to implement the dynamic connectivity.

Concerning access to metadata: it is slow. Like, really slow. The why is easy enough to understand: LuaJIT can't optimize the API calls, so each switching between Lua and C++ loses a lot of time. Besides, the cost of using strings everywhere needs to be taken into account when thinking about metadata, and it is higher that you might think. But if the code doesn't need to load metadata to give power information, it should save enough, at least for now.

Finally, about minimizing the number of callbacks: as I said, we can start a very naïve implementation that actually calls the callback on each step. Then, we could move to another a bit less naïve that just keeps the number of ticks before the callbacks needs to be called again, and decrease it each tick until we call that said callback. If that is still not enough, then we can just implement intelligent algorithms (I can take care of that if needed).

[raymoo]

Maybe devices should be able to have pure-Lua metadata like Minetest luaentities do so that callbacks have the option of doing useful work without needing to interface with C++ stuff. For example, counting down a work timer doesn't need to interact with the map or inventories.

[gdelazzari]

I have created a separated repository for this. I'll probably start from scratch but first I want to have a specifications sheet that defines everything, from API interface with nodes to performance tips&tricks and, obviously, overall system architecture. Please contribute there with your ideas, suggestions and requirements. Just create issues or, even better, pull requests are greatly accepted to help me write the specifications document. If you plan on contributing constantly I can give you access to the repository.

https://github.com/gdelazzari/tech_api

Thank you!

---

Will edit the opening post of the issue and put this also there