Physics system plans

[Drainful]

I see that the physics system is not super extensible at the moment. Leaning in to generic functions and CLOS could make it easier to deal with as long as it wouldn't be too slow, though that could be premature optimization talking.
I did some light refactoring purely for my own benefit here (not change reccomendations)
https://github.com/Drainful/sucle/blob/physics-refactor/src/sucle/physics%2B%2B.lisp

I'm curious about your plans/goals for the physics system, and I'm interested in helping out if you're open to it.

[gregcman]

As you can probably tell, I have not focused so much on the physics side, and don't really know how to change it. For the longest time I've wanted to have items and things running around, but I couldn't figure out how to structure it. Things like animals, buttons, monsters, rocks, in-game items, particles, etc...

Using CLOS wouldn't be a problem, there would probably only be a few hundred things anyway.

Looking at https://github.com/Drainful/sucle/blob/physics-refactor/src/sucle/physics%2B%2B.lisp and running it I can tell that its a lot more readable and it works [how did you refactor that?] but I'm not sure how the physics package system commented below would be extended to have other objects. Please, it would be great if you could tell me how to do this...

[Drainful]

I've tinkered with physics engines and CLOS enough to have a good idea of what to do I think. Here's my initial attempt. It's not functional yet, but it should give you the idea of my plan.
https://github.com/Drainful/sucle/blob/physics-clos/src/sucle/closphys.lisp
By splitting the physics function in to a set of generic functions and methods related by method combinations, any facet of the system could be extended.

[gregcman]

Thank you so much for providing insight into this! I never would have thought to use multiple inheritance with method combination this way, now the solution appears very intuitive and Common-Lispy.

How it seems like it would work:

1. In order to create a new type of game object, one would inherit from entity, living-entity, or has-position/physics/drag/mass/gravity
2. Then to customize rules for that object, one would use method specializers to override the defaults
3. Every physical object is wrapped in a before-physics and after-physics
4. Each game tick, step-physics is called on each of the game objects in the world

This seems like a great way to organize game objects, where there are tons of different types of objects that could exist, all having different properties and rules, so some would not have gravity, others would not have drag, etc... A long time ago I looked at the Minecraft source code in Java, but because Java does not support multiple inheritance, this strategy would not be available. Because CLOS is one of the most advanced object systems, perhaps this could be the first game system to effectively utilize CLOS, and do objects even better than the engines in other languages.

I can't wait to get around to implementing the new system! However I won't be able to work on it for a few days because of life.

Thanks for solving one of the longest-standing issues in sucle!!!!! If there's ever something you need help with, just ask!!

[Drainful]

I think this engine has a lot of potential; I should be thanking you. Would you be open to accepting a pull request for this when your time frees up? I’ve kinda gotten invested. I’d make sure to document well, and integrate my code cleanly with the rest of the project. It would just be a re-implementation of what currently exists, but after that it should be easy to implement, say, collision between entities.

[gregcman]

Sure a pull request that refactors the existing code and is well documented would be very welcome, but I think it would be better if we:

• Stepped through the decisions together
• Clarified what you are looking for out of this project
• Figure out what the project goals are/should be
• Clarify what the expectations are/should be
• etc...

The following are statements which are subject to change. I'm open if you want to take the project in a different direction, or have ideas of what it should/should not be. Or, if you are no longer interested, you do not have to stay, it is up to you.

TLDR: There are unique, never-before seen benefits to writing a voxel engine in Common Lisp, however this project does not have deep pockets or a viable business model. I'm doing it for fun.

What advantage would this have over other voxel simulations?

• Common Lisp
• CLOS, one of the best object systems, for entities and physics, possibly the first major implementation
• Simple infinite world full of lisp objects
• Programmatically create voxel worlds, with the live-coding ability of Common Lisp

What disadvantages does this have compared to other voxel games?

• In terms of fancy graphics, GUI, networking, animation, realism, there is a lot to be desired.
• It does not have the ecosystem of other languages, so implementation will be a challenge.
• I am not doing this for profit, and I don't have very much time or resources.
• It does not have a market monopoly or huge existing userbase or codebase.

Why I started and continued this project:
I started this project out of my own amusement, and at the moment it is mainly a voxel editor and walking simulator. I used it a lot, and created a bunch of maps that are more like abstract sculptures than anything else. I have yet to see voxel art or Minecraft maps that mimic the kinds of things you can create map-wise in this. With entities added, all sorts of worlds could be made that could not exist in other voxel games. One of the ideas is that while the majority of voxel editors, including minecraft, are geared towards being relatively static, this game would have procedural generation and interactivity all the way through.

[gregcman]

I'm telling you this so you can decide if you are still interested, so you know what you are getting into.

[Drainful]

First I want to thank you for writing that up; I am still interested, and I think we share a similar vision for a hackable live-codeable common lisp voxel engine. My goal for contributing to this project is to strengthen the core of the engine by creating well polished mechanics and systems with many extension points so that I can use Sucle as a basis for experiments in game design, and perhaps for a fully fledged game.
I will be approaching the project from the same hobbyist perspective, and I am familiar with the limitations brought on by using a relatively obscure language. I can't promise I won't lose interest in the future, but right now I'm excited to see what can be done. On that note, I wrote up this document explaining my choices with the physics system refactor. https://github.com/Drainful/sucle/blob/physics-clos/closphys.org

[gregcman]

Thanks for writing up the document explaining the physics system refactor. closphys.org and closphys.lisp have been move to the development branch. closphys.org has also been moved to doc/closphys.org. Now all that's left is to flesh it out a bit more and add more entities I presume. If there is any feature that you don't understand or would like to have explained, don't hesitate to reach out.

Thanks again!

[Drainful]

Sure. After the refactor is done I could make a branch for PR with the new physics.lisp all in 1 commit; might be cleaner. It might be ready tonight; I plan to work on it for a bit.

[Drainful]

I have to refactor the main loop a little bit in order to pass dt to the step-physics function; I'll set up a classic game loop with an adaptive time step. The per-frame funciton is a good candidate for a refactor after physics.
EDIT: Whoops I somehow missed the existing fps system. Nevermind.
EDIT2: Any suggestions on how I could give the time passed since last frame to run-physics-for-entity? The current framerate management system doesn't expose that information.

[gregcman]

Here are some new functions that expose the current physics-rate and dt: c140beb

(fps:dt) -> The target dt
(fps:ticks-per-second) -> The target physics rate

These functions don't expose the real amount of time passed, but what the expected dt should be for each physics tick. So (fps:dt) gives 0.016666 for 60 ticks per second, because at the moment the physics system tries to run at 60 ticks per second regardless of the graphical framerate, which is set by (fps:set-fps 60) at the beginning in the function sucle-app.

The call to (fps:tick ...) in sucle-per-frame may run the physics code within 0 to any number of times, up to a cap. The physics loop is an implementation of this: https://gafferongames.com/post/fix_your_timestep/ which tries to separate the rate of the physics from the rendering.

So I think it would ultimately be something like (step-physics *some-entity* (fps:dt))

[Drainful]

Got it. I have the new physics system working now; I just need to fix a bug with gravity and do a lot of code cleanup and I'm ready to submit a PR. I plan to move everything from closphysics.lisp to physics.lisp, replacing no-longer-needed code. What's your opinion on (clos)physics.lisp being in a separate package rather than #:sucle? It'll take a bit more work to sort out things like which utility functions should be in each one, but perhaps it's cleaner. I'd be happy either way.

[gregcman]

Thanks for refactoring the physics system, and breaking it into more manageable chunks. More packages are usually better, if you export symbols well.

However, with regards to the overall direction of the physics system, I have recently come across this thing called the 'Entity Component System' or ECS. Unreal, Unity, and Minecraft Bedrock edition use something like this. There's a simple library for a hybrid CLOS-ECS at https://sjl.bitbucket.io/beast/ which seems like it could map closely to closphys.lisp

So I would advise you to submit the pull request, it doesn't have to be complete. Then, I would port it to use https://sjl.bitbucket.io/beast/ which looks similar enough to what we're doing here. So you don't have to fully flesh out the pull request or tidy everything up, you can just PR what you already have and we'll work from there.

[Drainful]

I've used beast in the past! Yes I like the ECS style, and I've used an ECS written in rust (Specs) to do a physics system before. I will say that the traditional primary benefit of ECS is to follow data-oriented design (ECS is good for cache-coherency), while in common lisp we don't have the fine-grained control over memory necessary to fulfill that advantage.

Therefore I see little benefit to using Beast over the already quite flexible CLOS (over which it is a fairly thin abstraction layer), but I don't think its worse either, so if that's the way you want to go I'm on board. The advantage of Beast over CLOS is the ability to write "systems" which can iterate over every object which fulfills an aspect requirement, which can be done through CLOS with a bit of helper code.

[Drainful]

Unreal, Unity, and Minecraft Bedrock edition use ECS primarily for the performance benefit, using very specific contiguous memory layouts to maximize cache coherency. We could achieve the same effect by using an ECS written in a lower level language via ffi, though it's probably not worth the trouble.

[gregcman]

Buttons, particles, rocks, and items [and eventually animals and monsters] can finally exist in sucle! This is a major update and should be reflected in the [not-yet existing] versioning system.

On ECS:
You say that these big engines use ECS for the performance gains, and that using an ECS written in Common Lisp is nearly pointless, because it is not low-level enough. Yes, I may be cargo-culting somewhat. However, although Common Lisp may not be able to take advantage of some of the low-level strategies afforded by ECS, such as cache coherency, ECS has the ability to add/remove components of individual entities at runtime, although I think Beast doesn't have this capability. Maybe using Beast is not a well-thought out plan... Perhaps for now, the performance gains don't matter, and the big guns should not be pulled out yet, because there is not yet a bottleneck in sucle sufficient to warrant ECS.

[Drainful]

That would be my perspective. If it means anything, CLOS is 100% capable of adding and removing "Components" (superclasses/aspects) at runtime simply by redefining a class to have or not have the relevant superclasses. The limitation, if it could be called that, is that every instance of that class (and every instance of its subclasses) are altered at once. I believe it is possible to altar the class heiarchy of a single object at runtime as well. but I haven't looked in to it.

EDIT: here we go: change-class changes the class of an object. I'm sure we could make some macros to use this function to add or remove superclasses at runtime.

[Drainful]

But in reference to bottlenecks, the new phsyics system seems to have some intermittent lag. The slime profiler doesn't reveal much time spent in the new physics functions, but it does reveal that they do a bit of consing, so it might be garbage collection lag. The temp vectors should have mostly removed that problem though. More investigation is required.

Also, I don't really know what to do with the jumping mechanic. Currently if spacebar is pressed, every frame where (on-ground entity) is true entity gets some vertical boost to velocity. The issue is that it seems like on-ground is true for multiple frames in a row before a jump, making the actual velocity gained unpredictable.