Committing C# Crimes to Provide Better ECS APIs

On figuring out how to bridge the GameObject/ECS divide by leverage attributes, partials, and source generators.

July 11, 2024

Tagged: Programming   C#   Gamedev  

Introduction

I’ve been unsatisfied with how entity creation works in Dinghy (now renamed “Zinc”) for a while. I said as much a few months ago on the forums and I’ve been mulling it over ever since. Specifically, the more you lean into using ECS, the more the requirements around interfacing with it tend to grow like a virus in your codebase. Especially for “real” ECSs where you shouldn’t ever really be passing around references to components, having to use things like ref a lot to just do something simple like update a component variable can feel aggressively onerus.

ECS truthers will say this is by design and how it should be. I say: sometimes I just need to set X/Y directly and I don’t want to write a query or reach for the ref syntax with a GetComponent<T> call just to set a value. No!

I felt this acutely when I was reading this article about “Leaving Rust game development”. I’m not so concerned with Rust here (though it seems to point to some of my own thinking about using Rust for games), but there are some really good opines on ECS and its cargo cult of nodevs:

I’m now very much on the hard disagree side in complete generality, unless maximum performance is of concern, and then I’d concede on this point only for those entities where this performance concern matters.

Super general systems don’t make good games, they make super general systems. The other oft-stated benefit of ECS around memory locality “may actually have some valid applications, but I’d say for the vast majority of indie games that get shipped, this is not necessary … it’s easy to build mindblowingly complex prototypes that will require this, but those will also be infinitely far from ever being”played” by other people, and thus aren’t of concern for this article.”

Relatedly:

I also don’t think one would find a person who thinks [Unity’s] DOTS is all there should be in the future, and that game objects should be erased from existence, and all of Unity should be moved to DOTS. Even ignoring maintenance and backwards compatibility, this would be remarkably stupid, as there are so many workflows that naturally fit into game objects.

And IMO you can swap ECS in for Rust here generally and it still be applicable:

Rust on the other hand often feels like when you talk to a teenager about their preference about anything. What comes out are often very strong opinions and not a lot of nuance. Programming is a very nuanced activity, where one has to often make suboptimal choices to arrive at a result in a timely manner. The prevalence of perfectionism and obsession with “the correct way” in the Rust ecosystem often makes me feel that the language attracts people who are newer to programming, and are easily impressionable. Again, I understand this doesn’t apply to everyone, but I think the overall obsession with ECS is in some sense a product of this.

I could copy and paste more of the article here (and there’s a lot!) but IMO just read it. If anything it feels like a cogent take on the state of affairs of modern well-to-do game developers.

How things are now

Something about this article set off a little fire in me, validating how I was feeling about my own use of ECS in Zinc and again touching on how I felt like the design goals of the framework (immediacy, coziness, magic) felt antithetical to working with ECS.

Armed with this I started just thinking through some things. What was the desired goal here? Well in some ways I already had it in Zinc itself. Instead of requiring engine users to declare all their own entities/components, I provided a few “defaults” that did lowest-common-denominator things. So something like a Shape allows you to draw a colored rect on screen. Shape the class extends from a base entity class I wrote that provides nice API functionality to users of that type of class.

When I say “nice” here, what I mean is that you can create a Shape, then set it’s X value like:

var shape = new Shape(){X = 2};

But I’m using an ECS here, and if X is coming from a Position component (that is presumably a struct and hence doesn’t pass by reference), how does this work?

Well, look at X on BaseEntity (what Shape extends from):

private float x = 0;
public float X
{
    get => x;
    set
    {
        ref var pos = ref ECSEntity.Get<Position>();
        pos.x = value;
        x = value;
    }
}

The property itself basically “hides” the ECS-y thing to allow the user to have a nicer interface.

And here we already get alarm bells going off for ECS truthers for a few reasons:

1. Having a managed class that acts as an interface to an ECS class is “bad”/non-performant/something else.
2. Having X baked like that in BaseEntity presupposes that the entity will always have a position component.

Point 2 first is a valid one, but over the course of working on my own games, my own engine, and my own games with my own engine, I’ve found that adding/removing components is a bit of a code smell. Indeterminate component permutations can really spiral code complexity and lead to a lot of casing where casing shouldn’t really be. From the Rust article again:

Having also tried the other approach of having “fat components”, both before and after the “separated” approach, I feel that the “fat components” approach much better fits into games that have lots of logic that is unique to what is happening.

Modelling Health as a general purpose mechanism might be useful in a simple simulation, but in every game I end up wanting very different logic for player health and enemy health. I also often end up wanting different logic for different types of non-player entities, e.g. wall health and mob health. If anything I’ve found that trying to generalize this as “one health” leads to unclear code full of if player { ... } else if wall { ... } inside my health system, instead of having those just be part of the big fat player or wall systems.

For Point 1, this has been the biggest sticking point for me when using ECS. I think in an ideal ECS in a frictionless vacuum world (“Pure ECS”), you basically don’t touch entities at all outside of systems. To me, this feels absolutely insane and is antithetical to trying to do stuff in a game to test out ideas. Having to “think” about (ECS) systems to spike out an idea feels like unnecessary friction that only serves the codebase, not the idea.

But again here we come to the head where working with ECSs to do that is hard — you have to remember the component you want, what the var name is, remember to use ref, etc.

The alternative that I landed on up to this point in Zinc was do the property getter/setters that wrapped the component member variables and “exposed” them at the top level of the entity, and for other notable but maybe not totally important variables I stuffed them into the entity constructor and then passed in those values to the underlying components:

    public Shape(Color color, int width = 32, int height = 32, Scene? scene = null, bool startEnabled = true, 
        Action<Entity,double> update = null,
        Action<EntityReference,EntityReference> collisionStart = null, 
        Action<EntityReference,EntityReference> collisionStop = null, 
        Action<EntityReference,EntityReference> collisionContinue = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMouseUp = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMousePressed = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMouseDown = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMouseEnter = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMouseLeave = null,
        Action<Arch.Core.Entity,List<Modifiers>> OnMouseOver = null,
        Action<Arch.Core.Entity,List<Modifiers>,float,float> OnMouseScroll = null
        
    ) : base(startEnabled,scene,update:update)
    {
        c = color;
        var rend = new ShapeRenderer(color, width, height);
        sceneRenderOrder = Scene.GetNextSceneRenderCounter();
        ECSEntity.Add(
            rend,
            new RenderItem(sceneRenderOrder),
            new Collider(0,0,width,height,false,collisionStart,collisionContinue,collisionStop,OnMouseUp, OnMousePressed, OnMouseDown, OnMouseScroll,OnMouseEnter,OnMouseLeave,OnMouseOver));
    }

If this seems weird, it’s because it is. The constructor stuffing here is a compromise/laziness on my part for not exposing those parameters instead on the entity itself. But here’s the thing — EVERY entity permutation (including user-created ones!) would need all this, every single time. Writing all that boilerplate every time absolutely sucks. And it doubly sucks because being able to do this does allow really truly great object instantiation code:

static_colliderA = new Shape(defaultColor)
{
    CollisionStart = (self, other) => {
       self.Color = collideColor;
    },
    CollisionStop = (self, other) => {
        self.Color = defaultColor
    })
    Name = "static_colliderA",
    X = Engine.Width/2f,
    Y = Engine.Height/2f
};

You can encapsulate the full function of an object in its initialization code. You may not always want to do this, but there are a lot of “parts” of objects in a game that need to just do one thing once that you don’t want to write a full abstraction for, and encapsulating the ability to do those things directly at the (only? maybe?) place where the object is created is really nice.

Sketching a solution

At this point though, I’ve got a bit of a problem and also something of a solution — component members variables can be on a class, providing a nice, more gameobject-y way to work with objects even though those objects themselves are effectively shims to an underlying ECS object. The downside of this method though is that it’s a LOT of boilerplate and requires knowledge about every component you want to write.

Ideally we can find some way to write all that boilerplate automatically…

My first thought was to do something around default interface methods. Entities could declare interface membership and then inherit the properties of those interfaces. However, the downside was that you would still need to cast the entities themselves to the component (interface) type in order to access the property. Interfaces also can’t store state, so the component references themselves that may be looked up in the interface was going to need to be overwritten. So that one was out.

Something new on the C# horizon that looks promising is the expanded use of extension methods. Specifically, the new concept of “extension types” and the ability to extend a type with more “functionality”. I could imagine some base Component type that has explicit extensions tacked onto it so that components could be unique from each other but still effectively “typed”. I’m not totally sure if this wouldn’t work, as the types can declare instance members, but the feature is also not yet fully released so I’ll have to wait a bit until it’s out.

At this point I had a sort of insane idea. The thing I was trying to do with both the interface methods and extension methods were to essentially “tag” a class in some way such that it could take on additional behavior that a user could directly interact with. What if I literally used a tag, or said differently, a C# attribute?

Could I use an attribute to “tag” an entity class with a component? Well yes, but attributes are just class metadata, they don’t actually affect the underlying class. Unless…. could I use an attribute to source generate the component data for a partial-declared class?

Attributes, Partials, and Generators (Oh my!)

Turns out… yes!

So now we’re getting somewhere!

What’s happening here is that I have a base attribute (BaseComponentAttribute) and a “component” extends that. I then use that defined attribute on a class that extends BaseEntity and now have magical access to the attribute properties as part of the entity creation (line 12)! But how?

Here’s how the source generator works:

1. First, it finds all classes that extend BaseComponentAttribute and then actually creates the real component in the background as a generated record struct (we do still want “real” components to iterate through).
2. For that created component, we populate its fields based on the public properties defined in the attribute.
3. We then also use those properties to backfill the other partial definition of TestEntity and auto-generate those “nice” properties that bind to the underlying actual components on the object.

It’s sort of surprising how good this feels and just works! I went to solicit some feedback in the C# Discord to see what other people thought:

Absolutely incredible feedback all around. People were enamored with the idea and can definitely see how it’s an improvement over traditional methods like needing to call .GetComponent<T>. The support I received when trying this out was unprecedented, so just wanted to say thanks to everyone for supporting me!

Armed with the blessing of the community, I realized that something felt slightly off. In the frictionless vacuum ECS world components are never supposed to hold state. But in the reality of game development, sometimes components do need a little bit of state [as a treat].

But in this current paradigm, because components were auto-generated, you didn’t really have the ability to do anything inside the component itself.

One option would be to generate the component itself as a partial class and then allow the user to then declare component state by redeclaring the partial component in their own code. Another option would be to wholesale copy the text from an attribute method onto the component as well, though this option specifically felt truly cursed.

What it came down to was ultimately “what I wanted the user to write”. Writing an attribute to back a component did feel strange — could I instead make it where the user wrote a component that an attribute could support?

Inverting the Solution

Yes!

Now this is feeling a lot better.

What I do here is, instead of hijacking the attribute definition to turn it into a component, I instead leverage a generic in an attribute to declare “what component we should copy the properties of” onto the partial entity class. The attribute constraint forces us to use things that extend IComponent.

This means the user has full control of the component (and as such can also change its declaration to class, struct, etc.) but with the still-added benefit of being able to grab the fields from the component at the level of the entity. Nice!

There’s still some additional work to do here, mainly around also generating nice APIs for the entity around component access as well as making sure to generate constructors that properly actually create the backing entity in the ECS system and actually add the components to it. That part is relatively easy though now that I’ve got the generator properly targeting everything!

Claude 3.5

Also as a small aside here, this project was something that was a great new testbed for Claude 3.5. I knew for the generator I wanted to use the new(ish) incremental source generator API, but didn’t want to spend a lot of time properly learning the API of those (which is kind of intense!). I wanted ot basically try this out in 2-3 hours. I haven’t really used an LLM for a ‘bigger’ project since doing mood.site, so this proved a great use case.

And I have to say, Claude 3.5 is great. The new side-view “artifacts” portion is exceptional and from a pure UX standpoint makes using ChatGPT-4o feel like a step backwards. I prompted it by pasting in the conversation I had on the C# Discord about the initial idea, added some of my own notes, and went from there. There were a few bugs here and there solved by either Claude of myself, but overall the structure worked and I didn’t really have to touch any of the load-bearing parts of the incremental generator API.

Conclusion

As I said above, there’s a few more things to do here around providing (generated) ways for objects to properly set themselves up with their tagged components, but other than that I’m feeling really good about the solution here. Being able to work with the ECS through something like a GameObject gives us something of the best of both worlds, and also helps to provide clarity on “what” things should be. Instead of needing to choose up front if something should be an object, and ECS object, or an object with ECS shims, and picking based on tradeoffs around API access and boilerplate requirements, everything instead can default to the the later, giving people the benefits of ECS-centric architecture, with the ability to lean into it on as much as they want to!

---

Published on July 11, 2024.

Tagged: Programming   C#   Gamedev