kyle kukshtel
About  Projects  Blog  Inspiration  Archives  RSS  Search 

Bringing CastleDB to Unity

Bringing CastleDB to Unity

August 11, 2018

Tagged: Unity   Data   Tools  

About a month ago I got the idea to try and make Nicholas Canvases’ game database tool CastleDB meaningful integrate with Unity. If you just want to try out the plugin, see the repo here. If you want to know more, read on!

Introduction and Background

Separating game data from logic in Unity has never been something that’s been particularly easy. The introduction of ScriptableObjects was welcome addition that addressed some of this concern, but despite their utility they have certain limitations that still make them a non-ideal data container.

Namely:

None of the above make working with ScriptableObjects a “bad” experience. I think if you’re working on a small project they are totally serviceable, but as projects grow in scope ScriptableObjects can easily become unwieldy.

I was thinking about this recently and thought to myself that there had to be a better way. Something that really allowed you to separate data management almost entirely from Unity, but also provide a great interface to using it.

I think that a lot of people that get this itch settle on finding some way to manage external data through .csv or .xml files and then manage integrations with those data types. This satisfies the need to have your data be centralized, but even this felt unnecessarily bulky, as you still need to manually create objects and classes whose signatures somewhat match the data your loading. I instead wanted something that felt like it flowed, something that made data feel like an intrinsic part of my code, something that went beyond data loading and unloading.

CastleDB

It wasn’t until a few weeks after having this thought that I found this article on Dead Cells on Gamasutra and learned about a curious little tool called CastleDB, which made me think that something like what I wanted to do was possible.

CastleDB is, in short, a WYSIWYG JSON editor that makes working with JSON feel more like working with a traditional spreadsheet. It’s also made with the explicit purpose of being a “database editor for games”. It was created by Nicholas Canasse, a relatively prolific game developer who is probably better known as the creator of the Haxe programming language (which is also what CastleDB is written in). CastleDB represents his attempt to unify game data with game code, and looks to be successful at doing so. CastleDB has since been leveraged on the recently fully-released Dead Cells, and seems to be used behind the scenes for a fair number of other Haxe games.

CastleDB is also meant for Haxe, and as such uses a specific language integration that, once I saw it, looked exactly like what I was imagining when I was thinking about “data as code”. The code sample below is how you interface with CastleDB in Haxe:

package com.castle.demo;
import haxe.Resource;

import com.castle.demo.db.MyData;
class Main 
{
    static function main() 
    {
        
        #if js
        MyData.load(null);
        #else
        MyData.load(haxe.Resource.getString("test.cdb"));
        #end
        trace(MyData.myDB.get(Dragon).HitPoints);
    }
}

I won’t spend too much time talking about the appeal of Haxe for game development (there are already a few articles out there), nor is this article an attempt to convert you to Haxe (we’re talking about Unity!), but it is worth touching on what makes the Haxe/CastleDB interop special.

CastleDB leverages Haxe’s ability to use macros. A macro can mean a lot of things but in this case it allows you to do some really interesting things with edit-time code completion. In the video below you can see how, by creating and then importing a database object, you have direct access to the data at edit time and can, in a typesafe manner, access items from your database as soon as they are added.

Pretty nice right! Make a database, add some columns and lines, go back to your IDE, and they they are, nicely autocompleted. This was exactly what I was looking for! But given that I’m programming C# in Unity, how could I unlock something similar? CastleDB is targeted at Haxe users — was there a way to make it work with other frameworks or engines?

CastleDB + Unity

With this in mind I set out to figure out how I could go about making CastleDB work with Unity. If you just want to download the integration and don’t care about how I got it working, check out the project page here, otherwise read on!

Starting out, I needed to figure out what “working” with CastleDB would mean. Just being a JSON editor is interesting, but I wanted to unlock a similar flow to what I saw in Haxe. But how would you do this in C# without macros? I needed to figure out how to interpret the CastleDB file itself and figure out a way to make it “knowable” at edit time in Unity.

The first step seemed kind of obvious, make the CastleDB file readable in Unity and parse the JSON file.

What made the most sense to me was to think about the database as a collection of sheets, where every sheet defined a specific type of object (C# Object). Columns then would be the fields of that defined type. It then follows that if a sheet is an Object and the columns are fields, then every line in a given sheet is a specific possible instance of that Object.

This started to make sense but I also needed to figure out how to get the CastleDB file into Unity, as Unity doesn’t recognize the .cdb file natively. Luckily, Unity recently added an experimental ScriptedImporter API that would let me treat the file as an actual asset. Because it was just text, the actual “import” portion here was easy:

public override void OnImportAsset(AssetImportContext ctx)
{
    TextAsset castle = new TextAsset(File.ReadAllText(ctx.assetPath));

    ctx.AddObjectToAsset("main obj", castle);
    ctx.SetMainObject(castle);
}

The importer also allows us to run code any time that object is imported, which meant that, in the importing process, I could do something to make the CastleDB (CDB) file affect the overall state of the Unity Assembly. But what?

Stating plainly what I wanted to happen made what I needed to do clearer. At the time of import, I wanted what was defined in my database to be available to me in my editor. Saying that differently, I wanted CastleDB defined Types to be automatically generated when I imported my .cdb file into Unity. Ah, so I just wanted code generation!

Generating Types from a Database Object

Now, full disclosure, I had zero experience with C# code generation going into this issue, which only say because if this part of C# is weird or confusing to you, you can totally learn it! You can also skip the mistake I made of trying to learn C#’s AssemblyGenerator and instead skip to the part where I find out that Unity has it’s own in-built AssemblyGenerator class that is way easier to use than the native C# one. You can then also probably skip needing to use AssemblyGenerator at all and just write out some text to a file!

Instead of needing to emit C#’s intermediate language using obscure OpCode calls (which, even if you do right, isn’t guaranteed to work due to Unity’s managed version of C#), you just feed Unity a string of text that is your class, and then it writes that text to a file location, which, if it’s inside your Assets/ folder, gets automatically imported and built into your project’s solution!

In the AssemblyBuilder documentation, Unity has pretty good (and working) bit of example code that is copy-paste-able into your project for a simple version of how this works. Check it out here.

For this project, I basically used this example, but loped off the AssemblyGeneration portion, and wrote the generated code directly to a path in Assets. Here’s the whole class in the repo, but here’s the meat and potatoes of the file writing:

string fullClass = $"{fullClassText};
File.WriteAllText(cdbscriptPath, fullClass);
AssetDatabase.Refresh();

Easy right? This generation code is then called inside the imported script:

public override void OnImportAsset(AssetImportContext ctx)
{
    TextAsset castle = new TextAsset(File.ReadAllText(ctx.assetPath));

    ctx.AddObjectToAsset("main obj", castle);
    ctx.SetMainObject(castle);

    CastleDBParser newCastle = new CastleDBParser(castle);
    CastleDBGenerator.GenerateTypes(newCastle.Root);
}

So, parse the CastleDB file on import, generate the code that creates the type, and done right? Well yes and no. Unity is, I’m going to say, notoriously lacking when it comes to serialization/deserialization. Their native JSONUtility has righted some wrongs of the past, but it still suffers from other issues like the inability natively traverse a JSON tree deeper than one level. This isn’t a massive issue if you properly define what types you’re trying to serialize beforehand, but our issue is that, when deserialization happens, we don’t know what we’re deserializing. You can add a whole new sheet to the Database, which would mean a whole new type. Unless you change your deserialization code every time, there isn’t a way for Unity to natively know what JSON you’re going to pass it. I needed something that didn’t care, and could handle whatever I or any other user could throw at it.

Parsing Unknown JSON

Without beating around the bush, I quickly landed on SimpleJSON. It’s pretty small and has a nice API, so integrating it into the project was painless. Because it gives you the ability to index into a JSON file by a string value like node[“item”], I could essentially look at the cdb file’s field descriptions and then use what I found to the parse the whole tree. Hats off to Nicholas Canasse here as well — the format of the JSON file essentially contains field and sheet metadata, which proved invaluable when using this integration.

For every column in a CastleDB sheet, the .cdb file holds a “typeStr” value that is a… a string that maps to the type a column is supposed to be. This is a static map so I can just map column types to actual types in C# based on this value using a big switch statement. You can see how the native Haxe library handles this here.

SimpleJSON let me dig as deep in the file I wanted, and the CastleDB “typeStr” made the whole effort possible. Before I went on to writing the code that actually leveraged the parsing. I then wrote a small shim class that parses the .cdb file with SimpleJSON, and builds out a typesafe representation of the data that can be used to generate the actual dynamic parts of the database. So instead of needing to call something like RootNode[“columns”], I can, after parsing, just call RootNode.Columns and get the same return data back.

After all this is in place, you can generate a sheet’s class string by doing something like this:

foreach sheet in Root.Sheets
    // make a new type with sheet.Name
    string newClass = $“public class {sheet.Name} {...”
    ...
    string fieldText = “”;
    foreach column in sheet.Columns
        // add in type names
        fieldText += $"public {column.type} {column.Name};\n";

What’s nice about the code generation above is that you can just use a string as a type. That string “turns into” the type once the code is generated, which is part of the beauty of code generation.

You then put all of those disparate strings together in a beast that looks like this:

//use string interpolation and string literals to easily construct the class
//the component parts are constructed before this bit of code
string fullClassText = $@"
using UnityEngine;
using System;
using System.Collections.Generic;
using SimpleJSON;
using CastleDBImporter;
namespace {CastleDBParser.Config.GeneratedTypesNamespace}
  
        {getMethodText}
    }}
}}";

Spit that out into Assets, and now look what you can do!

CastleDB Parse Example

Leveraging Code Generation for some Nice Things

On top of being able to access a given row, as part of the generation I’m also making it so that when you get the row you’re also getting it initialized with all of its proper database values. I do this by dynamically creating an enum in the generated class that acts as a row lookup into the rows of the sheet the Type is built from. I then turn that enum into a JSON lookup (that is dynamically generated) that spits out code like this:

Creatures Get(CompiledTypes.Creatures.RowValues line) { return new Creatures(parsedDB.Root, line); }

And the object’s constructor looks like this (also dynamically generated):

public Creatures (CastleDBParser.RootNode root, RowValues line) 
{
    SimpleJSON.JSONNode node = root.GetSheetWithName("Creatures").Rows[(int)line];
        id = node["id"];
        Name = node["Name"];
        attacksPlayer = node["attacksPlayer"].AsBool;
        BaseDamage = node["BaseDamage"].AsInt;
        DamageModifier = node["DamageModifier"].AsFloat;
        foreach(var item in node["Drops"]) { DropsList.Add(new Drops(root, item));}
        DeathSound = (DeathSoundEnum)node["DeathSound"].AsInt;
        SpawnAreas = (SpawnAreasFlag)node["SpawnAreas"].AsInt;
}

This gives you a typesafe method to create objects with using your defined row values!

Conclusion

It’s not exactly macros, but it does feel like it has some of the same niceness of them. I think that over time your database scheme will likely (and rightfully) stabilize so the type completion becomes less important, but what will still be relevant is the ease of data editing/altering, with you being able to easily change references and such inside of the database instead of needing to use Unity.

Hopefully with some of this work I’m able to “unlock” our data from inside Unity! I also hope that with this plugin, it allows for non-coding collaborators on projects to feel more able to work with actual game data without the overhead of knowing how to code or knowing how to use Unity. Just open up CastleDB, change some values, and you’re done!

As I said above, if you’re interested in checking out this integration, I highly recommend you look at the project page and give it a download! I’d love to have some feedback on the project, and would love to know if people start using it in their games as well!

Final notes:

An interesting extension of some of this that I haven’t really pursued yet is making CastleDB natively use Unity’s native types like Vector3’s, Quaternions, Rects, etc. CastleDB supports custom types, so you could theoretically make a custom type that maps to a Unity type.

It’s also worth saying that a subset of what I’m doing here would be possible with XLS or CSV, but those formats aren’t necessarily human readable and don’t easily manage inner row referencing. A .csv file is also aggressively bad from a source control perspective. JSON is literally “object notation”, and as such is a better form for managing data objects. CastleDB also has some niceties in how it works, like auto sheet generating for nested types, that reduces overhead and gives you new types “for free”.

Published on August 11, 2018.

Tagged: Unity   Data   Tools  

subscribe to my newsletter