Sectr Prototype Dev Log
Q1-Q2~ 2024 Progression Log
Preface
This is an article dev logging my experience attempting another go at a projectional editor I’ve desired for many years. It records my progress from end of January till June. There will be some writing strung about and plenty of documented screencaps, vods. It will feel discontiguous; think of this as a rough public dev log.
Intro
Previously I posted a gallery article for the original project I made for a non-text based editor.
Near the end of February 2024 after Graven was released. I began to take a break from Unreal and had the urge to for another go at making a prototype using a codebase following a more “handmade” approach. Making a codebase from scratch from the systems level for the purposes of having tailored tooling for the job. I was also looking to try out an alternative systems programming language to C/C++. Of them all, Jai looked the most appealing, but I didn’t have access. So I went ahead and decided to give it a shot with Odin.
I started by reproducing portions of the codebase following constructs and definitions Casey introduced or emphasized in the early videos of Handmade Hero. I had previously followed along the series a year prior; getting to around video 50..
This involved setting up a host application to load a client dynamic module for hot-reloading, setting up virtual memory arenas, basic input polling, and windowing. Early on the input, 2d graphics, and windowing were handled with Raylib instead of directly dealing with the Win32 API. It was a framework available from the vendor collection that Odin comes bundled with, and I was aware of prior from the handmade & gamedev community promoting it.
So that’s how it began, using vscode as my editor with powershell to script usage of the Odin toolchain, Odin’s Core library collection and Raylib as an extra simplification layer for native app development. For the debugger however, I did not go with LLDB via vscode or RemedyBG. A new graphical debugger had recently been silently released called the RAD Debugger. The latest author for RAD’s project was Ryan Fleury who had posted some WIP shots on his discord. Since Odin advertised itself as supporting debug symbols and knowing it had a much simpler symbol resolution to C++, I thought it would be a good time to test drive it.
The result was a wonderful initially. It is quite possibly the most responsive non-CLI based tool I have used in programming other than the 10XEditor, or Sublime.
Using the Odin core library was also a great experience. It was quite easy to sift through, identify anything I had a discrepancy with, and then modify it with my own solution. Such as with the virtual memory interface:
The original did not expose the base address argument that all major APIs (POSIX’s mmap, and Win32’s VirtualAlloc) have for debug purposes that Casey demonstrated use for.
Making my own was trivial. Odin’s provided logger API was also a nice surpise.
It was easy to make modifications to the source for my own formatting. Within a couple of days, I had the basics all working. In the following example here is the same replay feature Casey demonstrated in Handmade Hero:
UI : First steps
A few more days later I had the initial code for the UI based off of studying raddbg, VUI, and Nuklear’s imguis. Most of those notes I’ve gathered in a repo on github.
(This project helped iron out some of my deficiencies with 2D math….)
Thanks to Raylib, there was an initial setup of text with zoom on the preliminary canvas workspace, however, this would lead to a can of worms down the line when it came to performance targets…
Learned quite a bit from this project on the unique difficulties text has to render properly with linear zoom.
Quite surprising how little code was required for such a simple easing camera zoom..
Some early diagramming to figure out the basics for a hiearchal “AST” for an IMGUI box UI graph:
Unfortunately learning about how to make a robust imgui for the project’s needs was the biggest hurdle that I never fully resolved within the current development period. There is quite a bit of content but there is an underlying assumption from most content. It oversimplifies the level of complexity and pipelining required to maintain the staging of an immutable “immediate-mode” interface for the UI’s frontend vs the backend parser and engine of the resulting UI graph’s data.
(Someone should write a good book on this…)
The most important takeaway is that any procedural UI with a good amount of degrees of freedom and other visual prowess requires a “PIMGUI” (Persistent IMGUI) flavored architecture. To get things working “from-the first frame stable” is apparently possible, but for a novice like myself not something easily obtainable..
In the middle of March, I began to profile the project using spall. It had direct support in the core library:
There was some bugs with the auto-layout..
Sidetrack 1 : Memory bugs
While bootstrapping the codebase, I decided to make my own allocators. These were:
Virtual arena managing paged
memory blocks.
Fixed sized block pool allocators (to be used directly by a container or by another allocator)
Slab Allocator: Collection of managed pool allocators specified by a size class policy struct. Acted as a simple solution for “soft-unbounded” sized containers such as dynamic arrays, strings, and hash tables.
They naturally had a few hard to resolve bugs, but the logger and debugger helped resolve all of them.
Continuation in May
A break was taken in April to look into learning Unreal’s Gameplay Ability system from Druid Mechanic’s course on udemy. I didn’t finish, but it was a good introduction to that…
From where it was last left off, there was a clear bottleneck with text rendering as it would drastically increase frametime based on how many boxes were rendered to the screen. Addressing this problem would require dropping Raylib eventually.
For now more progress was may on the auto-layout for the UI.
I eventually had to deal with overlapping boxes and their order being dependent on user interaction.
Setting up the vertical box introduced me to some significant auto-layout frame-delay lag (not the easiest thing to fix..).
I still don’t know the best general solution for the current implementation.. But I have a general heuristic to follow to resolve the issue in the future.
It caused some cool effects when used intentionally:
Getting basic color themes working:
Along with a rudimentary input box.
Dropping Raylib for the Sokol Libraries.
During my testing to see if I could render a large amount of text on the screen for visualizing ASTs, I noticed Raylib seemed to heavy performance issues with text.
Its DrawTextEx and DrawCodepoints were just not suited for a heavy amount of individual calls.
So I began, at the end of May, to replace Raylib initially with what Sokol offered: sokol_app.h and sokol_gfx.h. Sokol app provides basic OS abstraction for events and windowing. While sokol gfx was a graphics api wrapper primarily focused on web apis however I used it to handle Opengl and DirectX 11 using a more modern upfront pipeline interface.
This was the first time I started to use pureref not just for debugging but also to outline entire single-header libraries. I’ve always had a frustration (one of the reasons I’m doing this project in the first place) with navigating large files. Composing view compositions of code definitions separated across a codebase is painful by just tiling different files or different scroll positions of a file. Especially if I haven’t been initiated in a codebase for months to years. Having the code in a canvas really felt a breath of fresh air.
Next was to figure out text… As sokol doesn’t provide a fully featured text renderer out-of-the-box..
Text Rendering R&D…
I started with where I’m assuming many non-text or rendering wizards go to learn the basis for text rendering.. learnopengl.com’s text rendering post. It did the job for grasping the most barebones way todo non-bitmapped trutype text rendering using freetype.
There was definitely an initial struggle as learning sokol_gfx at the same time was most likely not the wisest decision…
Raddbg was a great help here with its pre-defined bitmap view-rule:
This worked, but naturally worse performing than Raylib..
Porting VEFontCache
Like with learning IMGUIs, I did a deep dive scouring through all readily available resources for text rendering and compiled what I thought was useful into a repo.
I started first with fontstash as that was the recommended library by the Odin community. I soon found out however that it didn’t directly support rendering with Sokol gfx (it used another vendor library called nanovg, and had some other quirks specific to Odin’s port). This all turned me off…
I instead eventually settled with a C++ library called VEFontCache.
The implementation was all within a small single-header (< 2k lines) and the C++ was quite C-like. But… it was still C++. When it comes to these libraries, when they’re this small.. It’s better to have them in the native language used for a project. So I decided to port it over to Odin.
According to the git commit logs… The main process transpired between June 2nd, and June 17.. so 15 days. The initial draft pass for porting took around 4 days. The rest was general bug fixing from typos, and wrangling my limited knowledge of sokol gfx. VEFontCache needs the user to setup rendering passes that its draw list expects. Renderdoc came in really handy…
It took me a while to figure out why the letters were not showing up. It was related to a UV convention issue for how the textcoords are interpreted for the fragment shader. (not sure if its a sokol-gfx thing..)
This ended up being the most satisfying part of the project to get working.
Recovering back to feature parity
I took a break todo some input event setup for sokol app, along with other graphical features to get back to parity with the UI. I used sokol_gp.h as the 2D rendering library base.
Naturally new rendering bugs appeared…
Eventually things started to behave, and it looked magical to see the text rendering passes in renderdoc.
Looked at comparing the current state of rendering with what pureref had:
There was some artifacting while zooming, nothing a little padding and other fine-tuning couldn’t fix with the texture atlas.
Optimizing VEFontCache & general bug fixing
Now that I had my own library for text rendering, next was to get it hitting the performance I desired. Microseconds or less per call to draw_text.
Naturally found more bugs related to memory corruption..
And to canvas coordinate math…
Performance got to the point where I could bring out more serious tests.
Then finally got the performance targets I could be satisfied with for future prototyping:
Feature parity with great performance at last
So at the end of June, I got back to where I was in end of May.
This had be the end for now. Coming into the month of July, none of the promising potential contracts for the year had resolved into deals. So, after 5 months of recreational coding it was time for me to get back into the usual business, and begin the process of networking for new opportunities.
Due to how successful the VEFontCache port was, before entirely stopping, I took some time to properly package the library into its own repo. With a fully working example of its original demo as an example project, and a sokol rendering backend sample provided. I hope it will be of some utility to others.
Well that’s the dev log. If you enjoyed skimming through this give it a like. It’ll get me to keep this up beyond being part of my portfolio.