Ash Framework is ~4 years old, and we’ve only just now introduced the ability to generate resources, or new Elixir projects with Ash installed using a CLI. We’ve added these thanks to our latest project Igniter! Before I talk about our new generators, however, I’d like to talk about why I avoided them for so long.
Problems with Generators
Generators are a slippery slope
When a library begins providing code generation, a common design problem is leaning too heavily on these generators. Often times, instead of changing how the underlying tool works, the generators will be modified and you end up with a sort of “best-practices-drift”. The opinions of the maintainers are partially embedded into the bits that you generate, and partially embedded into the underlying library code.
To have a successful relationship with code generators and your library , you must ensure that generators do not introduce new or different opinions. If there is something that a framework/library author thinks that users should be doing by default, that opinion must be embedded into the library directly. For example, lets say that you’ve got a web framework with a toggle that allows SSL to be turned on or off. Instead of updating your generator/installer to set that option to true by default because you think that is the right default behavior, make the option default to true if its not specified.
Another factor here is that we want our framework code to be as concise as possible. In Ash, defining a resource with hand-written code and generating it with our new mix ash.gen.resource command take almost the same number of keystrokes. When you have generators, the incentives go the other way. It becomes much easier to add something to generated code than it is to remove a piece of required boilerplate.
Configuration files become stale
When working users initially set up credo , they will typically run mix credo.gen.config to generate a configuration file. This configuration file contains all of the checks that credo knows about when the config was generated. What this means is that when new Credo checks are created added by its maintainers, there is no reasonable way to automatically include them in your config file without monitoring how the project is changed, what the new modules are, etc.
A critical observation here is that there is no elegant way to do this without the introduction of a new set of tools. This isn’t a shot at Credo, or any other library/framework that does code generation this way. This is an example of something that requires a tool like Igniter.
Security updates & bug fixes require manual resolution
The de facto authentication solution for Phoenix is mix phx.gen.auth. The way this works is that it generates an entire authentication system for you, directly into your Phoenix application. If a meaningful change, how would people find out about what changed and why? If someone were to discover a security vulnerability in that generated code, what would we, as a community, do?
We can’t say “go update to the latest version of Phoenix” and you’re good. It will be up to you to figure out how to mitigate the security issue in your now-customized authentication code. This is an example of something that cannot be solved by Igniter.
We solve this in AshAuthentication by providing code that we manage on your behalf. There are trade-offs here of course, but this strategy means that if some security vulnerability is discovered in AshAuthentication, getting a fix is as simple as mix deps.update ash_authentication.
Installers are one way roads
Many developers, even experienced developers, have trouble with the decision point at the outset of a project. Do I use mix phx.new? Do I use mix new and add Phoenix? What options do I pass to phx.new?
Here are some examples of options you might want to add when creating a new phoenix project with mix phx.new. --no-assets, --no-dashboard, --no-ecto, --no-esbuild , --no-gettext, --no-html, --no-live, --no-mailer, --no-tailwind . For experienced Phoenix developers, maybe you know which of these you want and when. For beginners, they have two choices:
- Learn everything you need to know to know about how Phoenix works to understand which parts you don’t need.
- Opt in to everything, regardless of whether or not they need it, dealing with a potentially more complex piece of machinery than they actually needed at the outset.
In my opinion, the journey should go the exact other way around. mix phx.new should get you the most basic version of a Phoenix project. Then when you want Ecto, or Tailwind, or ESbuild, gettext, you run a command to add that to your application.
Rethinking code generation with project patching
There is a better way to approach this entire problem. It requires a new kind of tool and a new way of thinking. 🏗️ Igniter is the tool, and it operates under this new way of thinking. What are the new rules?
- We think not in terms of “generating files”, but in terms of “patching a project”. This might mean “creating or modifying modules”, or “adding configuration”, or “adding dependencies”.
- We don’t write things that modify text, we write things that modify ASTs (Abstract Syntax Trees). We modify code, not files. No regular expressions or string operations. Instead, we use Sourceror to modify AST.
- The source code is the source of truth, not the compiled code. For example, if we want to know if a module exists, we don’t check with
Code.ensure_compiled?/1, we search the source code for that module withIgniter.Code.Module.find_module/2. - We acknowledge that project patching is a best effort transformation, not a perfect science. We must present information to the user explaining any changes that couldn’t be made, allowing them to manually intervene to make those changes.
What does it look like for library authors?
We write a type of composable mix task called an Igniter.Mix.Task. Here is a very simple one, taken from the spark library:
defmodule Mix.Tasks.Spark.Install do
@moduledoc "Installs spark by adding the `Spark.Formatter` plugin, and providing a basic configuration for it in `config.exs`."
@shortdoc @moduledoc
use Igniter.Mix.Task
def igniter(igniter, _argv) do
igniter
|> Igniter.Project.Formatter.add_formatter_plugin(Spark.Formatter)
|> Igniter.Project.Config.configure(
"config.exs",
:spark,
[:formatter, :remove_parens?],
true,
updater: &{:ok, &1}
)
end
end
Builder Pattern
All of the work happens in igniter/2. Notice that the first argument is an igniter, which we modify and return. This is called the builder-pattern. If you’ve used Plug, or Ash.Changeset or Ecto.Changeset you will be familiar with the concept.
High level project API
Another thing to note, is that we provide pre-built useful project patchers out of the box in igniter. For example, Igniter.Project.Formatter.dd_formatter_plugin/2, which adds a plugin to the .formatter.exs. It is Igniter's job to deal with any complexities of how the user may have written a .formatter.exs file, and to show them a message if we can’t apply the patch.
This same thing is true of Igniter.Project.Config.configure/6. You tell Igniter, “this specific config should have this value”, and it handles the rest!
Low level code API
Not pictured in that task, is that Igniter also has a low level API for modifying source code. This is critical when a project wants to do some kind of patching unique to that library. We encourage libraries to implement custom high level APIs that use these low level APIs. For example, when using mix ash.extend Resource ExtensionName , we call a function provided by spark :
Spark.Igniter.add_extension(igniter, Resource, Ash.Resource, :extensions, ExtensionName)
I will show an abridged version of Spark.Igniter.add_extension here, so you can get an idea of it.
def add_extension(igniter, module, type, key, extension) do
Igniter.Code.Module.find_and_update_module!(igniter, module, fn zipper ->
case Igniter.Code.Module.move_to_use(zipper, type) do
{:ok, zipper} ->
Igniter.Code.Function.update_nth_argument(zipper, 1, fn zipper ->
Igniter.Code.Keyword.put_in_keyword(zipper, [key], [extension], fn zipper ->
Igniter.Code.List.prepend_new_to_list(
zipper,
extension
)
end)
end)
_ ->
{:ok, zipper}
end
end)
end
Spark.Igniter.addextension/5 (abridged)
The workflow looks something like this:
- move to the code
use Ash.Resource - move to the argument at index one, i.e
use Ash.Resource, opts← here - set the
:extensionskey to[extension], and if its already present, set it to[extension | other_extensions]
Notice how we use tools to traverse source code. We are using a data structure called a Zipper. Notice how even in this case, we aren’t doing any “parsing” or string operations. We’re moving around through code. This is an example of applying rule #2. By operating on the source code and not the text, we can make semantic changes. This results in a better user experience, and in the tooling being able to help you in surprisingly delightful ways.
For example, in step 3 above, if the extension we’re adding has been aliased above, Igniter can still determine equality in prepend_new_to_list. It will also use existing aliases. For example, lets say you had a patcher for updating the list found in @list below.
defmodule Foo do
alias Foo.Bar
@list [Bar]
end
If you did Igniter.prepend_new_to_list(zipper, Foo.Bar), the resulting list would be untouched, because we know that Bar is actually Foo.Bar with an alias. Try writing that regex and see how you do. 😆
Additionally, if you did Igniter.prepend_new_to_list(zipper, Foo.Bar.Baz), the resulting list would be @list [Bar, Bar.Baz], because we determined that Foo.Bar was already aliased.
There is one last bit of glue that really enhances the user experience:
mix igniter.install
mix igniter.install package_mame will actually check for an appropriate version dependency from hex, and then will add it to your mix.exs file automatically.
Then, we look for an Igniter.Mix.Task defined in that package called mix package_name.install. If it exists, we run it! It is a very simple concept, but it makes a huge impact on the developer experience. Lets take a look at what the new developer experience actually looks like.
Developer experience with Igniter
Lets say I’m starting a new project. I know that part of my project will include a DSL for defining workflows. I’ve got nothing so all I know is that I need a new Elixir project and that spark needs to be present. Lets make a new project:
mix igniter.new my_app && cd my_app
This is a thin wrapper around mix new that installs Igniter immediately. You can also install packages while creating a new project, or tell it to use mix phx.new (or whatever other installer you want), which is great for giving users a single command that starts a new project with your tool set up. For now, though, we’ll just use mix igniter.new
Now lets install Spark
mix igniter.install spark
First, it tells me that there are some dependencies I need to fetch:
I accept, and I’m presented with an explanation of the changes it would like to make to install spark.
Next I realize that I want an API to call the workflows I’ll be defining. As seen above, mix igniter.install ash will take me through the whole process.
Oh, you know what? I need to persist the results of these workflows in a database! mix igniter.install ash_postgres. Its just that simple!
Oh, now I realize I want to build an admin dashboard with Phoenix LiveView. What do I do? Thats right, just mix igniter.install phoenix_live_view...uh oh. All that does is add the dependency to my mix.exs file. Where is my fancy installation process?? There is no Igniter.Mix.Task called phoenix_live_view.install 😢
And this is the catch with generators that aren’t written as project patches. All of the initial structure that you get from mix phx.new can only be gotten with mix phx.new. There is no process to add Phoenix to an existing application (that I know of). This is where it comes down to the community.
Where do we go from here?
The more our projects are written to be additive to a project even if all that we do when we see a conflict between a file we wanted to create and one that is already there is say “hey I wanted to create this file, but you already did, so I couldn’t make that change”, the better. This means that project patching doesn’t have to be a mega fancy process, it just has to be written in such a way that it expects that the project already exists. Thats where the “best effort” rule comes into play.
As for Ash Framework, all of our projects will soon have an igniter installation setup, and any existing generators will be migrated to use igniter as well. Igniter will be the basis for any code generation, installers, upgrade scripts, etc.
I know that others are very interested in this tooling, and I hope that we can extend this DX to as many packages in the Elixir ecosystem (that need it) as possible. I also welcome others to help improve and expand Igniter over time! It takes a village, and if I get my way, Igniter will soon be a ubiquitous tool across the Elixir ecosystem.
I hope you are all as excited about Igniter as I am!
Happy Hacking 😎
