Web-Projects/blog-static
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Add a draft article about X Macros

Browse Source
This commit is contained in:
Danila Fedorin 2023-10-09 20:23:57 -07:00
parent 12aca7ca58
commit 4f281ef108
1 changed files with 699 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

699
content/blog/chapel_x_macros.md Normal file
View File

@ -0,0 +1,699 @@
---
title: "My Favorite C++ Pattern: X Macros"
date: 2023-10-09T15:06:11-07:00
draft: true
tags: ["C++", "Chapel", "Compilers"]
---
When I first joined the [Chapel](https://github.com/chapel-lang/chapel/) team,
one pattern used in its C++-based compiler made a strong impression on me. Since
then, I've used the pattern many more times, and have been very satisfied with
how it turned out. However, it feels like the pattern is relatively unknown, so
I thought I'd show it off, and some of its applications in the
[Chapel compiler](https://github.com/chapel-lang/chapel/). I've slightly tweaked
a lot of the snippets I directly present in this article for the sake of simpler
presentation; I've included links to the original code (available on GitHub)
if you want to see the unabridged version.
Broadly speaking, the "X Macros" pattern is about generating code. If you have a _lot_
of repetitive code to write (declaring many variables or classes, performing
many very similar actions, etc.), this pattern can save a lot of time, lead
to much more maintainable code, and reduce the effort required to add _more_
code.
I will introduce the pattern in its simplest form with my first example:
[interning strings](https://en.wikipedia.org/wiki/String_interning).
### Application 1: String Interning
The Chapel compiler interns a lot of its strings. This way, it can reduce the
memory footprint of keeping identifiers in memory (every string `"x"` is
actually the _same_ string) and make for much faster equality comparisons
(you can just perform a pointer comparison!). Generally, a `Context` class
is used to manage interning state. A new interned string can be constructed
using the context object in the following manner:
```C++
UniqueString::get(ctxPtr, "the string");
```
Effectively, this performs a search of the currently existing unique strings.
If one with the content (`"the string"` in this case) doesn't exist, it's
created and registered with the `Context`. Otherwise, the existing string is
returned. Some strings, however, occur a lot in the compiler, to the point that
it would be inefficient to perform the whole "find-or-create" operation every
time. One example is the `"this"` string, which is an identifier with a lot of
special behavior in the language (much like `this` in languages such as Java).
To support these frequent flier strings, the compiler initializes them once,
and creates a variable per-string that can be accessed to get that string's value.
There's that repetitive code. Defining a brand new variable for each string,
of which there are around 100 at the time of writing, is a lot of boilerplate.
There are also at least two places where code needs to be added:
{{< sidenote "right" "template-note" "once in the declaration of the variables, once in the code that initializes them." >}}
A third use in the compiler is actually a variadic template defined over
character arrays. The template is defined and specialized in such a way that
you can refer to a variable by its string contents (i.e., you can write
<code>USTR("the string")</code> instead of
<code>theStringVariable</code>).
{{< /sidenote >}}
It would be very easy to accidentally modify the former but not the latter,
especially for developers not familiar with how these "common strings" are
implemented.
This is where the X Macros come in. If you look around the compiler source code,
there's a header file that looks something like the following:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/framework/all-global-strings.h" "C++" 31 >}}
X(align , "align")
X(atomic , "atomic")
X(bool_ , "bool")
X(borrow , "borrow")
X(borrowed , "borrowed")
X(by , "by")
X(bytes , "bytes")
// A lot more of these...
{{< /githubsnippet >}}
What's This `X` thing? That right there is the essence of the pattern: the macro
`X` _isn't defined in the header!_. Effectively, `all-global-strings.h` is just
a list, and we can "iterate" over this list to generate some code for each
one of its elements, in as many places as we want. What I mean by this is
that we can then write code like this:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/framework/global-strings.h" "C++" 76 >}}
struct GlobalStrings {
#define X(field, str) UniqueString field;
#include "all-global-strings.h"
#undef X
};
{{< /githubsnippet >}}
In this case, we define the macro `X` to ignore the value of the string (we're
just declaring it here), and create a new `UniqueString` variable declaration.
Since the declaration is inside the `GlobalStrings` struct, this ends up
creating a field. Just like that, we've declared a class with over 100
fields. Initialization is equally simple:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/lib/framework/Context.cpp" "C++" 49 >}}
GlobalStrings globalStrings;
Context rootContext;
static void initGlobalStrings() {
#define X(field, str) globalStrings.field = UniqueString::get(&rootContext, str);
#include "chpl/framework/all-global-strings.h"
#undef X
}
{{< /githubsnippet >}}
With this, we've completely automated the code for for both declaring and
initializing all 100 of our unique strings. Adding a new string doesn't require
a developer to know all of the places where this is implemented: just by
modifying the `all-global-strings.h` header with a new call to `X`, they can
add both a new variable and code to initialize it. Pretty robust!
### Application 2: AST Class Hierarchy
Altough the interned strings are an excellent first example, it wasn't the
first usage of X Macros that I encountered in the Chapel compiler. Beyond
strings, the compiler uses X Macros to represent the whole class hierarchy
of [abstract syntax tree (AST)](https://en.wikipedia.org/wiki/Abstract_syntax_tree)
nodes that it uses. Here, the code is actually a bit more complicated; the
class hierarchy isn't a _list_ like the strings were; it is itself a tree.
To represent such a structure, we need more than a single `X` macro; the
compiler went with `AST_NODE`, `AST_BEGIN_SUBCLASSES`, and `AST_END_SUBCLASSES`.
Here's what that looks like:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/uast-classes-list.h" "C++" 96 >}}
// Other AST nodes above...
AST_BEGIN_SUBCLASSES(Loop)
AST_NODE(DoWhile)
AST_NODE(While)
AST_BEGIN_SUBCLASSES(IndexableLoop)
AST_NODE(BracketLoop)
AST_NODE(Coforall)
AST_NODE(For)
AST_NODE(Forall)
AST_NODE(Foreach)
AST_END_SUBCLASSES(IndexableLoop)
AST_END_SUBCLASSES(Loop)
// Other AST nodes below...
{{< /githubsnippet >}}
The class hierarchy defined in this header, called `uast-classes-list.h`, is
used for a lot of things, both in the compiler itself and in some libraries
that _use_ the compiler. I'll go through the use cases in turn.
#### Tags and Dynamic Casting
First, to deal with a general absence of
[RTTI](https://en.wikipedia.org/wiki/Run-time_type_information), it is used
to declare an "tag" enum. Each AST node has a tag matching its class;
this allows us inspect the AST and perform safe casts similar to `dynamic_cast`.
Note that for parent classes (defined via `BEGIN_SUBCLASSES`), we actually
end up creating _two_ tags: one `START_...` and one `END_...`. The reason
for this will become clear in a moment.
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstTag.h" "C++" 36 >}}
enum AstTag {
#define AST_NODE(NAME) NAME ,
#define AST_BEGIN_SUBCLASSES(NAME) START_##NAME ,
#define AST_END_SUBCLASSES(NAME) END_##NAME ,
#include "chpl/uast/uast-classes-list.h"
#undef AST_NODE
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
NUM_AST_TAGS,
AST_TAG_UNKNOWN
};
{{< /githubsnippet >}}
The above snippet makes `AstTag` contain elements such as `DoWhile`,
`While`, `START_Loop`, and `END_Loop`. For convenience, we also add a couple
of other elements: `NUM_AST_TAGS`, which is
{{< sidenote "right" "numbering-node" "automatically assigned the number of tags we generated," >}}
This is because C++ assigns integer values to enum elements sequentially, starting
at zero.
{{< /sidenote >}}
and a generic "unknown tag" value.
Having generated the enum elements in this way, we can write query functions.
This way, the API consumer can write `isLoop(tag)` instead of manually performing
a comparison. Code generation here is actually split into two distinct forms
of "is bla" methods: those for concrete AST nodes (`DoWhile,` `While`) and
those for abstract base classes (`Loop`). The reason for this is simple:
only a `AstTag::DoWhile` represents a do-while loop, but both `DoWhile`
and `While` are instances of `Loop`. So, `isLoop` should return true for both.
This is where the `START_...` and `END_...` enum elements come in. Reading
the header file top-to-bottom, we first end up generating `START_Loop`,
then `DoWhile` and `While`, and then `END_Loop`. Since C++ assigns integer
value to enums sequentially, to check if a tag "extends" a base class, it's
sufficient to check if its value is greater than the `START` token, and
smaller than the `END` token -- this means it was declared within the
matching pair of `BEGIN_SUBCLASSES` and `END_SUBCLASES`.
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstTag.h" "C++" 59 >}}
// define is___ for leaf and regular nodes
// (not yet for abstract parent classes)
#define AST_NODE(NAME) \
static inline bool is##NAME(AstTag tag) { \
return tag == NAME; \
}
#define AST_BEGIN_SUBCLASSES(NAME)
#define AST_END_SUBCLASSES(NAME)
// Apply the above macros to uast-classes-list.h
#include "chpl/uast/uast-classes-list.h"
// clear the macros
#undef AST_NODE
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
// define is___ for abstract parent classes
#define AST_NODE(NAME)
#define AST_BEGIN_SUBCLASSES(NAME) \
static inline bool is##NAME(AstTag tag) { \
return START_##NAME < tag && tag < END_##NAME; \
}
#define AST_END_SUBCLASSES(NAME)
// Apply the above macros to uast-classes-list.h
#include "chpl/uast/uast-classes-list.h"
// clear the macros
#undef AST_NODE
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
{{< /githubsnippet >}}
These helpers are quite convenient. Here are a few examples of what we end up
with:
```C++
isFor(AstTag::For) // Returns true; a 'for' loop is indeed a 'for' loop.
isIndexableLoop(AstTag::For) // Returns true; a 'for' loop is "indexable" ('for i in ...')
isLoop(AstTag::For) // Returns true; a 'for' loop is a loop.
isFor(AstTag::While) // Returns false; a 'while' loop is not a 'for' loop.
isIndexableLoop(AstTag::While) // Returns false; a 'while' loop uses a boolean condition, not an index
isLoop(AstTag::While) // Returns true; a 'while' loop is a loop.
```
On the top-level AST node class, we generate `isWhateverNode` and
`toWhateverNode` for each AST class. Thus, user code is able to inspect the
AST and perform (checked) casts using plain methods. I omit `isWhateverNode`
here for brevity (its definition is very simple), and include only
`toWhateverNode`.
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstNode.h" "C++" 313 >}}
#define AST_TO(NAME) \
const NAME * to##NAME() const { \
return this->is##NAME() ? (const NAME *)this : nullptr; \
} \
NAME * to##NAME() { \
return this->is##NAME() ? (NAME *)this : nullptr; \
}
#define AST_NODE(NAME) AST_TO(NAME)
#define AST_LEAF(NAME) AST_TO(NAME)
#define AST_BEGIN_SUBCLASSES(NAME) AST_TO(NAME)
#define AST_END_SUBCLASSES(NAME)
// Apply the above macros to uast-classes-list.h
#include "chpl/uast/uast-classes-list.h"
// clear the macros
#undef AST_NODE
#undef AST_LEAF
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
#undef AST_TO
{{< /githubsnippet >}}
These methods are used heavily in the compiler. For example, here's a completely
random snippet of code I pulled out:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/lib/resolution/Resolver.cpp" "C++" 1161 >}}
if (auto var = decl->toVarLikeDecl()) {
// Figure out variable type based upon:
// * the type in the variable declaration
// * the initialization expression in the variable declaration
// * the initialization expression from split-init
auto typeExpr = var->typeExpression();
auto initExpr = var->initExpression();
if (auto var = decl->toVariable())
if (var->isField())
isField = true;
{{< /githubsnippet >}}
Thus, developers adding new AST nodes are not required to manually implement
the `isWhatever`, `toWhatever`, and other functions. This and a fair bit
of other AST functionality (which I will cover in the next subsection) is
automatically generating using X Macros.
{{< dialog >}}
{{< message "question" "reader" >}}
You haven't actually shown how the AST node classes are declared, only the
tags. It seems implausible that they be generated using this same strategy -
doesn't each AST node have its own different methods and implementation code?
{{< /message >}}
{{< message "answer" "Daniel" >}}
You're right. The AST node classes are defined "as usual", and their constructors
must explicitly set their <code>tag</code> field to the corresponding
<code>AstTag</code> value. It's also on the person defining the new class to
extend the node that they promise to extend in <code>uast-classes-list.h</code>.
{{< /message >}}
{{< message "question" "reader" >}}
This seems like an opportunity for bugs. Nothing is stopping a developer
from returning the wrong tag, which would break the auto-casting behavior.
{{< /message >}}
{{< message "answer" "Daniel" >}}
Yes, it's not bulletproof. Just recently, a team meber found
<a href="https://github.com/chapel-lang/chapel/pull/23508"> a bug</a> in which
a node was listed to inherit from <code>AstNode</code>, but actually inherited
from <code>NamedDecl</code>. The <code>toNamedDecl</code> method would not
have worked on it, even though it inherited from the class.<br>
<br>
Still, this pattern provides the Chapel compiler with a lot of value; I will
show more use cases in the next subsection, like promised.
{{< /message >}}
{{< /dialog >}}
#### The Visitor Pattern without Double Dispatch
The Visitor Pattern is very important in general, but it's beyond ubiquitous
for us compiler developers. It helps avoid bloating AST node classes with methods
and state required for the various operations we perform on them. It also often
saves us from writing AST traversal code.
Essentially, rather than adding each new operation (e.g. convert to string,
compute the type, assign IDs) as methods on each AST node class, we extract
this code into a per-operation _visitor_. This visitor is a class that has methods
implementing the custom behavior on the AST nodes. A `visit(WhileLoop*)` method
might be used to perform the operation on 'while' loops, and `visit(ForLoop*)` might
do the same for 'for' loops. The AST nodes themselves only have a `traverse`
method that accepts a visitor, whatever it may be, and calls the appropriate
visit methods. This way, the AST node implementations remain simple and relatively
stable.
As a very simple example, suppose you wanted to count the number of loops used
in a program for an unspecified reason. You could add a `countLoops` method,
but then you've introduced a method to the AST node API for what might be a
one-time, throwaway operation. With the visitor pattern, you don't need to do
that; you can just create a new class:
```C++
struct MyVisitor {
int count = 0;
void visit(const Loop*) { count += 1; }
void visit(const AstNode*) { /* do nothing for other nodes */ }
}
int countLoops(const AstNode* root) {
MyVisitor visitor;
root->traverse(visitor);
return visitor.count;
}
```
The `traverse` method is a nice API, isn't it? It's very easy to add operations
that work on your syntax trees, without modifying them. There is still an important
open question, though: how does `traverse` know to call the right `visit` function?
If `traverse` were only defined on `AstNode*`, and it simply called `visit(this)`,
we'd always end up calling the `AstNode` version of the `visit` function. This
is because C++ doesn't dynamic dispatch
{{< sidenote "right" "vtable-note" "based on the types of method arguments." >}}
Obviously, C++ has the ability to pick the right method based on the runtime
type of the <em>receiver</em>: that's just <code>virtual</code> functions
and <code>vtable</code>s.
{{< /sidenote >}}
Statically, the call clearly accepts an `AstNode`, and nothing more specific.
The compiler therefore picks that version of the `visit` method.
The "traditional" way to solve this problem in a language like C++ or Java
is called _double dispatch_. Using our example as reference, this involves
making _each_ AST node class have its own `traverse` method. This way,
calls to `visit(this)` have more specific type information, and are resolved
to the appropriate overload. But that's more boilerplate code: each new AST
node will need to have a virtual traverse method that looks something like this:
```C++
void MyNode::traverse(Visitor& v) {
v.visit(this);
}
```
It would also require all visitors to extend from `Visitor`. So now you have:
* Boilerplate code on every AST node that looks the same but needs to be duplicated
* A parent `Visitor` class that must have a `visit` method for each AST node in
the language (so that children can override it).
* To make it easier to write code like our `MyVisitor` above, the `visit`
methods in the `Visitor` must be written such that `visit(ChildNode*)` calls
`visit(ParentNode*)` by default. Otherwise, the `Loop` overload wouldn't
have been called by the `DoWhile` overload (e.g.).
So there's a fair bit of tedious boilerplate, and more code to manually modify
when adding an AST node: you have to go and adjust the `Visitor` class with
new `visit` stub.
The reason all of this is necessary is that everyone (myself included) generally
agrees that code like the following is generally a bad idea:
```C++
struct AstNode {
void traverse(Visitor& visitor) {
if (auto forLoop = toForLoop()) {
visitor.visit(forLoop);
} else if (auto whileLoop = toWhileLoop()) {
visitor.visit(whileLoop);
} else {
// 100 more lines like this...
}
}
}
```
After all, what happens when you add a new AST node? You'd still have to modify
this list, and since everything still extends `Visitor`, you'd still need to
add a new `visit` stub there. But what if there were no base class? Instead,
what if `traverse` were a template?
```C++
struct AstNode {
template <typename VisitorType>
void traverse(VisitorType& visitor) {
if (auto forLoop = toForLoop()) {
visitor.visit(forLoop);
} else if (auto whileLoop = toWhileLoop()) {
visitor.visit(whileLoop);
} else {
// 100 more lines like this...
}
}
}
```
Note that this wouldn't be possible to write in C++ if `visit` were a virtual
method; have you ever heard of a virtual template? With code like this, the
`VisitorType` wouldn't need to define _every_ overload, as long as it had
a version for `AstNode`. Furthermore, C++'s regular overload resolution rules
would take care of calling the `Loop` overload if a more specific one for
`DoWhile` didn't exist.
The only problem that remains is that of having a 100-line if-else (which could
be a `switch` to little aesthetic benefit). But this is exactly where the
X Macro pattern shines again! We already have a list of all AST node classes,
and the code for invoking them is nearly identical. Thus, the Chapel compiler
has a `doDispatch` function (used by `traverse`) that looks like this:
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstNode.h" "C++" 377 >}}
static void doDispatch(const AstNode* ast, Visitor& v) {
switch (ast->tag()) {
#define CONVERT(NAME) \
case chpl::uast::asttags::NAME: \
{ \
v.visit((const chpl::uast::NAME*) ast); \
return; \
}
#define IGNORE(NAME) \
case chpl::uast::asttags::NAME: \
{ \
CHPL_ASSERT(false && "this code should never be run"); \
}
#define AST_NODE(NAME) CONVERT(NAME)
#define AST_BEGIN_SUBCLASSES(NAME) IGNORE(START_##NAME)
#define AST_END_SUBCLASSES(NAME) IGNORE(END_##NAME)
#include "chpl/uast/uast-classes-list.h"
IGNORE(NUM_AST_TAGS)
IGNORE(AST_TAG_UNKNOWN)
#undef AST_NODE
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
#undef CONVERT
#undef IGNORE
}
CHPL_ASSERT(false && "this code should never be run");
}
{{< /githubsnippet >}}
And that's it. We have automatically generated the traversal code, allowing
us to use the visitor pattern in what I think is a very elegant way. Assuming
a developer adding a new AST node updates the `uast-classes-list.h` header,
the traversal logic will be auto-modified to properly handle the new node.
#### Generating a Python Class Hierarchy
This is a fun one. For a while, in my spare time, I was working on
[Python bindings for Chapel](https://github.com/chapel-lang/chapel/tree/main/tools/chapel-py).
These bindings are oriented towards developing language tooling: it feels much
easier to write a language linter, auto-formatter, or maybe even a language
server in Python rather than in C++. It's definitely much easier to use Python to
develop throwaway scripts that work with Chapel programs, which is something
that developers on the Chapel team tend to do quite often.
I decided I wanted the Python AST node class hierarchy to match the C++ version.
This is convenient for many reasons, including being able to wrap methods on
parent AST nodes and have them be available through child AST nodes and having
`isinstance` work properly. It's also advantageous from the point of view
of conceptual simplicity. However, I very much did not want to write CPython
API code to define the many AST node classes that are available in the Chapel
language.
Once again, the `uast-classes-list.h` header came into play here. With little
effort, I was able to auto-generate `PyTypeObject`s for each AST node in the
class hierarchy:
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 563 >}}
#define DEFINE_PY_TYPE_FOR(NAME, TAG, FLAGS)\
PyTypeObject NAME##Type = { \
PyVarObject_HEAD_INIT(NULL, 0) \
.tp_name = #NAME, \
.tp_basicsize = sizeof(NAME##Object), \
.tp_itemsize = 0, \
.tp_flags = FLAGS, \
.tp_doc = PyDoc_STR("A Chapel " #NAME " AST node"), \
.tp_methods = (PyMethodDef*) PerNodeInfo<TAG>::methods, \
.tp_base = parentTypeFor(TAG), \
.tp_init = (initproc) NAME##Object_init, \
.tp_new = PyType_GenericNew, \
};
#define AST_NODE(NAME) DEFINE_PY_TYPE_FOR(NAME, chpl::uast::asttags::NAME, Py_TPFLAGS_DEFAULT)
#define AST_BEGIN_SUBCLASSES(NAME) DEFINE_PY_TYPE_FOR(NAME, chpl::uast::asttags::START_##NAME, Py_TPFLAGS_BASETYPE)
#define AST_END_SUBCLASSES(NAME)
#include "chpl/uast/uast-classes-list.h"
#undef AST_NODE
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
{{< /githubsnippet >}}
You may have noticed that I snuck templates into the code above. The motivation there
is to avoid writing out the (usually empty) Python method table for every single
AST node. In particular, I have a template that, by default, provides an empty
method table, which can be specialized per node to add methods when necessary.
This detail is useful for application 3 below, but not necessary to understand
the use of X Macros here.
I used the same `<` and `>` trick to generate the `parentTypeFor` each tag:
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 158 >}}
static PyTypeObject* parentTypeFor(chpl::uast::asttags::AstTag tag) {
#define AST_NODE(NAME)
#define AST_LEAF(NAME)
#define AST_BEGIN_SUBCLASSES(NAME)
#define AST_END_SUBCLASSES(NAME) \
if (tag > chpl::uast::asttags::START_##NAME && tag < chpl::uast::asttags::END_##NAME) { \
return &NAME##Type; \
}
#include "chpl/uast/uast-classes-list.h"
#include "chpl/uast/uast-classes-list.h"
#undef AST_NODE
#undef AST_LEAF
#undef AST_BEGIN_SUBCLASSES
#undef AST_END_SUBCLASSES
return &AstNodeType;
}
{{< /githubsnippet >}}
A few more invocations of the `uast-classes-list.h` macro, and I had a working
class hierarchy. I didn't explicitly mention any AST node at all; all was derived
from the Chapel compiler header. This also meant that as the language changed
and the AST class hierarchy developed, the Python bindings' code would not need
to be updated. As long as it was compiled with an up-to-date version of the
header, the hierarchy would match that present within the language.
This allows for code like the following to be written in Python:
```Python
def print_decls(mod):
"""
Print all the things declared in this Chapel module.
"""
for child in mod:
if isinstance(child, NamedDecl):
print(child.name())
```
### Application 3: CPython Method Tables and Getters
The Chapel Python bindings use the X Macro pattern another time, actually.
Like I mentioned earlier, I use [template specialization](https://en.cppreference.com/w/cpp/language/template_specialization)
to reduce the amount of boilerplate code required for declaring Python objects.
In particular, there's a general method table declared as follows:
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 541 >}}
template <chpl::uast::asttags::AstTag tag>
struct PerNodeInfo {
static constexpr PyMethodDef methods[] = {
{NULL, NULL, 0, NULL} /* Sentinel */
};
};
{{< /githubsnippet >}}
Then, when I need to add methods, I do something like the following:
```C++
template <>
struct PerNodeInfo<TheAstTag> {
static constexpr PyMethodDef methods[] = {
{"method_name", TheNode_method_name, METH_NOARGS, "Documentation string"},
// ... more like the above ...
{NULL, NULL, 0, NULL} /* Sentinel */
};
};
```
When reviewing a PR that adds more methods to the Python bindings (by
defining new `TheNode_methodname` functions and then including them in the
method table), I noticed that in the PR, the developer added some methods
but forgot to put them into the respective table, leaving them unusable by
the Python client code. This came with the additional observation that there
was a moderate amount of duplication when declaring the C++ functions and then
listing them in the table. The name (`method_name` in the code) occurred many
times.
The developer who opened the PR suggesting using X Macros to combine the
information (declaration of function and its use in the corresponding method table)
into a single list. This led to the following header file:
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/method-tables.h" "C++" 323 >}}
CLASS_BEGIN(FnCall)
METHOD_PROTOTYPE(FnCall, actuals, "Get the actuals of this FnCall node")
PLAIN_GETTER(FnCall, used_square_brackets, "Check if this FnCall was made using square brackets",
"b", return node->callUsedSquareBrackets())
CLASS_END(FnCall)
{{< /githubsnippet >}}
The `PLAIN_GETTER` macro in this case is used to define trivial getters
(precluding the need for handling the Python-object-to-AST-node conversion,
and other CPython-specific things), whereas the `METHOD_PROTOTYPE` is used
to refer to methods that needed explicit implementations. With
this, the method tables are generated as follows:
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/method-tables.h" "C++" 548 >}}
#define CLASS_BEGIN(TAG) \
template <> \
struct PerNodeInfo<chpl::uast::asttags::TAG> { \
static constexpr PyMethodDef methods[] = {
#define CLASS_END(TAG) \
{NULL, NULL, 0, NULL} /* Sentinel */ \
}; \
};
#define PLAIN_GETTER(NODE, NAME, DOCSTR, TYPESTR, BODY) \
{#NAME, NODE##Object_##NAME, METH_NOARGS, DOCSTR},
#define METHOD_PROTOTYPE(NODE, NAME, DOCSTR) \
{#NAME, NODE##Object_##NAME, METH_NOARGS, DOCSTR},
#include "method-tables.h"
{{< /githubsnippet >}}
The `CLASS_BEGIN` generates the initial `template <>` header and the code up
to the opening curly brace of the table definition. Then, for each method,
`PLAIN_GETTER` and `METHOD_PROTOTYPE` generate the relevant entries. Finally,
`CLASS_END` inserts the sentinel and the closing curly brace.
Another invocation of the macros in `method-tables.h` is used to generate the
implementations of "plain getters", which is boilerplate that I won't get into
it here, since it's pretty CPython specific.
### Discussion
I've presented to you a three applications of the pattern, in an order that happens
to be from least to most "extreme". It's possible that some of these are
over the line for using macros, especially for those who think of macros as
unfortunate remnants of C++'s past. However, I think that what I've demonstrated
demonstrates the versatility of the X Macro pattern -- feel free to apply it to
the degree that you find appropriate.
The thing I like the most pattern is that the header files read quite nicely:
you end up with a very declarative "scaffold" of what's going on. The
`uast-classes-list.h` makes for an excellent and fairly readable reference of
all the AST nodes in the Chapel compiler. The `method-tables.h` header provides
a fairly concise summary of what methods are available on what (Python) AST
node.
Of course, this approach is not without its drawbacks. Drawback zero is
the heavy use of macros: to the best of my knowledge, modern C++ tends to
discourage the usage of macros in favor of C++-specific features. Of course,
this "pure C++" preference is applicable to variable degrees in different use
cases and code bases; because of this, I won't count macros as (too much of)
a drawback.
The more significant downside is that this approach introduces a lot of dependencies
between source files. Any time the header changes, anything that uses any part
of the code generated by the header must be recompiled. Thus, if you're generating
classes, changing any one class will "taint" any code that uses _any_ of the
generated classes. In the Chapel compiler, touching the AST class hierarchy
requires a recompilation of all the AST nodes, and any compiler code that uses
the AST nodes (a lot). This is because each AST node needs access to the
`AstTag` enum, and that enum is generated from the hierarchy header.
That's all I have for today! Thanks for reading. I hope you got something useful
for your day-to-day programming out of this.