Compare commits
No commits in common. "88c5daa561a8db99ad4812219e5f97b5539918ac" and "8710a5554c52a0a08c8eabaed92f90b3acfe4fb8" have entirely different histories.
88c5daa561
...
8710a5554c
|
|
@ -1,699 +0,0 @@
|
|||
---
|
||||
title: "My Favorite C++ Pattern: X Macros"
|
||||
date: 2023-10-09T15:06:11-07:00
|
||||
draft: true
|
||||
tags: ["C++", "Chapel", "Compilers"]
|
||||
---
|
||||
|
||||
When I first joined the [Chapel](https://github.com/chapel-lang/chapel/) team,
|
||||
one pattern used in its C++-based compiler made a strong impression on me. Since
|
||||
then, I've used the pattern many more times, and have been very satisfied with
|
||||
how it turned out. However, it feels like the pattern is relatively unknown, so
|
||||
I thought I'd show it off, and some of its applications in the
|
||||
[Chapel compiler](https://github.com/chapel-lang/chapel/). I've slightly tweaked
|
||||
a lot of the snippets I directly present in this article for the sake of simpler
|
||||
presentation; I've included links to the original code (available on GitHub)
|
||||
if you want to see the unabridged version.
|
||||
|
||||
Broadly speaking, the "X Macros" pattern is about generating code. If you have a _lot_
|
||||
of repetitive code to write (declaring many variables or classes, performing
|
||||
many very similar actions, etc.), this pattern can save a lot of time, lead
|
||||
to much more maintainable code, and reduce the effort required to add _more_
|
||||
code.
|
||||
|
||||
I will introduce the pattern in its simplest form with my first example:
|
||||
[interning strings](https://en.wikipedia.org/wiki/String_interning).
|
||||
|
||||
### Application 1: String Interning
|
||||
The Chapel compiler interns a lot of its strings. This way, it can reduce the
|
||||
memory footprint of keeping identifiers in memory (every string `"x"` is
|
||||
actually the _same_ string) and make for much faster equality comparisons
|
||||
(you can just perform a pointer comparison!). Generally, a `Context` class
|
||||
is used to manage interning state. A new interned string can be constructed
|
||||
using the context object in the following manner:
|
||||
|
||||
```C++
|
||||
UniqueString::get(ctxPtr, "the string");
|
||||
```
|
||||
|
||||
Effectively, this performs a search of the currently existing unique strings.
|
||||
If one with the content (`"the string"` in this case) doesn't exist, it's
|
||||
created and registered with the `Context`. Otherwise, the existing string is
|
||||
returned. Some strings, however, occur a lot in the compiler, to the point that
|
||||
it would be inefficient to perform the whole "find-or-create" operation every
|
||||
time. One example is the `"this"` string, which is an identifier with a lot of
|
||||
special behavior in the language (much like `this` in languages such as Java).
|
||||
To support these frequent flier strings, the compiler initializes them once,
|
||||
and creates a variable per-string that can be accessed to get that string's value.
|
||||
|
||||
There's that repetitive code. Defining a brand new variable for each string,
|
||||
of which there are around 100 at the time of writing, is a lot of boilerplate.
|
||||
There are also at least two places where code needs to be added:
|
||||
{{< sidenote "right" "template-note" "once in the declaration of the variables, once in the code that initializes them." >}}
|
||||
A third use in the compiler is actually a variadic template defined over
|
||||
character arrays. The template is defined and specialized in such a way that
|
||||
you can refer to a variable by its string contents (i.e., you can write
|
||||
<code>USTR("the string")</code> instead of
|
||||
<code>theStringVariable</code>).
|
||||
{{< /sidenote >}}
|
||||
It would be very easy to accidentally modify the former but not the latter,
|
||||
especially for developers not familiar with how these "common strings" are
|
||||
implemented.
|
||||
|
||||
This is where the X Macros come in. If you look around the compiler source code,
|
||||
there's a header file that looks something like the following:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/framework/all-global-strings.h" "C++" 31 >}}
|
||||
X(align , "align")
|
||||
X(atomic , "atomic")
|
||||
X(bool_ , "bool")
|
||||
X(borrow , "borrow")
|
||||
X(borrowed , "borrowed")
|
||||
X(by , "by")
|
||||
X(bytes , "bytes")
|
||||
// A lot more of these...
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
What's This `X` thing? That right there is the essence of the pattern: the macro
|
||||
`X` _isn't defined in the header!_. Effectively, `all-global-strings.h` is just
|
||||
a list, and we can "iterate" over this list to generate some code for each
|
||||
one of its elements, in as many places as we want. What I mean by this is
|
||||
that we can then write code like this:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/framework/global-strings.h" "C++" 76 >}}
|
||||
struct GlobalStrings {
|
||||
#define X(field, str) UniqueString field;
|
||||
#include "all-global-strings.h"
|
||||
#undef X
|
||||
};
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
In this case, we define the macro `X` to ignore the value of the string (we're
|
||||
just declaring it here), and create a new `UniqueString` variable declaration.
|
||||
Since the declaration is inside the `GlobalStrings` struct, this ends up
|
||||
creating a field. Just like that, we've declared a class with over 100
|
||||
fields. Initialization is equally simple:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/lib/framework/Context.cpp" "C++" 49 >}}
|
||||
GlobalStrings globalStrings;
|
||||
Context rootContext;
|
||||
|
||||
static void initGlobalStrings() {
|
||||
#define X(field, str) globalStrings.field = UniqueString::get(&rootContext, str);
|
||||
#include "chpl/framework/all-global-strings.h"
|
||||
#undef X
|
||||
}
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
With this, we've completely automated the code for for both declaring and
|
||||
initializing all 100 of our unique strings. Adding a new string doesn't require
|
||||
a developer to know all of the places where this is implemented: just by
|
||||
modifying the `all-global-strings.h` header with a new call to `X`, they can
|
||||
add both a new variable and code to initialize it. Pretty robust!
|
||||
|
||||
### Application 2: AST Class Hierarchy
|
||||
|
||||
Altough the interned strings are an excellent first example, it wasn't the
|
||||
first usage of X Macros that I encountered in the Chapel compiler. Beyond
|
||||
strings, the compiler uses X Macros to represent the whole class hierarchy
|
||||
of [abstract syntax tree (AST)](https://en.wikipedia.org/wiki/Abstract_syntax_tree)
|
||||
nodes that it uses. Here, the code is actually a bit more complicated; the
|
||||
class hierarchy isn't a _list_ like the strings were; it is itself a tree.
|
||||
To represent such a structure, we need more than a single `X` macro; the
|
||||
compiler went with `AST_NODE`, `AST_BEGIN_SUBCLASSES`, and `AST_END_SUBCLASSES`.
|
||||
Here's what that looks like:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/uast-classes-list.h" "C++" 96 >}}
|
||||
// Other AST nodes above...
|
||||
|
||||
AST_BEGIN_SUBCLASSES(Loop)
|
||||
AST_NODE(DoWhile)
|
||||
AST_NODE(While)
|
||||
|
||||
AST_BEGIN_SUBCLASSES(IndexableLoop)
|
||||
AST_NODE(BracketLoop)
|
||||
AST_NODE(Coforall)
|
||||
AST_NODE(For)
|
||||
AST_NODE(Forall)
|
||||
AST_NODE(Foreach)
|
||||
AST_END_SUBCLASSES(IndexableLoop)
|
||||
|
||||
AST_END_SUBCLASSES(Loop)
|
||||
|
||||
// Other AST nodes below...
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
The class hierarchy defined in this header, called `uast-classes-list.h`, is
|
||||
used for a lot of things, both in the compiler itself and in some libraries
|
||||
that _use_ the compiler. I'll go through the use cases in turn.
|
||||
|
||||
#### Tags and Dynamic Casting
|
||||
First, to deal with a general absence of
|
||||
[RTTI](https://en.wikipedia.org/wiki/Run-time_type_information), it is used
|
||||
to declare an "tag" enum. Each AST node has a tag matching its class;
|
||||
this allows us inspect the AST and perform safe casts similar to `dynamic_cast`.
|
||||
Note that for parent classes (defined via `BEGIN_SUBCLASSES`), we actually
|
||||
end up creating _two_ tags: one `START_...` and one `END_...`. The reason
|
||||
for this will become clear in a moment.
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstTag.h" "C++" 36 >}}
|
||||
enum AstTag {
|
||||
#define AST_NODE(NAME) NAME ,
|
||||
#define AST_BEGIN_SUBCLASSES(NAME) START_##NAME ,
|
||||
#define AST_END_SUBCLASSES(NAME) END_##NAME ,
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
#undef AST_NODE
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
NUM_AST_TAGS,
|
||||
AST_TAG_UNKNOWN
|
||||
};
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
The above snippet makes `AstTag` contain elements such as `DoWhile`,
|
||||
`While`, `START_Loop`, and `END_Loop`. For convenience, we also add a couple
|
||||
of other elements: `NUM_AST_TAGS`, which is
|
||||
{{< sidenote "right" "numbering-node" "automatically assigned the number of tags we generated," >}}
|
||||
This is because C++ assigns integer values to enum elements sequentially, starting
|
||||
at zero.
|
||||
{{< /sidenote >}}
|
||||
and a generic "unknown tag" value.
|
||||
|
||||
Having generated the enum elements in this way, we can write query functions.
|
||||
This way, the API consumer can write `isLoop(tag)` instead of manually performing
|
||||
a comparison. Code generation here is actually split into two distinct forms
|
||||
of "is bla" methods: those for concrete AST nodes (`DoWhile,` `While`) and
|
||||
those for abstract base classes (`Loop`). The reason for this is simple:
|
||||
only a `AstTag::DoWhile` represents a do-while loop, but both `DoWhile`
|
||||
and `While` are instances of `Loop`. So, `isLoop` should return true for both.
|
||||
|
||||
This is where the `START_...` and `END_...` enum elements come in. Reading
|
||||
the header file top-to-bottom, we first end up generating `START_Loop`,
|
||||
then `DoWhile` and `While`, and then `END_Loop`. Since C++ assigns integer
|
||||
value to enums sequentially, to check if a tag "extends" a base class, it's
|
||||
sufficient to check if its value is greater than the `START` token, and
|
||||
smaller than the `END` token -- this means it was declared within the
|
||||
matching pair of `BEGIN_SUBCLASSES` and `END_SUBCLASES`.
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstTag.h" "C++" 59 >}}
|
||||
// define is___ for leaf and regular nodes
|
||||
// (not yet for abstract parent classes)
|
||||
#define AST_NODE(NAME) \
|
||||
static inline bool is##NAME(AstTag tag) { \
|
||||
return tag == NAME; \
|
||||
}
|
||||
#define AST_BEGIN_SUBCLASSES(NAME)
|
||||
#define AST_END_SUBCLASSES(NAME)
|
||||
// Apply the above macros to uast-classes-list.h
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
// clear the macros
|
||||
#undef AST_NODE
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
|
||||
// define is___ for abstract parent classes
|
||||
#define AST_NODE(NAME)
|
||||
#define AST_BEGIN_SUBCLASSES(NAME) \
|
||||
static inline bool is##NAME(AstTag tag) { \
|
||||
return START_##NAME < tag && tag < END_##NAME; \
|
||||
}
|
||||
#define AST_END_SUBCLASSES(NAME)
|
||||
// Apply the above macros to uast-classes-list.h
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
// clear the macros
|
||||
#undef AST_NODE
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
These helpers are quite convenient. Here are a few examples of what we end up
|
||||
with:
|
||||
|
||||
```C++
|
||||
isFor(AstTag::For) // Returns true; a 'for' loop is indeed a 'for' loop.
|
||||
isIndexableLoop(AstTag::For) // Returns true; a 'for' loop is "indexable" ('for i in ...')
|
||||
isLoop(AstTag::For) // Returns true; a 'for' loop is a loop.
|
||||
isFor(AstTag::While) // Returns false; a 'while' loop is not a 'for' loop.
|
||||
isIndexableLoop(AstTag::While) // Returns false; a 'while' loop uses a boolean condition, not an index
|
||||
isLoop(AstTag::While) // Returns true; a 'while' loop is a loop.
|
||||
```
|
||||
|
||||
On the top-level AST node class, we generate `isWhateverNode` and
|
||||
`toWhateverNode` for each AST class. Thus, user code is able to inspect the
|
||||
AST and perform (checked) casts using plain methods. I omit `isWhateverNode`
|
||||
here for brevity (its definition is very simple), and include only
|
||||
`toWhateverNode`.
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstNode.h" "C++" 313 >}}
|
||||
#define AST_TO(NAME) \
|
||||
const NAME * to##NAME() const { \
|
||||
return this->is##NAME() ? (const NAME *)this : nullptr; \
|
||||
} \
|
||||
NAME * to##NAME() { \
|
||||
return this->is##NAME() ? (NAME *)this : nullptr; \
|
||||
}
|
||||
#define AST_NODE(NAME) AST_TO(NAME)
|
||||
#define AST_LEAF(NAME) AST_TO(NAME)
|
||||
#define AST_BEGIN_SUBCLASSES(NAME) AST_TO(NAME)
|
||||
#define AST_END_SUBCLASSES(NAME)
|
||||
// Apply the above macros to uast-classes-list.h
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
// clear the macros
|
||||
#undef AST_NODE
|
||||
#undef AST_LEAF
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
#undef AST_TO
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
These methods are used heavily in the compiler. For example, here's a completely
|
||||
random snippet of code I pulled out:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/lib/resolution/Resolver.cpp" "C++" 1161 >}}
|
||||
if (auto var = decl->toVarLikeDecl()) {
|
||||
// Figure out variable type based upon:
|
||||
// * the type in the variable declaration
|
||||
// * the initialization expression in the variable declaration
|
||||
// * the initialization expression from split-init
|
||||
|
||||
auto typeExpr = var->typeExpression();
|
||||
auto initExpr = var->initExpression();
|
||||
|
||||
if (auto var = decl->toVariable())
|
||||
if (var->isField())
|
||||
isField = true;
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
Thus, developers adding new AST nodes are not required to manually implement
|
||||
the `isWhatever`, `toWhatever`, and other functions. This and a fair bit
|
||||
of other AST functionality (which I will cover in the next subsection) is
|
||||
automatically generating using X Macros.
|
||||
|
||||
{{< dialog >}}
|
||||
{{< message "question" "reader" >}}
|
||||
You haven't actually shown how the AST node classes are declared, only the
|
||||
tags. It seems implausible that they be generated using this same strategy -
|
||||
doesn't each AST node have its own different methods and implementation code?
|
||||
{{< /message >}}
|
||||
{{< message "answer" "Daniel" >}}
|
||||
You're right. The AST node classes are defined "as usual", and their constructors
|
||||
must explicitly set their <code>tag</code> field to the corresponding
|
||||
<code>AstTag</code> value. It's also on the person defining the new class to
|
||||
extend the node that they promise to extend in <code>uast-classes-list.h</code>.
|
||||
{{< /message >}}
|
||||
{{< message "question" "reader" >}}
|
||||
This seems like an opportunity for bugs. Nothing is stopping a developer
|
||||
from returning the wrong tag, which would break the auto-casting behavior.
|
||||
{{< /message >}}
|
||||
{{< message "answer" "Daniel" >}}
|
||||
Yes, it's not bulletproof. Just recently, a team meber found
|
||||
<a href="https://github.com/chapel-lang/chapel/pull/23508"> a bug</a> in which
|
||||
a node was listed to inherit from <code>AstNode</code>, but actually inherited
|
||||
from <code>NamedDecl</code>. The <code>toNamedDecl</code> method would not
|
||||
have worked on it, even though it inherited from the class.<br>
|
||||
<br>
|
||||
Still, this pattern provides the Chapel compiler with a lot of value; I will
|
||||
show more use cases in the next subsection, like promised.
|
||||
{{< /message >}}
|
||||
{{< /dialog >}}
|
||||
|
||||
#### The Visitor Pattern without Double Dispatch
|
||||
|
||||
The Visitor Pattern is very important in general, but it's beyond ubiquitous
|
||||
for us compiler developers. It helps avoid bloating AST node classes with methods
|
||||
and state required for the various operations we perform on them. It also often
|
||||
saves us from writing AST traversal code.
|
||||
|
||||
Essentially, rather than adding each new operation (e.g. convert to string,
|
||||
compute the type, assign IDs) as methods on each AST node class, we extract
|
||||
this code into a per-operation _visitor_. This visitor is a class that has methods
|
||||
implementing the custom behavior on the AST nodes. A `visit(WhileLoop*)` method
|
||||
might be used to perform the operation on 'while' loops, and `visit(ForLoop*)` might
|
||||
do the same for 'for' loops. The AST nodes themselves only have a `traverse`
|
||||
method that accepts a visitor, whatever it may be, and calls the appropriate
|
||||
visit methods. This way, the AST node implementations remain simple and relatively
|
||||
stable.
|
||||
|
||||
As a very simple example, suppose you wanted to count the number of loops used
|
||||
in a program for an unspecified reason. You could add a `countLoops` method,
|
||||
but then you've introduced a method to the AST node API for what might be a
|
||||
one-time, throwaway operation. With the visitor pattern, you don't need to do
|
||||
that; you can just create a new class:
|
||||
|
||||
```C++
|
||||
struct MyVisitor {
|
||||
int count = 0;
|
||||
|
||||
void visit(const Loop*) { count += 1; }
|
||||
void visit(const AstNode*) { /* do nothing for other nodes */ }
|
||||
}
|
||||
|
||||
int countLoops(const AstNode* root) {
|
||||
MyVisitor visitor;
|
||||
root->traverse(visitor);
|
||||
return visitor.count;
|
||||
}
|
||||
```
|
||||
|
||||
The `traverse` method is a nice API, isn't it? It's very easy to add operations
|
||||
that work on your syntax trees, without modifying them. There is still an important
|
||||
open question, though: how does `traverse` know to call the right `visit` function?
|
||||
|
||||
If `traverse` were only defined on `AstNode*`, and it simply called `visit(this)`,
|
||||
we'd always end up calling the `AstNode` version of the `visit` function. This
|
||||
is because C++ doesn't dynamic dispatch
|
||||
{{< sidenote "right" "vtable-note" "based on the types of method arguments." >}}
|
||||
Obviously, C++ has the ability to pick the right method based on the runtime
|
||||
type of the <em>receiver</em>: that's just <code>virtual</code> functions
|
||||
and <code>vtable</code>s.
|
||||
{{< /sidenote >}}
|
||||
Statically, the call clearly accepts an `AstNode`, and nothing more specific.
|
||||
The compiler therefore picks that version of the `visit` method.
|
||||
|
||||
The "traditional" way to solve this problem in a language like C++ or Java
|
||||
is called _double dispatch_. Using our example as reference, this involves
|
||||
making _each_ AST node class have its own `traverse` method. This way,
|
||||
calls to `visit(this)` have more specific type information, and are resolved
|
||||
to the appropriate overload. But that's more boilerplate code: each new AST
|
||||
node will need to have a virtual traverse method that looks something like this:
|
||||
|
||||
```C++
|
||||
void MyNode::traverse(Visitor& v) {
|
||||
v.visit(this);
|
||||
}
|
||||
```
|
||||
|
||||
It would also require all visitors to extend from `Visitor`. So now you have:
|
||||
|
||||
* Boilerplate code on every AST node that looks the same but needs to be duplicated
|
||||
* A parent `Visitor` class that must have a `visit` method for each AST node in
|
||||
the language (so that children can override it).
|
||||
* To make it easier to write code like our `MyVisitor` above, the `visit`
|
||||
methods in the `Visitor` must be written such that `visit(ChildNode*)` calls
|
||||
`visit(ParentNode*)` by default. Otherwise, the `Loop` overload wouldn't
|
||||
have been called by the `DoWhile` overload (e.g.).
|
||||
|
||||
So there's a fair bit of tedious boilerplate, and more code to manually modify
|
||||
when adding an AST node: you have to go and adjust the `Visitor` class with
|
||||
new `visit` stub.
|
||||
|
||||
The reason all of this is necessary is that everyone (myself included) generally
|
||||
agrees that code like the following is generally a bad idea:
|
||||
|
||||
```C++
|
||||
struct AstNode {
|
||||
void traverse(Visitor& visitor) {
|
||||
if (auto forLoop = toForLoop()) {
|
||||
visitor.visit(forLoop);
|
||||
} else if (auto whileLoop = toWhileLoop()) {
|
||||
visitor.visit(whileLoop);
|
||||
} else {
|
||||
// 100 more lines like this...
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
After all, what happens when you add a new AST node? You'd still have to modify
|
||||
this list, and since everything still extends `Visitor`, you'd still need to
|
||||
add a new `visit` stub there. But what if there were no base class? Instead,
|
||||
what if `traverse` were a template?
|
||||
|
||||
```C++
|
||||
struct AstNode {
|
||||
template <typename VisitorType>
|
||||
void traverse(VisitorType& visitor) {
|
||||
if (auto forLoop = toForLoop()) {
|
||||
visitor.visit(forLoop);
|
||||
} else if (auto whileLoop = toWhileLoop()) {
|
||||
visitor.visit(whileLoop);
|
||||
} else {
|
||||
// 100 more lines like this...
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Note that this wouldn't be possible to write in C++ if `visit` were a virtual
|
||||
method; have you ever heard of a virtual template? With code like this, the
|
||||
`VisitorType` wouldn't need to define _every_ overload, as long as it had
|
||||
a version for `AstNode`. Furthermore, C++'s regular overload resolution rules
|
||||
would take care of calling the `Loop` overload if a more specific one for
|
||||
`DoWhile` didn't exist.
|
||||
|
||||
The only problem that remains is that of having a 100-line if-else (which could
|
||||
be a `switch` to little aesthetic benefit). But this is exactly where the
|
||||
X Macro pattern shines again! We already have a list of all AST node classes,
|
||||
and the code for invoking them is nearly identical. Thus, the Chapel compiler
|
||||
has a `doDispatch` function (used by `traverse`) that looks like this:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "cd108338d321d0b3edf6258e0b2a58459d88a348" "frontend/include/chpl/uast/AstNode.h" "C++" 377 >}}
|
||||
static void doDispatch(const AstNode* ast, Visitor& v) {
|
||||
|
||||
switch (ast->tag()) {
|
||||
#define CONVERT(NAME) \
|
||||
case chpl::uast::asttags::NAME: \
|
||||
{ \
|
||||
v.visit((const chpl::uast::NAME*) ast); \
|
||||
return; \
|
||||
}
|
||||
|
||||
#define IGNORE(NAME) \
|
||||
case chpl::uast::asttags::NAME: \
|
||||
{ \
|
||||
CHPL_ASSERT(false && "this code should never be run"); \
|
||||
}
|
||||
|
||||
#define AST_NODE(NAME) CONVERT(NAME)
|
||||
#define AST_BEGIN_SUBCLASSES(NAME) IGNORE(START_##NAME)
|
||||
#define AST_END_SUBCLASSES(NAME) IGNORE(END_##NAME)
|
||||
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
|
||||
IGNORE(NUM_AST_TAGS)
|
||||
IGNORE(AST_TAG_UNKNOWN)
|
||||
|
||||
#undef AST_NODE
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
#undef CONVERT
|
||||
#undef IGNORE
|
||||
}
|
||||
|
||||
CHPL_ASSERT(false && "this code should never be run");
|
||||
}
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
And that's it. We have automatically generated the traversal code, allowing
|
||||
us to use the visitor pattern in what I think is a very elegant way. Assuming
|
||||
a developer adding a new AST node updates the `uast-classes-list.h` header,
|
||||
the traversal logic will be auto-modified to properly handle the new node.
|
||||
|
||||
#### Generating a Python Class Hierarchy
|
||||
|
||||
This is a fun one. For a while, in my spare time, I was working on
|
||||
[Python bindings for Chapel](https://github.com/chapel-lang/chapel/tree/main/tools/chapel-py).
|
||||
These bindings are oriented towards developing language tooling: it feels much
|
||||
easier to write a language linter, auto-formatter, or maybe even a language
|
||||
server in Python rather than in C++. It's definitely much easier to use Python to
|
||||
develop throwaway scripts that work with Chapel programs, which is something
|
||||
that developers on the Chapel team tend to do quite often.
|
||||
|
||||
I decided I wanted the Python AST node class hierarchy to match the C++ version.
|
||||
This is convenient for many reasons, including being able to wrap methods on
|
||||
parent AST nodes and have them be available through child AST nodes and having
|
||||
`isinstance` work properly. It's also advantageous from the point of view
|
||||
of conceptual simplicity. However, I very much did not want to write CPython
|
||||
API code to define the many AST node classes that are available in the Chapel
|
||||
language.
|
||||
|
||||
Once again, the `uast-classes-list.h` header came into play here. With little
|
||||
effort, I was able to auto-generate `PyTypeObject`s for each AST node in the
|
||||
class hierarchy:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 563 >}}
|
||||
#define DEFINE_PY_TYPE_FOR(NAME, TAG, FLAGS)\
|
||||
PyTypeObject NAME##Type = { \
|
||||
PyVarObject_HEAD_INIT(NULL, 0) \
|
||||
.tp_name = #NAME, \
|
||||
.tp_basicsize = sizeof(NAME##Object), \
|
||||
.tp_itemsize = 0, \
|
||||
.tp_flags = FLAGS, \
|
||||
.tp_doc = PyDoc_STR("A Chapel " #NAME " AST node"), \
|
||||
.tp_methods = (PyMethodDef*) PerNodeInfo<TAG>::methods, \
|
||||
.tp_base = parentTypeFor(TAG), \
|
||||
.tp_init = (initproc) NAME##Object_init, \
|
||||
.tp_new = PyType_GenericNew, \
|
||||
};
|
||||
|
||||
#define AST_NODE(NAME) DEFINE_PY_TYPE_FOR(NAME, chpl::uast::asttags::NAME, Py_TPFLAGS_DEFAULT)
|
||||
#define AST_BEGIN_SUBCLASSES(NAME) DEFINE_PY_TYPE_FOR(NAME, chpl::uast::asttags::START_##NAME, Py_TPFLAGS_BASETYPE)
|
||||
#define AST_END_SUBCLASSES(NAME)
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
#undef AST_NODE
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
You may have noticed that I snuck templates into the code above. The motivation there
|
||||
is to avoid writing out the (usually empty) Python method table for every single
|
||||
AST node. In particular, I have a template that, by default, provides an empty
|
||||
method table, which can be specialized per node to add methods when necessary.
|
||||
This detail is useful for application 3 below, but not necessary to understand
|
||||
the use of X Macros here.
|
||||
|
||||
I used the same `<` and `>` trick to generate the `parentTypeFor` each tag:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 158 >}}
|
||||
static PyTypeObject* parentTypeFor(chpl::uast::asttags::AstTag tag) {
|
||||
#define AST_NODE(NAME)
|
||||
#define AST_LEAF(NAME)
|
||||
#define AST_BEGIN_SUBCLASSES(NAME)
|
||||
#define AST_END_SUBCLASSES(NAME) \
|
||||
if (tag > chpl::uast::asttags::START_##NAME && tag < chpl::uast::asttags::END_##NAME) { \
|
||||
return &NAME##Type; \
|
||||
}
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
#include "chpl/uast/uast-classes-list.h"
|
||||
#undef AST_NODE
|
||||
#undef AST_LEAF
|
||||
#undef AST_BEGIN_SUBCLASSES
|
||||
#undef AST_END_SUBCLASSES
|
||||
return &AstNodeType;
|
||||
}
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
A few more invocations of the `uast-classes-list.h` macro, and I had a working
|
||||
class hierarchy. I didn't explicitly mention any AST node at all; all was derived
|
||||
from the Chapel compiler header. This also meant that as the language changed
|
||||
and the AST class hierarchy developed, the Python bindings' code would not need
|
||||
to be updated. As long as it was compiled with an up-to-date version of the
|
||||
header, the hierarchy would match that present within the language.
|
||||
|
||||
This allows for code like the following to be written in Python:
|
||||
|
||||
```Python
|
||||
def print_decls(mod):
|
||||
"""
|
||||
Print all the things declared in this Chapel module.
|
||||
"""
|
||||
for child in mod:
|
||||
if isinstance(child, NamedDecl):
|
||||
print(child.name())
|
||||
```
|
||||
|
||||
### Application 3: CPython Method Tables and Getters
|
||||
|
||||
The Chapel Python bindings use the X Macro pattern another time, actually.
|
||||
Like I mentioned earlier, I use [template specialization](https://en.cppreference.com/w/cpp/language/template_specialization)
|
||||
to reduce the amount of boilerplate code required for declaring Python objects.
|
||||
In particular, there's a general method table declared as follows:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/chapel.cpp" "C++" 541 >}}
|
||||
template <chpl::uast::asttags::AstTag tag>
|
||||
struct PerNodeInfo {
|
||||
static constexpr PyMethodDef methods[] = {
|
||||
{NULL, NULL, 0, NULL} /* Sentinel */
|
||||
};
|
||||
};
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
Then, when I need to add methods, I do something like the following:
|
||||
|
||||
```C++
|
||||
template <>
|
||||
struct PerNodeInfo<TheAstTag> {
|
||||
static constexpr PyMethodDef methods[] = {
|
||||
{"method_name", TheNode_method_name, METH_NOARGS, "Documentation string"},
|
||||
// ... more like the above ...
|
||||
{NULL, NULL, 0, NULL} /* Sentinel */
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
When reviewing a PR that adds more methods to the Python bindings (by
|
||||
defining new `TheNode_methodname` functions and then including them in the
|
||||
method table), I noticed that in the PR, the developer added some methods
|
||||
but forgot to put them into the respective table, leaving them unusable by
|
||||
the Python client code. This came with the additional observation that there
|
||||
was a moderate amount of duplication when declaring the C++ functions and then
|
||||
listing them in the table. The name (`method_name` in the code) occurred many
|
||||
times.
|
||||
|
||||
The developer who opened the PR suggesting using X Macros to combine the
|
||||
information (declaration of function and its use in the corresponding method table)
|
||||
into a single list. This led to the following header file:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/method-tables.h" "C++" 323 >}}
|
||||
CLASS_BEGIN(FnCall)
|
||||
METHOD_PROTOTYPE(FnCall, actuals, "Get the actuals of this FnCall node")
|
||||
PLAIN_GETTER(FnCall, used_square_brackets, "Check if this FnCall was made using square brackets",
|
||||
"b", return node->callUsedSquareBrackets())
|
||||
CLASS_END(FnCall)
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
The `PLAIN_GETTER` macro in this case is used to define trivial getters
|
||||
(precluding the need for handling the Python-object-to-AST-node conversion,
|
||||
and other CPython-specific things), whereas the `METHOD_PROTOTYPE` is used
|
||||
to refer to methods that needed explicit implementations. With
|
||||
this, the method tables are generated as follows:
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "31a296e80cfb69bfc0c79a48d5cc9e8891f54818" "tools/chapel-py/method-tables.h" "C++" 548 >}}
|
||||
#define CLASS_BEGIN(TAG) \
|
||||
template <> \
|
||||
struct PerNodeInfo<chpl::uast::asttags::TAG> { \
|
||||
static constexpr PyMethodDef methods[] = {
|
||||
#define CLASS_END(TAG) \
|
||||
{NULL, NULL, 0, NULL} /* Sentinel */ \
|
||||
}; \
|
||||
};
|
||||
#define PLAIN_GETTER(NODE, NAME, DOCSTR, TYPESTR, BODY) \
|
||||
{#NAME, NODE##Object_##NAME, METH_NOARGS, DOCSTR},
|
||||
#define METHOD_PROTOTYPE(NODE, NAME, DOCSTR) \
|
||||
{#NAME, NODE##Object_##NAME, METH_NOARGS, DOCSTR},
|
||||
#include "method-tables.h"
|
||||
{{< /githubsnippet >}}
|
||||
|
||||
The `CLASS_BEGIN` generates the initial `template <>` header and the code up
|
||||
to the opening curly brace of the table definition. Then, for each method,
|
||||
`PLAIN_GETTER` and `METHOD_PROTOTYPE` generate the relevant entries. Finally,
|
||||
`CLASS_END` inserts the sentinel and the closing curly brace.
|
||||
|
||||
Another invocation of the macros in `method-tables.h` is used to generate the
|
||||
implementations of "plain getters", which is boilerplate that I won't get into
|
||||
it here, since it's pretty CPython specific.
|
||||
|
||||
### Discussion
|
||||
|
||||
I've presented to you a three applications of the pattern, in an order that happens
|
||||
to be from least to most "extreme". It's possible that some of these are
|
||||
over the line for using macros, especially for those who think of macros as
|
||||
unfortunate remnants of C++'s past. However, I think that what I've demonstrated
|
||||
demonstrates the versatility of the X Macro pattern -- feel free to apply it to
|
||||
the degree that you find appropriate.
|
||||
|
||||
The thing I like the most pattern is that the header files read quite nicely:
|
||||
you end up with a very declarative "scaffold" of what's going on. The
|
||||
`uast-classes-list.h` makes for an excellent and fairly readable reference of
|
||||
all the AST nodes in the Chapel compiler. The `method-tables.h` header provides
|
||||
a fairly concise summary of what methods are available on what (Python) AST
|
||||
node.
|
||||
|
||||
Of course, this approach is not without its drawbacks. Drawback zero is
|
||||
the heavy use of macros: to the best of my knowledge, modern C++ tends to
|
||||
discourage the usage of macros in favor of C++-specific features. Of course,
|
||||
this "pure C++" preference is applicable to variable degrees in different use
|
||||
cases and code bases; because of this, I won't count macros as (too much of)
|
||||
a drawback.
|
||||
|
||||
The more significant downside is that this approach introduces a lot of dependencies
|
||||
between source files. Any time the header changes, anything that uses any part
|
||||
of the code generated by the header must be recompiled. Thus, if you're generating
|
||||
classes, changing any one class will "taint" any code that uses _any_ of the
|
||||
generated classes. In the Chapel compiler, touching the AST class hierarchy
|
||||
requires a recompilation of all the AST nodes, and any compiler code that uses
|
||||
the AST nodes (a lot). This is because each AST node needs access to the
|
||||
`AstTag` enum, and that enum is generated from the hierarchy header.
|
||||
|
||||
That's all I have for today! Thanks for reading. I hope you got something useful
|
||||
for your day-to-day programming out of this.
|
||||
|
|
@ -1,7 +1,7 @@
|
|||
---
|
||||
title: "Proving My Compiler Code Incorrect With Alloy"
|
||||
date: 2023-06-04T21:56:00-07:00
|
||||
tags: ["Chapel", "Compilers", "Alloy"]
|
||||
tags: ["Compilers", "Alloy"]
|
||||
description: "In this post, I apply Alloy to a piece of code in the Chapel compiler to find a bug."
|
||||
---
|
||||
|
||||
|
|
@ -435,7 +435,7 @@ apply Alloy. First, here's the code in C++ that defines the various flags
|
|||
I'd be working with (though I've omitted flags that are not currently
|
||||
used in the implementation).
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "390187df504f58cbf721a87d5632cdf7ea37563f" "frontend/include/chpl/resolution/scope-types.h" "C++" 45 >}}
|
||||
```C++
|
||||
enum {
|
||||
/** Public */
|
||||
PUBLIC = 1,
|
||||
|
|
@ -450,7 +450,7 @@ used in the implementation).
|
|||
/** Something other than a method declaration */
|
||||
NOT_METHOD = 128,
|
||||
};
|
||||
{{< /githubsnippet >}}
|
||||
```
|
||||
|
||||
These are the flags that we model using a `Bitset`: `PUBLIC`,
|
||||
`METHOD_FIELD`, and `METHOD` are modeled using `positiveFlags`, and
|
||||
|
|
@ -461,7 +461,7 @@ system (particularly when we _know_ it's an approximation). However, the
|
|||
flags aren't used arbitrarily; in fact, it wasn't too hard to track down the
|
||||
most important place in the code where bitsets are built.
|
||||
|
||||
{{< githubsnippet "chapel-lang/chapel" "390187df504f58cbf721a87d5632cdf7ea37563f" "frontend/lib/resolution/scope-queries.cpp" "C++" 914 >}}
|
||||
```C++{linenos=true, linenostart=914}
|
||||
IdAndFlags::Flags curFilter = 0;
|
||||
/* ... some unrelated code ... */
|
||||
if (skipPrivateVisibilities) {
|
||||
|
|
@ -472,7 +472,7 @@ most important place in the code where bitsets are built.
|
|||
} else if (!includeMethods && receiverScopes.empty()) {
|
||||
curFilter |= IdAndFlags::NOT_METHOD;
|
||||
}
|
||||
{{< /githubsnippet >}}
|
||||
```
|
||||
|
||||
The above code converts the current search parameters into `Bitfield`
|
||||
flags. For instance, if a `use` statement is being processed that doesn't
|
||||
|
|
|
|||
|
|
@ -1 +1 @@
|
|||
Subproject commit b5761d4d817c349fea2c1a81b1f6e5c9e391e43e
|
||||
Subproject commit 502190055d1eae191631e39ef315c5aeea9678dc
|
||||
Loading…
Reference in New Issue
Block a user