Compare commits

..

No commits in common. "b66c58b68e9c507f47b5b9ef8fb8d85281502ffe" and "6a168f2fe144850ed3a81b796e07266cbf80f382" have entirely different histories.

2 changed files with 46 additions and 349 deletions

View File

@ -1,7 +1,8 @@
--- ---
title: "Integrating Agda's HTML Output with Hugo" title: "Integrating Agda's HTML Output with Hugo"
date: 2024-05-30T00:29:26-07:00 date: 2024-05-25T21:02:10-07:00
tags: ["Agda", "Hugo", "Ruby"] draft: true
tags: ["Agda", "Hugo", "Ruby", "Nix"]
--- ---
One of my favorite things about Agda are its clickable HTML pages. If you don't One of my favorite things about Agda are its clickable HTML pages. If you don't
@ -36,29 +37,13 @@ And here's the denotational semantics for that expression:
Notice that you can click `Expr`, `__`, `⟦`, etc.! All of this integrates Notice that you can click `Expr`, `__`, `⟦`, etc.! All of this integrates
with my existing Hugo site, and only required a little bit of additional with my existing Hugo site, and only required a little bit of additional
metadata to make it work. The conversion is implemented as metadata to make it work.
[a Ruby script](https://dev.danilafe.com/Web-Projects/blog-static/src/commit/04f12b545d5692a78b1a2f13ef968417c749e295/agda.rb);
this script transfers the link structure from an Agda-generated documentation
HTML file onto lightly-annotated Hugo code blocks.
To use the script, your Hugo theme (or your Markdown content) must Now, the details. Right now, the solution is pretty tailored to my site
annotate the code blocks with several properties: and workflow, but the core of the script -- a piece that transfers links
* `data-agda-block`, which marks code that needs to be processed. from an Agda HTML file into a syntax highlighted Hugo HTML block -- should
* `data-file-path`, which tells the script what Agda file provided the
code in the block, and therefore what Agda HTML file should be searched
for links.
* `data-first-line` and `data-last-line`, which tell the script what
section of the Agda HTML file should be searched for said links.
Given this -- and a couple of other assumptions, such as that all Agda
projects are in a `code/<project>` folder, the script post-processes
the HTML files automatically. Right now, the solution is pretty tailored to my
site and workflow, but the core of the script -- the piece that transfers links
from an Agda HTML file into a syntax-highlighted Hugo HTML block -- should
be fairly reusable. be fairly reusable.
Now, the details.
### The Constraints ### The Constraints
The goal was simple: to allow the code blocks on my Hugo-generated site to The goal was simple: to allow the code blocks on my Hugo-generated site to
have links that take the user to the definition of a given symbol. have links that take the user to the definition of a given symbol.
@ -107,7 +92,6 @@ I've hit all of these requirements; now it's time to dig into how I got there.
### Implementation ### Implementation
#### Processing Agda's HTML Output
It's pretty much a no-go to try to resolve Agda from Hugo, or perform some It's pretty much a no-go to try to resolve Agda from Hugo, or perform some
sort of "heuristic" to detect cross-links. Agda is a very complex programming sort of "heuristic" to detect cross-links. Agda is a very complex programming
language, and Hugo's templating engine, though powerful, is just not language, and Hugo's templating engine, though powerful, is just not
@ -126,7 +110,7 @@ with links as I discover them. Effectively, what I need to do is a "link
transfer": I need to identify regions of code that are highlighted in Agda's HTML, transfer": I need to identify regions of code that are highlighted in Agda's HTML,
find those regions in Hugo's HTML output, and mark them with links. In addition, find those regions in Hugo's HTML output, and mark them with links. In addition,
I'll need to fix up the links themselves: the HTML output assumes that each I'll need to fix up the links themselves: the HTML output assumes that each
Agda file is its own HTML page, but this is ruled out by the second constraint of mine. Agda file is its own HTML page, but this is ruled out by the second constraint.
As a little visualization, the overall problems looks something like this: As a little visualization, the overall problems looks something like this:
@ -151,7 +135,6 @@ What I ended up doing is translating Agda's HTML output into offsets and data
about the code block's _plain text_ -- the source code being decorated. about the code block's _plain text_ -- the source code being decorated.
Both the Agda and Hugo HTML describe the same code; thus, the plain text Both the Agda and Hugo HTML describe the same code; thus, the plain text
is the common denominator between the two. is the common denominator between the two.
{#plain-text}
I wrote a Ruby script to extract the decorations from the Agda output; here I wrote a Ruby script to extract the decorations from the Agda output; here
it is in slightly abridged form. You can find the [original `agda.rb` file here](https://dev.danilafe.com/Web-Projects/blog-static/src/commit/04f12b545d5692a78b1a2f13ef968417c749e295/agda.rb). it is in slightly abridged form. You can find the [original `agda.rb` file here](https://dev.danilafe.com/Web-Projects/blog-static/src/commit/04f12b545d5692a78b1a2f13ef968417c749e295/agda.rb).
@ -164,7 +147,7 @@ def process_agda_html_file(file)
document = Nokogiri::HTML.parse(File.open(file)) document = Nokogiri::HTML.parse(File.open(file))
pre_code = document.css("pre.Agda")[0] pre_code = document.css("pre.Agda")[0]
# The traversal is postorder; we always visit children before their # The traversal postorder; we always visit children before their
# parents, and we visit leaves in sequence. # parents, and we visit leaves in sequence.
line_infos = [] line_infos = []
offset = 0 # Column index within the current Agda source code line offset = 0 # Column index within the current Agda source code line
@ -222,338 +205,52 @@ example above, the script might produce:
] ]
``` ```
#### Modifying Hugo's HTML
Given such line information, the next step is to transfer it onto existing {{< todo >}}This isn't as important probably, but might be worth talking about. {{< /todo >}}
Hugo HTML files. Within a file, I've made my `codelines` shortcode emit
custom attributes that can be used to find syntax-highlighted Agda code. The most challenging step is probably to identify the Agda "projects" that need
The chief such attribute is `data-agda-block`; my script traverses all to be built. Since different articles have different modules (possibly with
elements with this attribute. the same name), I would need to keep them separate. Also, I'm not ruling
out the possibility of one project including another as a submodule. To
make this work, I wrote a little Ruby script to find all Agda files,
guess their project folder, and invoke the Agda compiler there. It boils
down to something like this:
```Ruby ```Ruby
def process_source_file(file, document) # For each Agda file, find the most specific project / subproject to which
# Process each highlight group that's been marked as an Agda file. # it belongs.
document.css('div[data-agda-block]').each do |t| files_for_paths = {}
# ... Dir.glob("**/*.agda", base: root_path) do |agda_file|
``` best_path = max_path.call(agda_file)
files_for_path = files_for_paths.fetch(best_path) do
To figure out which Agda HTML file to use, and which lines to search for links, files_for_paths[best_path] = []
the script also expects some additional attributes.
```Ruby
# ...
first_line, last_line = nil, nil
if first_line_attr = t.attribute("data-first-line")
first_line = first_line_attr.to_s.to_i
end
if last_line_attr = t.attribute("data-last-line")
last_line = last_line_attr.to_s.to_i
end end
if first_line and last_line # Strip the project prefix from the Agda file's path, since
line_range = first_line..last_line # Agda compiler will be invoked in the project's folder.
else files_for_path << agda_file[best_path.length + File::SEPARATOR.length..-1]
# no line number attributes = the code block contains the whole file
line_range = 1..
end
full_path = t.attribute("data-file-path").to_s
# ...
```
At this point, the Agda file could be in some nested directory, like
`A/B/C/File.agda`. However, the project root -- the place where Agda modules
are compiled from -- could be any one of the folders `A`, `B`, or `C`.
Thus, the fully qualified module name for `File.agda` could be `File`,
`C.File`, `B.C.File`, or `A.B.C.File`. Since Agda's HTML output produces
files named after the fully qualified module name, the script needs to guess
what the module file is. This is where some conventions come in play:
I keep my code in folders directly nested within a top-level `code` directory;
thus, I'll have folders `project1` or `project2` inside `code`, and those
will always be project roots. As a result,
I guess that the first directory relative to `code` should be discarded,
while the rest should be included in the path. The only exception to this is
Git submodules: if an Agda file is included using a submodule, the root
directory of the submodule is considered the Agda project root. My Hugo theme
indicates the submodule using an additional `data-base-path` attribute; in all,
that leads to the following logic:
```Ruby
# ...
full_path_dirs = Pathname(full_path).each_filename.to_a
base_path = t.attribute("data-base-path").to_s
base_dir_depth = 0
if base_path.empty?
# No submodules were used. Assume code/<X> is the root.
# The path of the file is given relative to `code`, so need
# to strip only the one outermost directory.
base_dir_depth = 1
base_path = full_path_dirs[0]
else
# The code is in a submodule. Assume that the base path / submodule
# root is the Agda module root, ignore all folders before that.
base_path_dirs = Pathname(base_path).each_filename.to_a
base_dir_depth = base_path_dirs.length
end
# ...
```
With that, the script determines the actual HTML file path ---
by assuming that there's an `html` folder in the same place as the Agda
project root --- and runs the above `process_agda_html_file`:
```Ruby
# ...
dirs_in_base = full_path_dirs[base_dir_depth..-1]
html_file = dirs_in_base.join(".").gsub(/\.agda$/, ".html")
html_path = File.join(["code", base_path, "html", html_file])
agda_info = process_agda_html_file(html_path)
# ...
```
The next step is specific to the output of Hugo's syntax highlighter,
[Chroma](https://github.com/alecthomas/chroma). When line numbers are enabled
-- and they are on my site -- Chroma generates a table that, at some point,
contains a bunch of `span` HTML nodes, each with the `line` class. Each
such `span` corresponds to a single line of output; naturally, the first
one contains the code from `first_line`, the second from `first_line + 1`,
and so on until `last_line`. This is quite convenient, because it saves the
headache of counting newlines the way that the Agda processing code above has to.
For each line of syntax-highlighted code, the script retrieves the corresponding
list of links that were collected from the Agda HTML file.
```Ruby
# ...
lines = t.css("pre.chroma code[data-lang] .line")
lines.zip(line_range).each do |line, line_no|
line_info = agda_info[line_no]
next unless line_info
# ...
```
The subsequent traversal -- which picks out the plain text of the Agda file
as [reasoned above](#plain-text) -- is very similar to the previous
one. Here too there's an `offset` variable, which gets incremented with
the length of a new plain text pieces. Since we know the lines match up
to `span`s, there's no need to count newlines.
```Ruby
# ...
offset = 0
line.traverse do |lt|
if lt.text?
content = lt.content
new_offset = offset + content.length
# ...
```
At this point, we have a line number, and an offset within that line number
that describes the portion of the text under consideration. We can
traverse all the links for the line, and find ones that mark a piece of
text somewhere in this range. For the time being -- since inserting overlapping
spans is quite complicated -- I require the links to lie entirely within a
particular plain text region. As a result, if Chroma splits a single Agda
identifier into several tokens, it will not be linked. For now, this seems
like the most conservative and safe approach.
```Ruby
# ...
matching_links = line_info.links.filter do |link|
link[:from] >= offset and link[:to] <= new_offset
end
# ...
```
All that's left is to slice up the plain text fragment into a bunch of HTML
pieces: the substrings that are links will turn into `a` HTML nodes, while
the substrings that are "in between" the links will be left over as plain
text nodes. The code to do so is relatively verbose, but not all that complicated.
```Ruby
replace_with = []
replace_offset = 0
matching_links.each do |match|
# The link's range is an offset from the beginning of the line,
# but the text piece we're splitting up might be partway into
# the line. Convert the link coordinates to piece-relative ones.
relative_from = match[:from] - offset
relative_to = match[:to] - offset
# If the previous link ended some time before the new link
# began (or if the current link is the first one, and is not
# at the beginning), ensure that the plain text "in between"
# is kept.
replace_with << content[replace_offset...relative_from]
tag = (match.include? :href) ? 'a' : 'span'
new_node = Nokogiri::XML::Node.new(tag, document)
if match.include? :href
# For nodes with links, note what they're referring to, so
# we can adjust their hrefs when we assign global IDs.
href = match[:href].to_s
new_node['href'] = note_used_href file, new_node, href
end
if match.include? :id
# For nodes with IDs visible in the current Hugo file, we'll
# want to redirect links that previously go to other Agda
# module HTML files. So, note the ID that we want to redirect,
# and pick a new unique ID to replace it with.
id = match[:id].to_s
new_node['id'] = note_defined_href file, "#{html_file}##{id}"
end
new_node.content = content[relative_from...relative_to]
replace_with << new_node
replace_offset = relative_to
end
replace_with << content[replace_offset..-1]
```
There's a little bit of a subtlety in the above code: specifically, I use
the `note_used_href` and `note_defined_href` methods. These are important
for rewriting links. Like I mentioned earlier, Agda's HTML output assumes
that each source file should produce a single HTML file -- named after its
qualified module -- and creates links accordingly. However, my blog posts
interweave multiple source files. Some links that would've jumped to a different
file must now point to an internal identifier within the page. Another
important aspect of the transformation is that, since I'm pulling HTML files
from distinct files, it's not guaranteed that each of them will have a unique
`id` attribute. After all, Agda just assigns sequential numbers to each
node that it generates; it would only take, e.g., including the first line
from two distinct modules to end up with two nodes with `id="1"`.
The solution is then twofold:
1. Track all the nodes referencing a particular `href` (made up of an HTML
file and a numerical identifier, like `File.html#123`). When we pick
new IDs -- thus guaranteeing their uniqueness -- we'll visit all the
nodes that refer to the old ID and HTML file, and update their `href`.
2. Track all existing Agda HTML IDs that we're inserting. If we transfer
an `<a id="1234">` onto the Hugo content, we know we'll need to pick a new
ID for it (since `1234` need not be unique), and that we'll need to redirect
the other links to that new ID as the previous bullet describes.
Here's how these two methods work:
```Ruby
def note_defined_href(file, href)
file_hrefs = @local_seen_hrefs.fetch(file) do
@local_seen_hrefs[file] = {}
end
uniq_id = file_hrefs.fetch(href) do
new_id = "agda-unique-ident-#{@id_counter}"
@id_counter += 1
file_hrefs[href] = new_id
end
unless @global_seen_hrefs.include? href
@global_seen_hrefs[href] = { :file => file, :id => uniq_id }
end
return uniq_id
end end
def note_used_href(file, node, href) original_wd = Dir.getwd
ref_list = @nodes_referencing_href.fetch(href) { @nodes_referencing_href[href] = [] } files_for_paths.each do |path, files|
ref_list << { :file => file, :node => node } # There might be a cleaner way of doing this, but it's convenient.
return href Dir.chdir(original_wd)
end Dir.chdir(File.join(root_path, path))
```
Note that they use class variables: these are methods on a `FileGroup` class. # Wherever the target directory is, create a folder that corresponds to
I've omitted the various classes I've declared from the above code for brevity, # the project being built, to avoid "cross-contaminating" the output
but here it makes sense to show them. Like I mentioned earlier, you can # folder with distinct modules with the same name.
view the [complete code here](https://dev.danilafe.com/Web-Projects/blog-static/src/commit/6a168f2fe144850ed3a81b796e07266cbf80f382/agda.rb). html_dir = File.join [target_dir, path, "html"]
FileUtils.mkdir_p html_dir
Interestingly, `note_defined_href` makes use of _two_ global maps: # Just shell out to Agda using the --html folder.
`@local_seen_hrefs` and `@global_seen_hrefs`. This helps satisfy the third files.each do |file|
constraint above, which is linking between code defined in the same series. command = "#{ARGV[0]} --local-interfaces #{file} --html --html-dir=#{html_dir}"
The logic is as follows: when rewriting a link to a new HTML file and ID, puts command
if the code we're trying to link to exists on the current page, we should link puts `#{command}`
to that. Otherwise, if the code we're trying to link to was presented in
a different part of the series, then we should link to that other part.
So, we consult the "local" map for `href`s that will be rewritten to HTML
nodes in the current file, and as a fallback, consult the "global" map for
`hrefs` that were introduced in other parts. The `note_defined_href` populates
both maps, and is "biased" towards the first occurrence of a piece of code:
if posts A and B define a function `f`, and post C only references `f`, then
that link will go to post A's definition, which came earlier.
The other method, `note_used_href`, is simpler. It just appends to a list
of Nokogiri HTML nodes that reference a given `href`. We keep track of the file
in which the reference occurred so we can be sure to consult the right sub-map
of `@local_seen_hrefs` when checking for in-page rewrites.
After running `process_source_file` on all Hugo HTML files within a particular
series, the following holds true:
* We have inserted `span` or `a` nodes wherever Agda's original output
had nodes with `id` or `href` elements. This is with the exception of the
case where Hugo's inline HTML doesn't "line up" with Agda's inline HTML,
which I've only found to happen when the leading character of an identifier is a digit.
* We have picked new IDs for each HTML node we inserted that had an ID,
noting them both globally and for the current file. We noted their original
`href` value (in the form `File.html#123`) and that it should be transformed
into our globally-unique identifiers, in the form `agda-unique-ident-1234`.
* For each HTML node we inserted that links to another, we noted the `href`
of the reference (also in the form `File.html#123`).
Now, all that's left is to redirect the `href`s of the nodes we inserted
from their old values to the new ones. I do this by iterating over `@nodes_referencing_href`,
which contains every link we inserted.
```Ruby
def cross_link_files
@nodes_referencing_href.each do |href, references|
references.each do |reference|
file = reference[:file]
node = reference[:node]
local_targets = @local_seen_hrefs[file]
if local_targets.include? href
# A code block in this file provides this href, create a local link.
node['href'] = "##{local_targets[href]}"
elsif @global_seen_hrefs.include? href
# A code block in this series, but not in this file, defines
# this href. Create a cross-file link.
target = @global_seen_hrefs[href]
other_file = target[:file]
id = target[:id]
relpath = Pathname.new(other_file).dirname.relative_path_from(Pathname.new(file).dirname)
node['href'] = "#{relpath}##{id}"
else
# No definitions in any blog page. For now, just delete the anchor.
node.replace node.content
end
end
end end
end end
``` ```
Notice that for the time being, I simply remove links to Agda definitions that In short, it traverses all the folders in my `code` directory -- which is where
didn't occur in the Hugo post. Ideally, this would link to the plain, non-blog I keep my code, looking for Agda source files. Once it finds them,
documentation page generated by Agda; however, this requires either hosting
those documentation pages, or expecting the Agda standard library HTML pages
to remain stable and hosted at a fixed URL. Neither was simple enough to do,
so I opted for the conservative "just don't insert links" approach.
And that's all of the approach that I wanted to show off today! There
are other details, like finding posts in the same series (I achieve this
with a `meta` element) and invoking `agda --html` on the necessary source files
(my [`build-agda-html.rb`](https://dev.danilafe.com/Web-Projects/blog-static/src/branch/master/build-agda-html.rb)
script is how I personally do this), but I don't think it's all that valuable
to describe them here.
Unfortunately, the additional metadata I had my theme insert makes it
harder for others to use this approach out of the box. However, I hope that by
sharing my experience, others who write Agda and post about it might be able
to get a similar solution working. And of course, it's always fun to write
about a recent project or endeavor.
Happy (dependently typed) programming and blogging!

@ -1 +1 @@
Subproject commit 97e965feece9fd3d3a5328156c043bd2a0cca80f Subproject commit 5bfbaf397b26e20b9f2e8f7e1246674e2c2417b3