YOCaml

The full API can be found here

YOCaml is a free and open-source content management system (CMS) written in OCaml. In other words, YOCaml is a static blog generator written in OCaml. And yes, another one!

The objective of the project is mainly to learn how to use OCaml (and to discover its ecosystem). It is therefore likely that some parts of the code are not idiomatic and please do not hesitate to tell me or to contribute. In addition, it was an opportunity to experiment with the ergonomics of the Preface library and to provide it with slightly less academic examples.

When thinking about how to compute file dependencies, I had initially settled on the idea of using a comonad transformation (TracedT) but then I remembered the paper Generalising Monads to Arrows, which describes the construction of static and dynamic parsers which seemed relevant to capturing dependencies.

On the other hand, I was perfectly aware of the existence of Hakyll, an excellent "static blog generator, generator" (notably used by my friend xvw). But in my understanding of the definition flow of a generator (at user level, I have never observed the source code), the document construction routine was monadic. days ago, msp pointed out to me that Hakyll, prior to version 4, used dependency capture logic incredibly similar to that of YOCaml, funny! Hakyll decided to use a monadic construction to simplify the DSL. Maybe I'll come to the same conclusions when I have to maintain a blog with complicated construction rules, I'll totally replace my API and in that case, I'll probably take inspiration from the work done on Hakyll. But for the moment I'm quite happy with it.

Full Documentation

The full API can be found here

Runtimes

A runtime describes the set of "low-level" primitives to operate in a specific context. This separation allows to have a pure and platform agnostic kernel (the Yocaml module) and to define specific runtimes as needed. Currently, there is only a UNIX runtime.

• Yocaml_unix An UNIX Runtime for YOCaml
• Yocaml_git Uses Git as a compilation target

Plugins

Plugins are wrappers on top of popular libraries from the OCaml community in order to keep the core (the Yocaml module) as small as possible and with the least amount of dependencies.

• Yocaml_markdown Render Markdown String to HTML String using omd
• Yocaml_yaml Allows document metadata to be treated as being defined in Yaml using ocaml-yaml. This module can be passed directly to the read_file_with_metadata function as a provider
• Yocaml_mustache Allows to use Mustache as templating language using ocaml-mustache, This module can be passed directly to the apply_as_template function
• Yocaml_jingoo Allows to use Jingoo as templating language using jingoo, This module can be passed directly to the apply_as_template function

Alternatives

As my main motivation is to discover OCaml while having a tool to build my personal page, it is likely that YOCaml is absolutely not usable for anyone but me, so here are some alternatives.

• Stog is a static web site compiler that uses a custom XML dialect to store metadata together with pages, instead of front matter, and supports OCaml plugin loading via dynlink

• Stone is a static website generator: it takes a template, a css stylesheet, the content itself written in a high-level formatting syntax, and generates the corresponding html pages

• Sesame is a library of tools for building smaller, greener, less resource intensive and more accessible website and blogs inspired by Low Tech Magazine

• Soupault is a static website generator and postprocessor that is based on HTML element tree rewriting and provides a DOM-like API to Lua plugins.

If for some obscure reason you would like to be included in this list... drop me a line

Credits

YOCaml makes use of several libraries from the OCaml ecosystem, you can find an exhaustive list in the Opam file at the root of the project. For an exhaustive list of contributors, I invite you to visit the Github page of the project.

Tools

I haven't written OCaml for a very long time and the very clear progress of the ecosystem is very impressive!

• OCaml (of course), I guess that the project was mainly developped using OCaml 4.12
• Dune, OPAM (and Gnu Make as build-system
• odoc as a documetation parser and generator

Libraries

Even though the libraries are part of the tooling, I was very pleased to quickly discover a collection of well documented libraries with a pleasant user experience. Each of these libraries also has dependencies which I invite you to consult (or apply ocamldep) to get a full understanding of what made this project possible.

• Preface as a complement to the standard library and as an effects manager and abstraction provider. As this project was started to test the usability of the library, a very large part of the code is based on this library.
• Alcotest is a very funny name for a very nice unit test library
• omd Markdown is a fairly common format for writing on the internet. Fortunately, OCaml has an excellent txt -> markdown conversion library
• ocaml-yaml for describing metadata as Yaml document
• ocaml-mustache by default, I use Mustache for templating

Tutorial

YOCaml is slightly different from many tools that statically build web pages. Instead of imposing a template to follow, YOCaml is a library and it is up to the user to compose their generator. This approach does, unfortunately, make the rapid bootstrapping of a blog a little more complicated but it does allow the user more freedom in how they want to organise and generate their page collection.

In this little tutorial, I'll show you several ways to build pages with YOCaml, in peace and quiet. But the tutorial assumes that you use (and understand) OPAM and Dune. So I won't dwell on how to install YOCaml (using a pin) and sometimes I'll use Preface.

This tutorial is very prescriptive and essentially uses the default behaviours of YOCaml. However, keep in mind that while the library makes arbitrary decisions to facilitate bootstrapping a project, you can build your own build rules based on the libraries of your choice.

As YOCaml doesn't offer an integrated development server (which is a shame by the way), I got into the habit of launching a Python server with python3 -m http.server --directory _build/ in the directory where I build a site.

As a simple template engine

Source code of the example

When designing static sites, it is sometimes common to only want a list of pages that respect the same template. Writing all the content in HTML and copying/pasting the templates into each document works fine, but when you want to modify the template, you have to do it... for all the pages... what a hell! As a first tutorial, I suggest you discover how to separate the templates from the content.

Here is the file tree I propose:

./
templates/
pages/
bin/

In templates/ we will place our templates. For the purposes of the example, an header and a footer, and in pages/ we will place our pages. For example index.html for the home page, project.html for a list of projects and about.html to describe the role of the website. bin will be used to host the source code of our site generator. Quite common in short.

Setting up the project

Create a bin/dune and bin/my_site.ml file (if you want to name the binary that will be used to create a site my_site.exe) and define the dune file as such:

(executable
 (name my_site)
 (promote (until-clean))
 (libraries yocaml yocaml_unix))

Nothing very clever, we just say we want an executable and that will have YOCaml as a dependency... it makes sense! And we add Yocaml_unix which allows to execute, with the Unix Runtime, a construction plan. (The separation between the runtime and the description of a plan allows the YOCaml library to be entirely pure and not dependent on the Unix module)

Defining some pages and templates

I offer you high quality HTML code for the templates, a header and a footer. The idea is to pipe the header, the page and the footer.

Here is an example of header. As you can see, I'm pretty experimented with HTML.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>My website lol</title>
</head>
<body>
    <h1>My Website</h1>
    <ul>
        <!-- "A powerful menu"  -->
        <li><a href="index.html">Home</a></li>
        <li><a href="projects.html">Projects</a></li>
        <li><a href="about.html">About</a> </li>
    </ul>
    <hr>
    <main>

Let's create a footer with the ambition of our header!

    </main>
    <hr>
    copyright <strong>Myself</strong>
  </body>
</html>

You can now create several pages, for example, index.html, project.html and about.html with arbitrary content.

Defining the generator

Let's go back to our my_site.ml file to create our generator!

open Yocaml

let destination = "_build"

let () =
  print_endline "Hello"

First, let's define where we want to generate our site. I chose the _build directory, so I don't have to modify the .gitignore of the project.

To create a page, the process is quite simple. We will browse all the files in the pages directory and for each file, we will create a file with the same name in our destination directory which will read the header.html template, piping its content with the file we are reading and piping it with the footer.html template.

Most of the functions we will use are in the Yocaml.Build module.

open Yocaml

let destination = "_build"

let task =
  process_files ["pages/"] (with_extension "html") (fun file ->
      let target = basename file |> into destination in
      let open Build in
      create_file target (
        read_file "templates/header.html"
        >>> pipe_content file
        >>> pipe_content "templates/footer.html")
    )

let () =
  print_endline "Hello"

The API tries to be as clear as possible. The process_files function takes a list of directories as an argument and filters the entries with a predicate. Here, the files must end in .html. Then, for each file, we will create an image in our destination, read the header, read the browsed file and pipe it with the header content, read the footer and pipe it with the previous content.

Now you have to run the program described above. Nothing could be easier, we can use Yocaml_unix.execute. (It is possible to provide its own execution function, for that I refer you to the guide on the Preface effect handlers).

open Yocaml

let destination = "_build"

let task =
  process_files ["pages/"] (with_extension "html") (fun file ->
      let target = basename file |> into destination in
      let open Build in
      create_file target (
        read_file "templates/header.html"
        >>> pipe_content file
        >>> pipe_content "templates/footer.html")
    )

let () =
  Yocaml_unix.execute task

That's it! You have your first template engine that you can try out and that replaces the PHP includes!

Adding the generator as a dependency

The functions in the Yocaml.Build module capture their dependencies and compositions, with the >>> operator merging them. In our example, each page to be built will have as dependencies templates/header.html, templates/footer.html and the page in the pages directory being observed. This means that each page will be rebuilt if and only if necessary.

On the other hand, if the generator is ever recompiled, which could have the effect of completely changing our site, we would also like to be able to consider that a file has to be regenerated. Fortunately the Yocaml.Build.watch function allows us to add a file to the dependencies without reading it, so we can modify our task in this way:

open Yocaml

let destination = "_build"
let track_binary_update = Build.watch Sys.argv.(0)

let task =
  process_files [ "pages/" ] (with_extension "html") (fun file ->
      let target = basename file |> into destination in
      let open Build in
      create_file
        target
        (track_binary_update
        >>> read_file "templates/header.html"
        >>> pipe_content file
        >>> pipe_content "templates/footer.html"))
;;

let () = Yocaml_unix.execute task

Now, every time the generator is recompiled, the pages will have to be rebuilt!

Using a proper templating strategy

Source code of the example

At the moment we have cheated by splitting our layout into two files but this is not usually done! We would like to be able to inject the content directly into a file containing the entire layout like this, in templates/layout.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>My website lol</title>
</head>
<body>
    <h1>My Website</h1>
    <ul>
        <!-- "A powerful menu"  -->
        <li><a href="index.html">Home</a></li>
        <li><a href="projects.html">Projects</a></li>
        <li><a href="about.html">About</a> </li>
    </ul>
    <hr>
    <main>
        {{{body}}}
    </main>
    <hr>
    copyright <strong>Myself</strong>
</body>
</html>

You can use Mustach via the excellent ocaml-mustache library to describe templates. The library is packaged into yocaml_mustache. The idea is to attempt to read a file and its metadata and inject it into a template that is ready for the metadata. I invite you to read the Mustach documentation to understand all that can be described.

First you have to update your dune file for handling Mustache:

(executable
 (name my_site)
 (promote (until-clean))
 (libraries yocaml yocaml_mustache yocaml_unix))

Applying a template

Now we need to modify our generator so that it reads a file and injects it into our template. The Yocaml.Metadata module offers a structured set of metadata. For the purposes of this tutorial, we will use Yocaml.Metadata.Page which does not impose much. Indeed, it offers two optional fields: Title and Description.

The modification in the generator to be made is that the file and its potential metadata must be read using the Yocaml.Build.read_file_with_metadata function and then applied to the template using the Yocaml.Build.apply_as_template function. Both functions take a module that describes how to parse/inject metadata. And read_file_with_metadata takes takes a first module which describes how the metadata are written (in Yaml, in S-Expression, in TOML for example). For our example we will use Yaml because YOCaml comes with a plugin Yocaml_yaml that allows you to easily process Yaml based on ocaml-yaml. First, let's update our dune file in order to add Yocaml_yaml in the dependencies list:

(executable
 (name my_site)
 (promote (until-clean))
 (libraries yocaml yocaml_mustache yocaml_yaml yocaml_unix))

Here we use Yocaml.Metadata.Page. As you can see, Yocaml_yaml.Yocaml_yaml.read_file_with_metadata (module Metadata.Page) file is strictly equivalent to Yocaml.Build.read_file_with_metadata (module Yocaml_yaml) (module Metadata.Page) file. And we use also Yocaml_mustache.apply_as_template (module Metadata.page) file which is also strictly equivalent to Yocaml.Build.apply_as_template (module Metadata.Page) (module Yocaml_mustache) file.

let task =
  process_files [ "pages/" ] (with_extension "html") (fun file ->
      let target = basename file |> into destination in
      let open Build in
      create_file
        target
        (track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Page) file
        >>> Yocaml_mustache.apply_as_template (module Metadata.Page) "templates/layout.html"
        >>^ Stdlib.snd))
;;

Yocaml_yaml.read_file_with_metadata and Yocaml.Build.Yocaml_mustache.apply_as_template return a pair with an option for the metadata and the file content. Fortunately, the application of a template takes optional metadata as an argument but the function will return the metadata unchanged and the contents of the template application. So in the end, it is only necessary to keep the processed content, hence the use of >>^ Stdlib.snd which allows a normal function to be applied as an arrow.

Now we should have exactly the same site as before except that our layout is better defined!

Using Metadata

At the moment we do not use the optional metadata at all. Which is a shame! Let's see how to inject data into the pages to enrich the meaning of our pages! By default, metadata is expressed in Yaml via the ocaml-yaml library and uses a format similar to Jekyll. Let's add metadata to our pages. For example for pages/about.html:

---
title: The famous about page
description: This page TALKS ABOUT ME!
---
You are on the about page.

And let's modify our template to display this metadata if it exists... or not:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <title>My website lol</title>
</head>
<body>
    <h1>My Website</h1>
    <ul>
        <!-- "A powerful menu"  -->
        <li><a href="index.html">Home</a></li>
        <li><a href="projects.html">Projects</a></li>
        <li><a href="about.html">About</a> </li>
    </ul>
    <hr>
    {{#title}}<h2>{{.}}</h2>{{/title}}
    {{#description}}<p>{{.}}</p>{{/description}}
    <main>
        {{{body}}}
    </main>
    <hr>
    copyright <strong>Myself</strong>
</body>
</html>

The template modification uses the "conditional" syntax to display the title and description only if the metadata is present. And yes, remember, the title and description are optional!

Mixing Markdown and Html pages

Source code of the example

Writing HTML by hand can be tiring, and one often wishes one could write a document in a slightly less verbose format like Markdown or Org!

The modification of the generator is quite simple because there is a library Yocaml_markdown (which relies on omd) that offers two arrows for rendering Markdown into HTML:

• Yocaml_markdown.to_html which is an arrow of type (string, string) Build.t, in other words, it takes a string and turns it into a parsed string
• Yocaml_markdown.content_to_html which is a function of type unit -> ('a' * string, 'a * string) Build.t, as it is very common to read the content of a file and its metadata represented as a metadata * file_content pair, the function returns an arrow that acts on the second element of the pair. (Yocaml_markdown.content_to_html () is equivalent to Build.snd Yocaml_markdown.to_html).

So we have to patch our dune file in order to take advantage of Yocaml_markdown:

(executable
 (name my_site)
 (promote (until-clean))
 (libraries yocaml yocaml_mustache yocaml_yaml yocaml_markdown yocaml_unix))

So rather than only browsing the files that have the extension, we will browse the files that have the extension md and html then, once we have read the file and its metadata, if the file has the extension md we will apply the arrow Yocaml_markdown.to_html on the second member of the pair (the content and not the metadata), so we can use Yocaml_markdown.content_to_html () otherwise we do nothing... that is to say the application of the identity function:

let may_process_markdown file =
  let open Build in
  if with_extension "md" file then
     Yocaml_markdown.content_to_html ()
  else arrow Fun.id
;;

And our generator becomes:

let task =
  process_files
    [ "pages/" ]
    (fun f -> with_extension "html" f || with_extension "md" f)
    (fun file ->
      let fname = basename file |> into destination in
      let target = replace_extension fname "html" in
      let open Build in
      create_file
        target
        (track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Page) file
        >>> may_process_markdown file
        >>> Yocaml_mustache.apply_as_template (module Metadata.Page) "templates/layout.html"
        >>^ Stdlib.snd))
;;

That's it! Our generator is able to process HTML files naturally without modifying the output of the reading, and to apply a transformation (from Markdown to HTML) if the file has the extension md! Great, we'll soon be able to describe a real static blog generator, with articles and all.

A first real blog

Source code of the example

After having familiarized ourselves with page generation, we have enough knowledge to build a real blog! However, there is still a difficulty. How to build the index of articles? We will try to answer this question in this guide!

The file tree is identical to the previous ones except that this time we add a directory articles which will contain our articles, a directory css for our stylesheets and a directory images for our images.

./
templates/
articles/
pages/
bin/
images/
css/

The page generator will not change because its behaviour does not change:

open Yocaml

let destination = "_build"
let track_binary_update = Build.watch Sys.argv.(0)

let may_process_markdown file =
  let open Build in
  if with_extension "md" file then
    Yocaml_markdown.content_to_html ()
  else arrow Fun.id
;;

let pages =
  process_files
    [ "pages/" ]
    (fun f -> with_extension "html" f || with_extension "md" f)
    (fun file ->
      let fname = basename file |> into destination in
      let target = replace_extension fname "html" in
      let open Build in
      create_file
        target
        (track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Page) file
        >>> may_process_markdown file
        >>> Yocaml_mustache.apply_as_template (module Metadata.Page) "templates/layout.html"
        >>^ Stdlib.snd))
;;

let () = Yocaml_unix.execute pages

Processing static files

In addition to pages and articles, it is quite common to have static files, for example images or css style sheets. We are going to create two rules to move these images and stylesheets into the appropriate directories.

We can use Yocaml.Build.copy_file which is an arrow that simply copies a file somewhere. The rule is a hell of a lot easier to write than for pages, you just copy and paste a css file into the target.

let css_destination = into destination "css"

let css =
  process_files [ "css/" ] (with_extension "css") (fun file ->
      Build.copy_file file ~into:css_destination)
;;

The same can be done for images, assuming for the purposes of the tutorial that only a limited number of formats are supported: svg, png and gif (yes, I love gifs).

let images_destination = into destination "images"

let images =
  process_files
    [ "images" ]
    (fun f ->
      with_extension "svg" f
      || with_extension "png" f
      || with_extension "gif" f)
    (fun file -> Build.copy_file file ~into:images_destination)
;;

Note that it is possible to simplify the predicates by using Predicate, from Preface:

let images =
  let open Preface.Predicate in
  process_files
    [ "images" ]
    (with_extension "svg" || with_extension "png" || with_extension "gif")
    (fun file -> Build.copy_file file ~into:images_destination)
;;

Now we have to compose our different rules to execute them sequentially. As the execution of an Arrow produces a value of type 'a Effect.t we can use the sequential composition >>:

let () = Yocaml_unix.execute (pages >> css >> images)

Processing articles

The rule for building articles is not fundamentally different from the one for building pages, except that we will add a new template for describing an article. As for pages, we will use a metadata already described: Yocaml.Metadata.Article.

<a href="/index.html">Back to index</a>

<article>
    <h2>{{article_title}}</h2>
    {{{body}}}
</article>

And we can write a first article with this metadata:

---
date: 2021-05-22
article_title: This is an example
article_description: This is the description of the example
---

There is more metadata available for articles but these are the 3 mandatory data. So let's not complicate this already too long tutorial and focus on the essentials.

As mentioned, the rule for articles is quite similar to that for pages:

let article_destination file =
  let fname = basename file |> into "articles" in
  replace_extension fname "html"
;;

let articles =
  process_files [ "articles/" ] (with_extension "md") (fun file ->
      let open Build in
      let target = article_destination file |> into destination in
      create_file
        target
        (track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Article) file
        >>> Yocaml_markdown.content_to_html ()
        >>> Yocaml_mustache.apply_as_template
              (module Metadata.Article)
              "templates/article.html"
        >>> Yocaml_mustache.apply_as_template
              (module Metadata.Article)
              "templates/layout.html"
        >>^ Stdlib.snd))
;;

The main difference is that we only deal with Markdown files (but I could have re-used may_process_markdown) and that we apply two templates, the first being the article template which we apply to the general template.

And as before, the rule is added to the general task.

let () = Yocaml_unix.execute (pages >> css >> images >> articles)

Indexing articles on the front page

Here is the tricky part! Currently, the procedure for building an article index (or archive page) is a bit complex. Mainly to keep it generic. However, if I can find a clearer API that can act as a wrapper, I'll be sure to improve it. Also, if you have any suggestions, I'd love to hear them!

The idea is to read all the files involved, a bit like process_files but to accumulate all the dependencies. Fortunately, it is possible to use the Yocaml.Build.collection function to reduce a list of values wrapped in an effect.

The function takes three arguments: a list wrapped in an effect, an arrow that will act on each element of the list (to calculate dependencies dynamically) and a transformation of this list to produce a value. Here, we will build an Articles metadata based on the list of articles and then inject it into our templates. Once this new arrow is built, we can freely use it in a pipeline, as seen previously!

So before generating our index, we will build an arrow to collect the list of items while tracking each of the items in the dependency list!

let index =
  let open Build in
  let* articles =
    collection
      (read_child_files "articles/" (with_extension "md"))
      (fun source ->
        track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Article) source
        >>^ fun (x, _) -> x, article_destination source)
      (fun x (meta, content) ->
        x
        |> Metadata.Articles.make
             ?title:(Metadata.Page.title meta)
             ?description:(Metadata.Page.description meta)
        |> Metadata.Articles.sort_articles_by_date
        |> fun x -> x, content)
  in

As you can see, we use Yocaml.Effect.read_child_files to read the articles and we use an arrow to extract only their metadata. Then we transform this metadata into a new metadata that manages all the articles. And after that, we can simply describe an arrow that builds our index and adds the index building rule to the general task!

let index =
  let open Build in
  let* articles =
    collection
      (read_child_files "articles/" (with_extension "md"))
      (fun source ->
        track_binary_update
        >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Article) source
        >>^ fun (x, _) -> x, article_destination source)
      (fun x (meta, content) ->
        x
        |> Metadata.Articles.make
             ?title:(Metadata.Page.title meta)
             ?description:(Metadata.Page.description meta)
        |> Metadata.Articles.sort_articles_by_date
        |> fun x -> x, content)
  in
  create_file
    (into destination "index.html")
    (track_binary_update
    >>> Yocaml_yaml.read_file_with_metadata (module Metadata.Page) "index.md"
    >>> Yocaml_markdown.content_to_html ()
    >>> articles
    >>> Yocaml_mustache.apply_as_template (module Metadata.Articles) "templates/list.html"
    >>> Yocaml_mustache.apply_as_template (module Metadata.Articles) "templates/layout.html"
    >>^ Stdlib.snd)
;;

let () = Yocaml_unix.execute (pages >> css >> images >> articles >> index)

The list.html template is fairly plainly written and simply lists the published articles.

{{{body}}}

<h3>Blog</h3>

<ol reversed class="list-articles">
{{#articles}}
<li>
  <span class="date">{{#date}}{{canonical}}{{/date}}</span>
  <a href="{{url}}">{{article_title}}</a><br />
  <p>{{article_description}}</p>
</li>
{{/articles}}
</ol>

And there you have it, all the ingredients to build a real static blog!

Conclusion

Although many of the trivial cases are quite simple, once dynamic dependencies are introduced, the system can become a little more complicated. However, I think that once the logic behind the collection function is understood, many of the more complex scenarios become unlocked! Please feel free to give me feedback.