Enhance Your Hugo JSON API Using Custom Output Formats and Netlify Redirects

Enhance Your Hugo JSON API Using Custom Output Formats and Netlify Redirects

Hugo makes it super easy to build simple APIs with its built-in output formats. In my previous article, we built a fully functional JSON API. Today we’re going to extend the capabilities of this API and improve the user experience with better URLs.

At the moment our API can look up specific items, but can’t look at them in relation to each other. Say you want to know what players are in a specific team: with our current API, you can’t do that easily. To solve this, we’ll use Custom Output Formats to create another JSON route to read and use these relationships.

Something else we’ll have to consider is that currently our URLs look very 2005! We’ll look into Custom Media Types to dynamically manage URL rewriting rules as well using Query Parameters in order to end up with something like this: school.api/players?team=sly-turtles.

1 Linking Models With Relationships

The different models of our API are our players and teams. Both can be accessed separately. However, we can retrieve a lot more information if we can see how these different items relate to each other. In our example it makes sense to find out what players are in a specific team. After all, each player has to be part of a team!

1.1 Creating the relationship

We should take a second to understand how to structure our data best. First, it can be assumed that each player will only belong to one team at any given time. Thus, we’ll organize our players by team.

Second, the most obvious way to look up a team would be by their name, i.e. sly-turtles or flying-toasters. However, if the team names look like this 🐯❤️🐈 or cool-tiger-sparkly-cat-hot-fish-quick-whale-wet-snake-boring-baracudas - looking up teams could quickly become cumbersome and error prone. We would like to make it a little simpler.

To do that we’ll set an alias in through a uniqueID parameter like so:

    ---
    title: Sly Turtles
    uniqueID: turtles

To connect our players to their team we simply add the team-alias to their respective Front Matter:

    ---
    title: "Frank J. Robinson"
    team: turtles

1.2 Listing players by team

Now that our players and teams are linked, we want be able to list all players of a given team. We already have an endpoint URL for accessing a team, using Hugo’s Built-In JSON Output Format: school.api/teams/sly-turtles/index.json.

To access the players, we need to create a new endpoint URL at school.api/teams/sly-turtles/players.json.

For this, we need to create a teams/sly-turles/players.json file to be generated by Hugo in addition to the existing teams/sly-turtles/index.json file.

In order to achieve this we’ll have to create a custom output format.

In our config.toml we’ll add these configurations:

[outputFormats.players]
baseName = "players"
mediatype ="application/json"

Let’s have a closer look at what we’re doing here.

    [outputFormats.players]

This adds an Output Format called « players » to the list of our project’s existing output formats.

    mediatype ="application/json"

Our players output format will use the same Media Type as the other previous one: application/json

    baseName = "players"

Lastly, we’ll have to name the file which will be generated by our players output format, since the default is « index » and it is already in use by the endpoint school.api/teams/sly-turtles/index.json. To have our new endpoint look like school.api/teams/sly-turtles/players.json we need to set its baseName to « players ».

Lastly we’ll need to create a template for our output format. In Part 1 we explained how to name our template files for Hugo to pick them up. It requires the following naming convention:

{pageKind}.{outputFormatName}.{extension}

So in layouts/teams/ we’ll create single.players.json and add this code to it. I explain the following lines in detail below:

    {{ define "response" }}
    {{- $players := where (where .Site.RegularPages "Type" "players") "Params.team" .Params.uniqueID -}}
        {
            "team" : "{{ .Title }}",
            "count" : "{{ len $players }}"
            ,"players" : [
            {{ range $index, $e := $players }}
                {{ if $index }}, {{ end }}{{ .Render "item" }}
            {{ end }}
            ]
        }
    {{ end }}

We need to consider our context here: Since we are actually in a single team template, the usual .Data.Pages loop isn’t helpful to us. We will need to query all of the single pages in our site (Hugo calls these RegularPages) and then do the following things:

  1. Filter the results to only include pages with a Type of players.
  2. Filter the resulting players to only include pages where the team matches the uniqueID of the team we’re currently viewing.
  3. Store these results in a variable called $players, because we will need to use this information in a couple of places.

We do all of these things with a one-liner on the second line of our example:

{{- $players := where (where .Site.RegularPages "Type" "players") "Params.team" .Params.uniqueID -}}

On line 5 we’ve got our players, to output their count we use Hugo’s template function len:

"count" : "{{ len $players }}"

Finally, we get to iterate over our players to render good old item.json.json and use $index to guard those trailing commas to ensure we’re outputting valid JSON.

At the moment Hugo only uses the JSON output format from the previous article’s index.json. we need to tell Hugo to use our new players output format as well in our config.toml:

    [outputs]
    page = ["json", "players"]

Check out your new team players endpoint URL at http://localhost:1313/teams/sly-turtles/players.json. It should look something like this:

    {
        "data": {
            "team": "Sly Turtles",
            "count": "3",
            "players": [
                {
                    "name": "Frank J. Robinson",
                    "contact": "+1 (555) 555 5555",
                    "permalink": "http://localhost:1313/players/frank-j-robinson/index.json",
                    "year": "junior"
                },
                {...},
                {...}
            ]
        }
    }

2 Filtering Players by Taxonomy

2.1 Listing Players by Sport

Now it becomes very easy to port our Players Custom Output to, say, one of our sports! Contrary to a regular page like a team, a taxonomy page uses a list template, so we need to create layouts/_default/list.players.json. If we wanted to use this template only for taxonomy pages we could also use layouts/_default/taxonomy.players.json. We can use our previous template code with only one difference. Since we’re in a list context, we can use .Data.Pages instead of .Site.RegularPages. The only drawback is that it will contain pages of all types that have a relation to our taxonomy (sport). So just like in our single.players.json we need to use a where statement to filter our base data to pages of Type players.

Looking at our code we need to change the following line:

{{- $players := where (where .Site.RegularPages "Type" "players") "Params.team" .uniqueID -}} 

Our new line of code includes .Data.Pages and should look like this:

    {{- $players := where .Data.Pages "Type" "players" -}}

We’ll also insert .Title and .Type to identify the taxonomy we are displaying in the following way: Type will output the Taxonomy while Title will output the Term

    "group" : "{{.Type}}::{{ .Title }}"

After our adjustments, our code will look like this:

    {{ define "response" }}
    {{- $players := where .Data.Pages "Type" "players" -}}
        {
            "group" : "{{.Type}}::{{ .Title }}",
            "count" : "{{ len $players }}"
            ,"players" : [
            {{ range $i, $e := $players }}
                {{ if $i }}, {{ end }}{{ .Render "item" }}
            {{ end }}
            ]
        }
    {{ end }}

Then, in config.toml we add the players Custom Output Format to taxonomy pages:

    [outputs]
      page = ["json","players"]
      taxonomy = ["json", "players"]

Now we can access our players by sport with http://localhost:1313/sports/football/players.json and it should show our fresh new output:

    {
        "data": {
            "group": "sport::Football",
            "count": "2",
            "players": [
                {
                    "name": "Jody Garland",
    "contact": "+1 (555) 555 5555",
                    "permalink": "http://localhost:1313/players/jody-garland/index.json",
                },
                {
                    "name": "John Artfield",
                    "contact": "+1 (555) 555 5555",
                    "permalink": "http://localhost:1313/players/john-artfield/index.json",
                }
            ]
        }
    }

2.2 Adding a Second Taxonomy

If we were to create a new taxonomy, say practice to identify the weekdays when each player practices, this taxonomy will instantly inherit the players Custom Output Format already assigned to taxonomies. Let’s try it. In our config.toml:

    [taxonomies]
      sport = "sports"
      practice = "practices"

We also need to add practice days to our players in a list format:

    ---
    title: "Frank J. Robinson"
    [ ... ]
    team: turtles
    practices: ["Monday", "Thursday"]
    ---

That’s it! We can now look up players sorted by the day they have practice. Go to http://localhost:1313/practices/thursday/players.json to see this in action.

3 Beautifying URLs

Our API now serves Endpoints for all these different URLs:

  • school.api/index.json
  • school.api/players/index.json
  • school.api/players/{:slug}/index.json
  • school.api/teams/index.json
  • school.api/team/{:slug}/index.json
  • school.api/{:taxonomy}/{:term}/index.json
  • school.api/{team}/players.json
  • school.api/{taxonomy}/{term}/players.json

This is great! The most important part is done. However, the developer experience of using our API could stand to be improved. For this step, I have identified two problems that I would like to solve:

  1. It would be nice if developers didn’t have to append index.json to all of these requests.
  2. Our API would be more intuitive if looking up players by team or sport used the same route as looking up all players, since the type of data being returned is fundamentally the same. Ideally, we would like for the developer to feel as if they’re applying a filter to the base players route.

We can solve these problems with URL rewrites. Reviewing the two problems above, I propose the following changes to our URLs:

  1. Assume index.json by default, so that school.api/players/PLAYER-NAME/index.json becomes school.api/players/PLAYER-NAME/
  2. Change school.api/TEAM-NAME/players.json to school.api/players?team=TEAM-NAME
  3. Use the same convention as #2 for our taxonomies: change school.api/TAXONOMY/TERM/players.json to school.api/players?TAXONOMY=TERM

In order to create URL rewrites, we need to be able to modify our webserver configuration. The two most popular options for webserver software, Apache and NGINX, each have their own way of exposing their configuration for this purpose. They are of no use to us though, as they require us to manage our own server. We built a static API! We want to use serverless tech and host it on a CDN!

Fortunately, Netlify gives us the serverless power we are looking for and also allows us to rewrite URLs. To handle these rewrites, Netlify looks for a file called _redirects in the root of our website.

For those of you not familiar with Netlify, the Hugo website has a great start up guide on hosting your Hugo site with Netlify.

3.1 Getting rid of index.json

Let’s start by getting rid of this index.json at the end of each of our main endpoint URLs. We create a file called _redirects and put it in our /static directory. Because it is at the root of static, Hugo puts it a the root of our project where Netlify will pick it up. To tell Netlify to forego of the index.json, we’ll add this one simple rule to _redirect:

/* /:splat/index.json 200

We can break this line down into three parts:

  1. The first part is the string to match using simple RegEx. The \\\* matches everything following the root /.
  2. The second part is the URL that should be called. Here :splat is where the matched string will go, just before our index.json
  3. The third part is the response code. 200 means it is not a redirect, but a rewriting.

From now on, Netlify will automatically point any URL to their index.json file. Meaning school.api/players/index.json will be called via school.api/players/, api.com/players/jacky-f-robinson/index.json will be called via school.api/players/jacky-f-robinson/, and so on.

That simple line makes our URLs already so beautiful. But we can do more!

3.2 Adding Query Parameters to List Players by Team

In this part our main objective is to clarify the « players by team » endpoint URL. We’ll use query parameters to create a path that clarifies that we’re looking up players by team: school.api/players/?team=sly-turtles Netlify has a very convenient way to deal with query string parameters. In order for /teams/{team}/players.json to be called via players/?team={team} We add this extra line in our _redirect:

/players team=:id /teams/:id/players.json 200

Again, we have three parts here: what to match, what to fetch, and the response code. Netlify knows to replace :id with the value of the team parameter and where to place it when fetching data for this URL. Now in our _redirects file we have:

    /players team=:id  /teams/:id/players.json 200
    /* /:splat/index.json 200

Note that we added the new « team » rule at the top. The redirect rules need to start with the most specific rules at the top, and get more general as we go down the file. This is because Netlify will use the first matching rule that it finds. It is important that the general « splat » rule remains at the bottom so it does not override the more specific « team » rule.

3.3 Adding Query Parameters to List Players by Taxonomy

The last part of this chapter is concerned with fixing the URLs for our players lists that are filtered by our taxonomies sports and practices. Inspired by the step above we could just add those two lines to _redirects

    /players sport=:id  /sports/:id/players.json 200
    /players practice=:id  /practices/:id/players.json 200

But this would mean modifying by hand the _redirects file each time a new taxonomy is added to the project. We cannot expect the content manager or the development team to modify _redirects every time we need an extra layer of information on a player. We need to create an output format which will generate our _redirects file at the root of our website and add one redirect rule for each taxonomy. Up to this point, we’ve used a Built-In Output Format for our index.json, and a Custom Output Format for our players.json. However, they both use a common Built-In Media Type: JSON.

_redirects is not a common Media Type, it doesn’t even have an extension. Hugo’s Built-In Media Types therefore can’t help us here: we need to create a Custom Media Type in config.toml

    [mediaTypes."text/netlify"]
    suffix = ""
    delimiter = ""

The first line adds a Media Type called text/netlify. You can call this whatever you want, but I like to match the media type convention, hence the / in the middle. suffix is the extension. Here we don’t have any extension, so it’s blank. delimiter is whatever joins the extension to the file name. Since there is no extension, we don’t need a delimiter, so it’ll stay blank. Now that we have created our Media Type, we can add it to our Custom Output Formats:

    [outputFormats.redirect]
    mediatype = "text/netlify"
    baseName = "_redirects"
    isPlainText = true

You can call the output format whatever you’d like, but for our purposes redirect makes the most sense. We assign it the mediatype we just created. The baseName is the name of the file to output, we need it to be _redirects. And just like JSON or CSS, our file will be plain text.

Because we only need one _redirects at the root of our website, we’ll assign it to the home page in config.toml:

    [outputs]
      page = ["json","players"]
      taxonomy = ["json", "players"]
      home = ["json", "redirect"] # create it along side the homepage.

Just like before, we now have to create a template for our file and add it to the layouts/_default/ folder. Following the name convention discussed before, we will name it index.redirect. Here, index is what Hugo will look for when searching for any template for the homepage while redirect is the name of our Custom Output Format, that’s what Hugo will look for when needing a template for this output. We don’t have any file extension, so we can stop here. From this template, we’ll start by adding the line from our previous static file, remember the splat rule will go at the bottom of our _redirect:

    /players team=:id  /teams/:id/players.json 200 

    # Always at the bottom
    /* /:splat/index.json 200

Then, all we have to do in between those lines is to iterate over our site’s taxonomies to create the redirect rules. The .Site.Taxonomies map lists all taxonomies of our project with their pluralized name as keys and a list of the corresponding terms as values. Looking at it in a formatted way it would look something like the following:

    - sports :
        - football
        - baseball
        [ .. ]
    - practice
        - monday
        - tuesday
        [ .. ]

So in our template:

    {{- range $key, $value := .Site.Taxonomies }}
    /players {{ $key | singularize }}=:id  /{{ $key }}/:id/players.json 200
    {{- end }}

The $key is the pluralized name of our taxonomy. Because we can only pass one sport or practice day per URL, we will use singularize to make sure our parameter names reflect this limitation:

/players/?sport=football to /sports/football/players.json

and not /players/?sports=football.

Those-attached to the opening of ourrangeandend curly braces is to make sure Hugo does not add too many line breaks.

Now, from your Netlify hosted website, you can access the list of players playing football with

    school.api/players?sport=football

Of course, as previously mentioned, there are many other server technologies out there and all come with their particular URL rewriting logic. This Netlify introduction detailed above was only one of them. But after experimenting with it, it does not take much to add another rewriting file like .htaccess for Apache or Web.config for IIS. All you’d have to do is create the dedicated Custom Output Format and Media Type to have Hugo generate the redirect file your server environment needs.

Implementing the API Alongside a Conventional HTML Website

As seen in Part 1, it does not take any more work to make this API live alongside a conventional HTML static site. Just one thing to remember when dealing with URL Rewriting.

When navigating to a url like school.site/players/frank-j-robinson/, the browser will look for an index.html file present in the directory. This is why with no server configuration whatsoever the URL above will conveniently point to school.site/players/frank-j-robinson/index.html.

Problem is, the current redirect rules of our API would conflict heavily with this behaviour. To make sure our JSON files work nicely with our HTML files, we have to distinguish our page URLs from our API URLs. For this, we’ll prepend every API URL with something like /api/. Every redirect rule including our bottom line universal one should reflect this. Provided your website is more or less conventional like our school’s, this is what you’d have in your layouts/_default/index.redirect:

    # Every api/players/?team={team} rewrites to /teams/{team}/players.json
    /api/players team=:id  /teams/:id/players.json 200

    # Every api/players/?{taxonomy}={taxonomyTerm} redirects to {taxonomy}/{taxonomyTerms}/players.json
    {{- range $key, $value := .Site.Taxonomies }}
    /api/players {{ $key | singularize }}=:id  /{{ $key }}/:id/players.json 200
    {{- end }}

    # Every API endpoints URL redirect to its /index.json
    /api/* /:splat/index.json 200

Conclusion

During the process of building an API with Hugo, we were able to see how, using Output Formats, you can generate different files for each of your pages. Not only can we output basic content of our pages with the Built-In JSON Output Format’s index.json, we can use our own Custom Output Format to create another file like players.json which, sitting next to the page’s index.json, outputs another set of detailed information for the page.

Not satisfied with our ugly URLs, we were able to use our Custom Media Type, to create another Custom Output Format which creates a redirect file in which our rewriting rules are dynamically listed by Hugo’s Custom Output Format template.

Addendum: Creating JSON Output With jsonify

The JSON templates we created in this article were designed around being easy to read and follow along. However, a more reliable way to create JSON output in Hugo would be to build a native data object using Scratch, and formatting the results with jsonify.

Using Scratch, we can create temporary variables to store the data structure we ultimately want to send over the API. We could do that for our example using something like the following:

{{- .Scratch.Set "items" slice -}}
{{- range .Pages -}}
    {{- .Render "transformer" -}}
    {{- $.Scratch.Add "items" (slice (.Scratch.Get "item")) -}}
{{- end -}}
{{ with eq .Kind "section" }}
{{- $.Scratch.SetInMap "data" "section" $.Section -}}
{{ end }}
{{ with eq .Kind "taxonomy" }}
{{- $.Scratch.SetInMap "data" "taxonomy" $.Data.Singular -}}
{{- $.Scratch.SetInMap "data" "term" $.Data.Term -}}
{{ end }}
{{- .Scratch.SetInMap "data" "count" (len .Pages) -}}
{{- .Scratch.SetInMap "data" "items" (.Scratch.Get "items") -}}
{{- .Scratch.SetInMap "output" "data" (.Scratch.Get "data") -}}
{{- .Scratch.Get "output" | jsonify -}}

This is a more idiomatic way to create JSON output. Since JSON is fundamentally a data format and was designed to be easy for machines to process, we should expect jsonify to do a better job of marshaling our JSON than the built-by-hand templates demonstrated earlier in this article. And while it may be a little less obvious to a beginner, using this strategy makes it much easier to change our data structure in the future, and not have to worry about silly things like avoiding trailing commas.

Happy coding!

Join us every Friday 📅

Frontend Friday is a weekly series where we write in-depth posts about modern web development.

Next week: We'll take a look at Cloudinary's image transformations.

Last week: We showed you how to build a static e-commerce site with Snipcart.

Have something to add?

Discuss on Hacker News