ZIO HTTP Explained: The REST of the Owl

Request => Response

In zhttp, Requests are processed into Responses via implementations of a sealed trait Http[-R, +E, -A, +B], which itself extends (A => ZIO[R, Option[E], B]). From the latter, we can quickly infer that R and E are the resource and Error channels of a ZIO effect, and we’re going to be converting an A to a B effectually.

type HttpApp[-R, +E] = Http[R, E, Request, Response]
type UHttpApp        = HttpApp[Any, Nothing]
type RHttpApp[-R]    = HttpApp[R, Throwable]
type UHttp[-A, +B]   = Http[Any, Nothing, A, B]

Let’s take a moment to dig into our first endpoint:

  val app: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "owls" => Response.text("Hoot!")
  }

We will use A and B here, knowing that above A = Request and B = Response. Http.collect[A] is a PartialCollect[A] - which behaves like a PartialFunction, meaning we’re going to pattern match on something relating to A and return a B.

We’re matching against a Request, so let’s look closer at the case statement above. The tricky syntax is the infixed -> operator, so let’s first look to the immediate right of it: !! / "owls". This is a Path, and !! denotes the root of the path (i.e. ”/” - not to be confused with the PathSyntax operator /). On the left of -> is Method.GET - a Method. What -> does, is tuple2 together the things on the left/right of it. The definition is

@inline def -> [B](y: B): (A, B) = (self, y)

In our case

case Method.GET -> !! / "owls" => Response.text("Hoot!")

and

case (Method.GET, !! / "owls") => Response.text("Hoot!")

should behave identically. So what’s going on, is we are looking at a Request value, and matching on it’s Method and Path - if they match, we will return our Response.

It will be important for later, but we can reference the request req in our response, for example, via something like

case req @ Method.GET -> !! / "owls" => Response.text("Hoot!")

As a quick aside, Http.collect also internally lifts these to Options, so it can handle the case of None when nothing may match.

A Method models an HTTP request method, i.e. GET, POST, DELETE, etc…

A Path models an HTTP request path. Let’s take a moment to briefly outline !!, /, and /:.

As mentioned above, !! represents the a path root, i.e. a ”/”.

/ is a path delimiter that starts extraction of the left-hand side (a left associative operator).

/: is a path delimiter that starts extraction of the right-hand side (a right associative operator), and can match paths partially. For example, if we look at the code below:

case Method.GET -> "" /: "owls" /: name  => Response.text(s"$name says: Hoot!")

and if we took to curl:

➜ the-rest-of-the-owl (main) ✗ curl http://localhost:9000/owls
Hoot!%
➜ the-rest-of-the-owl (main) ✗ curl http://localhost:9000/owls/Hooty
Hooty says: Hoot!%
➜ the-rest-of-the-owl (main) ✗ curl http://localhost:9000/owls/Hooty/The/Owl
Hooty/The/Owl says: Hoot!%

We can see in the second, and third case, we’re partially matching the remaining path, and it can capture more than just one segment representing a name!

Composing many Http[-R, +E, -A, +B]

In our example, we also have:

  val zApp: Http[Any, Nothing, Request, Response] =
    Http.collectZIO[Request] { case Method.POST -> !! / "owls" =>
      Random.nextIntBetween(3, 6).map(n => Response.text("Hoot! " * n))
    }

Http.collectZIO[Request] behaves just like Http.collect[Request], except here instead of returning a Response, we’ll return a ZIO[R, E, Response]. Being ZIO users, it would make sense to see this form heavily in an app that relies on our resourceful logic. In the example above, this endpoint will use the built-in zio.Random (which no longer needs to be declared in the R channel, as we’re using ZIO 2), and Hoot at us 3 to 5 times randomly, per request.

We then combine app and zApp to pass to the server:

  val combined: Http[Any, Nothing, Request, Response] = app ++ zApp

There are four operators to compose these “HTTP applications”: ++, <>, >>> and <<<, and the behavior of each is as described from the official documentation.

++ is an alias for defaultWith. While using ++, if the first HTTP application returns None the second HTTP application will be evaluated, ignoring the result from the first. If the first HTTP application is failing with a Some[E] the second HTTP application won’t be evaluated.

<> is an alias for orElse. While using <>, if the first HTTP application fails with Some[E], the second HTTP application will be evaluated, ignoring the result from the first. If the first HTTP application returns None, the second HTTP application won’t be evaluated.

>>> is an alias for andThen. It runs the first HTTP application and pipes the output into the other.

<<< is the alias for compose. Compose is similar to andThen. It runs the second HTTP application and pipes the output to the first HTTP application.

Middleware

Broadly, the definition of middleware is context dependent; in our realm of our discussion, if we’re turning a Request into a Response, then it’s anything we do in the middle of that process. It may be something behind the scenes, like adding logging through the process, or more center-stage like modifying the request to add headers to the request. Middleware is great at helping de-couple/re-use business logic.

Specifically, in the context of zhttp, a Middleware is a transformation function that converts one Http to another.

type Middleware[R, E, AIn, BIn, AOut, BOut] = Http[R, E, AIn, BIn] => Http[R, E, AOut, BOut]

We attach middleware to our Http via the @@ operator. For example, we could update our logic to use a built-in debug Middleware like so:

  val wrapped: Http[Console with Clock, IOException, Request, Response] =
    combined @@ Middleware.debug

and then, when running our application, we would see some debug messaging printed when a client interacts with our server:

[info] Starting server on http://localhost:9000
[info] 200 GET /owls 9ms

In the next section, we will build our own Logging Middleware, and then look at a few of the other built in ones.

Logging

Our custom middleware is going to log some information about the Request received, and the Response about to be sent back. We’ll set up a new object Verbose and define a method log that returns a new Middleware, in which we will define the trait’s apply method.

We will be able to do this with the .mapZIO and .contramapZIO functionality of the http argument the Middleware takes in the apply method.

For an http of type Http[R, E, A, B], the A is the type of input, and B is the type of the output. In our example, A = Request and B = Response. If we want to do something with the input, like printing all of the request headers, we can use .contramapZIO[R1, E1, Request]:

http
    .contramapZIO[R1, E1, Request] { r =>
      for {
        _ <- ZIO.foreach(r.headers.toList) { h =>
                Console.printLine(s"> ${h._1}: ${h._2}")
              }
      } yield r
    }

and if we want to do something with the output, like printing all of the response headers, we can use .mapZIO[R1, E1, Response]:

http
    .mapZIO[R1, E1, Response] { r =>
      for {
        _ <- Console.printLine(s"< ${r.status}")
        _ <- ZIO.foreach(r.headers.toList) { h =>
                Console.printLine(s"< ${h._1}: ${h._2}")
              }
      } yield r
    }

The complete example might look something like

package com.alterationx10.troto.middleware

import zhttp.http._
import zio._

object Verbose {

  def log[R, E >: Throwable]
      : Middleware[R, E, Request, Response, Request, Response] =
    new Middleware[R, E, Request, Response, Request, Response] {

      override def apply[R1 <: R, E1 >: E](
          http: Http[R1, E1, Request, Response]
      ): Http[R1, E1, Request, Response] =
        http
          .contramapZIO[R1, E1, Request] { r =>
            for {
              _ <- Console.printLine(s"> ${r.method} ${r.path} ${r.version}")
              _ <- ZIO.foreach(r.headers.toList) { h =>
                     Console.printLine(s"> ${h._1}: ${h._2}")
                   }
            } yield r
          }
          .mapZIO[R1, E1, Response] { r =>
            for {
              _ <- Console.printLine(s"< ${r.status}")
              _ <- ZIO.foreach(r.headers.toList) { h =>
                     Console.printLine(s"< ${h._1}: ${h._2}")
                   }
            } yield r
          }

    }

}

We’re not modifying the input/output here, so the types remain the same and their values un-altered. We used .contramapZIO to accesses the Request, print some information about it, and then return the un-altered value. We then did the same thing with mapZIO for the Response.

This is a very simple example, but is very illustrative of how you can easily update the Request/Response values if desired, or even fail-fast if a particular header is missing, or unverified.

We attach our custom Middleware just as before:

val wrapped: Http[Any,Throwable,Request,Response] =
    combined @@ Verbose.log

and if we run our server, and make a request via curl, in our console we should see something like:

[info] Starting server on http://localhost:9001
[info] > POST /owls Http_1_1
[info] > Host: localhost:9001
[info] > User-Agent: curl/7.79.1
[info] > Accept: */*
[info] < Ok
[info] < content-type: text/plain

CORS

CORS stands for Cross-Origin Resource Sharing. It is a mechanism to allow a website to allow traffic from only certain origins. For example, if we had https:///my-site.com without any CORS config, then someone at https://their-site could load our images, content, etc (and we’d get billed for the data usage). By applying a CORS configuration, we can make sure our resources are loaded when the Origin header is set to my-site.com. Requests trying to render from the other page would then get rejected!

In addition to this, CORS can help with verifying you can interact with the server at all! Wouldn’t it be nice to know you can’t upload a 50mb png to a site before you sent the request? You can send a CORS (OPTIONS) preflight request that says “Hey, I’m from this origin, and I would like to POST you a file with some MIME type that’s this big at this endpoint. We good?” And if your pre-flight request succeeds, you know you can make the actual request - but if it fails, you don’t have to waste the time/traffic finding it out.

Many browsers today will automatically try to make pre-flight queries when content is being loaded from a domain different that the host being accessed, and if there is no CORS policy returned from the server, then the resources won’t be loaded. This even means between sub.my-domain.com and sub2.my-domain.com.

To use the built-in CORS, you need to instantiate a CorsConfig such as:

  val config: CorsConfig =
    CorsConfig(
      anyOrigin = false,
      anyMethod = false,
      allowedOrigins = s => s.equals("localhost"),
      allowedMethods = Some(Set(Method.GET, Method.POST))
    )

and provide it as an argument via @@ Middleware.cors(config).

If we look at the output of an example using the above config, we can see that if we send an Origin header that doesn’t match our config’s allowedOrigins, no CORS headers are returned - thus a browser would block the request. This would be the same as if we sent no Origin header.

➜ the-rest-of-the-owl (main) ✗ curl -v --header "Origin: somewhere"  http://localhost:9001/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> GET /owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain
< set-cookie: x-csrf-token=e9250e77-49a8-4b75-bac0-27307980afba
< content-length: 5
<
* Connection #0 to host localhost left intact
Hoot!%

If we make the request again, with a valid Origin header, we can see the CORS access-control- headers are returned.

➜ the-rest-of-the-owl (main) ✗ curl -v --header "Origin: localhost"  http://localhost:9001/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> GET /owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
> Origin: localhost
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain
< set-cookie: x-csrf-token=d4f7ddc2-63a7-4b78-b99f-1f9b6d6a2d0e
< access-control-expose-headers: *
< access-control-allow-origin: localhost
< access-control-allow-methods: GET,POST
< access-control-allow-credentials: true
< content-length: 5
<
* Connection #0 to host localhost left intact
Hoot!%

CSRF

CSRF stands for Cross-Site Request Forgery. At a broad level, this is when some nefarious code tries to trick you into performing an action with your already logged in credentials. For example, let’s say you were logged into a popular online store, and a browser plugin went rouge. Opening the plugin takes you to a link that actually triggers an email change for your account with the store via a hidden form - which your browser happily send along your sessions information, giving a chance for an attacker to take over your account. CSRF tries to help stop/lessen that attack vector.

CSRF involves the generation of a fairly secure/unique string (we’ll call it a token), and submitting it back to our trusted site when we send information, as an extra level of verification that it was intentionally sent by the user.

A popular way to do this is called Double Submit Cookie. This means that a secure http-only (i.e. browser javascript cannot access this!) cookie is set with the value of the token, and any routes you want protected will need to both send this cookie, as well as the token as a parameter. You could imagine that a trusted web page that is rendered with form submissions already have this value on a hidden input pre-filled and sets a cookie. When the form is submitted, the cookie will go along with the values, and the back-end server can verify that they are preset and match! From here, you can also take it a step further and encrypt the cookie, so the back-end can verify that it can decrypt the token as well to ensure authenticity.

zhttp includes Middleware.csrfGenerate() and Middleware.csrfValidate() as built-in options. For our example, we’ll split these and add the csrfGenerate to out Http that has the GET routes, and the csrfValidate to our POST:

  val app: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "owls"          => Response.text("Hoot!")
    case Method.GET -> "owls" /: name /: !! =>
      Response.text(s"$name says: Hoot!")
  } @@ Middleware.csrfGenerate()

  val zApp: Http[Any, Nothing, Request, Response] =
    Http.collectZIO[Request] { case Method.POST -> !! / "owls" =>
      Random.nextIntBetween(3, 6).map(n => Response.text("Hoot! " * n))
    } @@ Middleware.csrfValidate()

Let’s interact and inspect with these endpoint via curl:

➜ ~ curl -X GET -v http://localhost:9001/owls
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> GET /owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain
< set-cookie: x-csrf-token=2075bc8b-c64b-494c-8249-c3a87ca72fcd
< content-length: 5
<
* Connection #0 to host localhost left intact
Hoot!%

We made a GET request, and we can see that the server told us to set a cookie with our x-csrf-token.

If we try to access our POST without the token, we will get a 403 Forbidden!

➜ ~ curl -X POST -v http://localhost:9001/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> POST /owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 403 Forbidden
< content-length: 0
<
* Connection #0 to host localhost left intact

This middleware does use the Double Submit Cookie method, so if we make a POST including our token as a cookie, and a corresponding header, then we can obtain access to our endpoint.

➜ ~ curl -X POST -v --cookie "x-csrf-token=2075bc8b-c64b-494c-8249-c3a87ca72fcd" -H "x-csrf-token: 2075bc8b-c64b-494c-8249-c3a87ca72fcd" http://localhost:9001/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> POST /owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
> Cookie: x-csrf-token=2075bc8b-c64b-494c-8249-c3a87ca72fcd
> x-csrf-token: 2075bc8b-c64b-494c-8249-c3a87ca72fcd
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain
< content-length: 18
<
* Connection #0 to host localhost left intact
Hoot! Hoot! Hoot! %

Basic Auth

Basic auth is used for hiding a site behind a simple user/password check. The credentials are base64 encoded, but not encrypted, so it shouldn’t be used except over https.

We’ll add a super secret route to our app using the built-in Middleware.basicAuth.

  val authApp: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "secret" / "owls" =>
      Response.text("The password is 'Hoot!'")
  } @@ Middleware.basicAuth("hooty", "tootie")

In our example, we use

final def basicAuth(u: String, p: String): HttpMiddleware[Any, Nothing]

which takes two parameters, a username and a password. When a request comes in, and the credentials are present on the HTTP header, it will compare those values to the username and password for validation. If you wanted to pass in your own logic for this, you could use

case class Credentials(uname: String, upassword: String)
final def basicAuth(f: Credentials => Boolean): HttpMiddleware[Any, Nothing]
final def basicAuthZIO[R, E](f: Credentials => ZIO[R, E, Boolean]): HttpMiddleware[R, E]

We will not cover it in this article, however there are other helpful pre-made MiddleWare, for bearer tokens and custom authorization:

final def bearerAuth(f: String => Boolean): HttpMiddleware[Any, Nothing]
final def bearerAuthZIO[R, E](f: String => ZIO[R, E, Boolean]): HttpMiddleware[R, E]
  final def customAuth(
    verify: Headers => Boolean,
    responseHeaders: Headers = Headers.empty,
    responseStatus: Status = Status.Unauthorized,
  ): HttpMiddleware[Any, Nothing]
  final def customAuthZIO[R, E](
    verify: Headers => ZIO[R, E, Boolean],
    responseHeaders: Headers = Headers.empty,
    responseStatus: Status = Status.Unauthorized,
  ): HttpMiddleware[R, E]

Looking back at out basic auth example, if we try to access our endpoint without credentials, we’ll get a 401 Unauthorized, and a polite www-authenticate header indicating that we may be able to access it via Basic auth.

➜ ~ curl -v http://localhost:9001/secret/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
> GET /secret/owls HTTP/1.1
> Host: localhost:9001
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic
< content-length: 0
<
* Connection #0 to host localhost left intact

We’ll use the power of curls --user flag to do the encoding for us this time:

➜ ~ curl -v --user hooty:tootie http://localhost:9001/secret/owls
*   Trying 127.0.0.1:9001...
* Connected to localhost (127.0.0.1) port 9001 (#0)
* Server auth using Basic with user 'hooty'
> GET /secret/owls HTTP/1.1
> Host: localhost:9001
> Authorization: Basic aG9vdHk6dG9vdGll
> User-Agent: curl/7.79.1
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: text/plain
< content-length: 23
<
* Connection #0 to host localhost left intact
The password is 'Hoot!'%

We can see that our Authorization: Basic aG9vdHk6dG9vdGll was sent in plain text (base64 encoded), as well as that our request was successful.

Juggling Middleware Priority

At this point, we’ve tacked on a few pieces of Middleware. Combining your routes can be very tricky, so let’s address some issues. Let’s looks at what we have so far:

  val app: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "owls"          => Response.text("Hoot!")
    case Method.GET -> "owls" /: name /: !! =>
      Response.text(s"$name says: Hoot!")
  } @@ Middleware.csrfGenerate()

  val zApp: Http[Any, Nothing, Request, Response] =
    Http.collectZIO[Request] { case Method.POST -> !! / "owls" =>
      Random.nextIntBetween(3, 6).map(n => Response.text("Hoot! " * n))
    } @@ Middleware.csrfValidate()

  val authApp: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "secret" / "owls" =>
      Response.text("The password is 'Hoot!'")
  } @@ Middleware.basicAuth("hooty", "tootie")

  val combined: Http[Any, Nothing, Request, Response] = app ++ authApp ++ zApp

We have to add authApp before zApp, or our basic auth route won’t be available! Why is that? It’s because we aren’t sending in any CSRF validation tokens! Because we’ve added Middleware.csrfValidate() to zApp, that portion happily succeeds in handing us back a 403 Forbidden when we don’t send the CSRF data - thus if our authApp were after it, we’d never reach it.

The same situation would occur if we added authApp to the front - everything afterwards would also require basic auth. This also makes it not possible to apply a second Middleware.basicAuth("hooty2", "tootie2") at an Http passed the first, because we’d always fail the credential validation at the first middleware evaluation of the credentials (it would check for user == hooty and password == tootie). The best we could do with basic auth is allow a set of users/passwords that all have the same level of access to various protected routes, but not fine-grained individual access per route.

Websockets

Websockets are also created using Http, but instead of collecting Requests, we typically collect WebSocketChannelEvent. Communication happens over the channel, so instead of returning Response we will return Unit. At a lower level, there is Channel[A], which allows sending arbitrary messages of type A.There is alsoChannelEvent, which encapsulates the types of messages that can be sent/received.

final case class ChannelEvent[-A, +B](channel: Channel[A], event: ChannelEvent.Event[B])

WebSocketChannelEvent is actually a type alias for ChannelEvent[WebsocketFrame, WebSocketFrame].

In our code example below, in addition to logging on some connections hooks, we mainly return back the data sent from the user, but formatted differently. You could further pattern match on the incoming message to perform different actions based on the incoming payload - our bundled multiple Https with different endpoints to handle different functionality.

  val sarcastically: String => String =
    txt =>
      txt.toList.zipWithIndex.map { case (c, i) =>
        if (i % 2 == 0) c.toUpper else c.toLower
      }.mkString

  val wsLogic: Http[Any, Throwable, WebSocketChannelEvent, Unit] =
    Http.collectZIO[WebSocketChannelEvent] {

      case ChannelEvent(ch, ChannelRead(WebSocketFrame.Text(msg))) =>
        ch.writeAndFlush(WebSocketFrame.text(sarcastically(msg)))

      case ChannelEvent(ch, UserEventTriggered(event)) =>
        event match {
          case HandshakeComplete => ZIO.logInfo("Connection started!")
          case HandshakeTimeout  => ZIO.logInfo("Connection failed!")
        }

      case ChannelEvent(ch, ChannelUnregistered) =>
        ZIO.logInfo("Connection closed!")

    }

  val wsApp: Http[Any, Nothing, Request, Response] = Http.collectZIO[Request] {
    case Method.GET -> !! / "ws" => wsLogic.toSocketApp.toResponse
  }

We will use a handy cli tool called websocat to test our websockets. It will allow us to connect to our server, send messages, and see the responses that come back.

➜ ~ websocat ws://localhost:9002/ws
Hello
HeLlO
Sarcasm is hard to convey on the internet
SaRcAsM Is hArD To cOnVeY On tHe iNtErNeT

Streaming

Streaming responses is handled via ZStream, and works straight forwardly, if you are comfortable with that topic.

In our example below, we take a String sentence, and repeat it 1000 times to bulk it up a bit, being sure to use the HTTP_CHARSET encoding when we create a Chunk for it. At this point, it’s as easy as HttpData.fromStream(ZStream.fromChunk(data))! Anything you can ZStream.from... is fair game to stream, which is fairly powerful.

  val content: String =
    "All work and no Play Framework makes Jack a dull boy\n" * 1000

  val data: Chunk[Byte] = Chunk.fromArray(content.getBytes(HTTP_CHARSET))

  val stream: Http[Any, Nothing, Request, Response] = Http.collect[Request] {
    case Method.GET -> !! / "stream" =>
      Response(
        status = Status.Ok,
        headers = Headers.contentLength(data.length.toLong),
        data = HttpData.fromStream(ZStream.fromChunk(data))
      )
  }