Creating web sites with Suave How to contribute to F# Snippets

The core of many web sites and web APIs is very simple. Given an HTTP request, produce a HTTP response. In F#, we can represent this as a function with type Request -> Response. To make our server scalable, we should make the function asynchronous to avoid unnecessary blocking of threads. In F#, this can be captured as Request -> Async<Response>. Sounds pretty simple, right? So why are there so many evil frameworks that make simple web programming difficult?

Fortunately, there is a nice F# library called Suave.io that is based exactly on the above idea:

Suave is a simple web development F# library providing a lightweight web server and a set of combinators to manipulate route flow and task composition.

I recently decided to start a new version of the F# Snippets web site and I wanted to keep the implementation functional, simple, cross-platform and easy to contrbute to. I wrote a first prototype of the implementation using Suave and already received a few contributions via pull requests! In this blog post, I'll share a few interesting aspects of the implementation and I'll give you some good pointers where you can learn more about Suave. There is no excuse for not contributing to F# Snippets v2 after reading this blog post!

Getting started with Suave

I recently did a couple of talks about Suave at user groups and conferences and many of them have been recorded. There are also a couple of nice examples online and some good documentation on the official web site. So if you want to learn more about Suave, here are some links for you:

• Channel 9 Interview. Seth Juarez did an interview with me when I was in Redmond and I did a quick 20 minute demo showing how to Deploy an F# Web Application with Suave to Azure. This is the last part of a mini series, so you might also want to check out Making the Case for using F# if you are new to F#, Domain Modeling in F# and Type Providers in F#.

• NDC Oslo Talk. Next, I talked about Suave at NDC in Oslo. The talk shows two demo application - a web portal showing weather and news and a simple chat written using agents. The talk End-to-end Functional Web Development has been recorded and you can also get the full source code. and slides. It is also worth noting that I'm using the awesome F# Atom plugin in the talk, together with some custom FAKE build scripts to get a nice live reloading when developing the web sites.

• Community for F# Talk. Henrik Feldt who is one of the Suave contributors did a nice talk Suave from Scratch on the Community for F# channel. This shows many more Suave features and so it is a great follow-up to the above. Also, Henrik is showing Suave on Mac using Xamarin Studio, so you can see that it truly is cross-platform.

• Web Site and Dojo. For more information, there is a bunch of examples and documentation on the official Suave web site. This includes various ways of deploying Suave applications too. If you then want to get some hands-on experience, try completing the simple Suave Dojo that I put together!

Introducing F# Snippets v2

As already mentioned, I started using Suave for the new version of the F# Snippets web site. The web site is basically a pastebin for F# code snippets. The nice thing is that it uses F# Formatting for formatting the code snippets and generating tool tips. I never released the source code for the old version, because it was simoply too ugly. The new version fixes this!

• The source code is on GitHub - to run it locally, you'll need to download sample data as discussed in the README.
• The prototype runs on Azure - this is automatically deployed from the master branch in the GitHub project and it runs as Azure Website.
• And here is a list of remaining issues before it can replace the old version - the project is quite simple, so this is a great place where you can contribute!

The previous version of F# Snippets stored all data in an SQL database. When creating the new one, I was wondering what is the best option given the size of the web site. It turns out that the meta-data about all the snippets is small enough to fit in memory (about 1MB in JSON format) and so the new version is a lot simpler.

It keeps the meta-data in memory. The formatted snippets are stored in local file system (when testing things locally) or in Azure blob storage (when running on Azure) - though you can also use Azure storage during development. When the meta-data change, it is also saved to a JSON file in the blob storage (so that it can be reloaded if the application is shut down).

You can find more details in the project architecture section of the project README document.

Interesting Suave snippets

There is a number of things that make Suave really nice to use. As you can have a look at the materials above to learn everything about it, I want to give you just a few examples based on my experience with F# Snippets.

The first nice thing about Suave is that it is a library rather than a framework. This means that you are in control of starting and running the server. This makes it easy to deploy it to Azure, Heroku or anywhere else. In F# Snippets, we have one entry-point in the app.fsx file. This composes the server from individual components.

Composing server from web parts

The following code snippet shows how the server is composed. As you can see, we have functionality for showing the home page, displaying snippets, inserting new snippets, listing snippets and the RSS feed:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 

let app = 
  choose 
    [ // When accessing '/' we display the homepage
      path "/" >>= Home.showHome 
      
      // Display snippet (latest, specific version and raw source)
      pathWithId "/%s" (fun id -> Snippet.showSnippet id Latest) 
      pathWithId "/raw/%s" (fun id -> Snippet.showRawSnippet id Latest) 
      pathScan "/%s/%d" 
        (fun (id, r) -> Snippet.showSnippet id (Revision r)) 
      pathScan "/raw/%s/%d"  
        (fun (id, r) -> Snippet.showRawSnippet id (Revision r)) 
      
      // Insert page, with simple REST API to check snippet for errors
      path "/pages/insert" >>= Insert.insertSnippet 
      path "/pages/insert/check" >>= Insert.checkSnippet 
      
      // Listing of snippets by author and by tag
      path "/authors/" >>= Author.showAll 
      pathScan "/authors/%s" Author.showSnippets 
      path "/tags/" >>= Tag.showAll 
      pathScan "/tags/%s" Tag.showSnippets 
      
      // Display RSS feed (allowing number of different path formats)
      ( path "/rss/" <|> path "/rss" <|> 
        path "/pages/Rss" <|> path "/pages/Rss/" ) >>= Rss.getRss 
      
      // Otherwise, try to process the request as a static file
      // (this handles all the CSS and JS files as well as images)
      browseStaticFiles ] 

The choose combinator takes a list of web parts and composes them. A Suave web part is essentially one of those functions from the introduction - web parts can handle requests and produce response. Here, we are building a single web part that goes through the web parts in the list and uses the first one that can handle an incoming request. The path combinator is used to restrict what requests a web part handles - so for example path "/" >>= Home.showHome means that we should display the home page if the request is for the path /. A very nice function is pathScan - it takes an F# format string and builds a web part that recognizes requests to URL with the specified pattern. We can, for example, say pathScan "/raw/%s/%d" to detect URLs such as /raw/cJ/5.

Displaying snippets with DotLiquid

The Suave library does not force you to use any specific templating engine and I actually used Suave for some time with just string concatenation or str.Replace. But if you want to use some templating library, it is really easy to add support for it. To see just how easy, look at my pull request adding support for DotLiquid. We're using DotLiquid in F# snippets, so here is how the code looks.

The code sample below shows how we handle request to display a snippet. We get the snippet ID, get information about it from the meta-data and read the file from storage. If everything succeeds, we create a record FormattedSnippet and pass it to the template loaded from snippet.html:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 

type FormattedSnippet =
  { Html : string
    Details : Data.Snippet
    Revision : int }

let showSnippet id r =
  let id' = demangleId id
  let snippetOpt = 
    publicSnippets
    |> Seq.tryFind (fun s -> s.ID = id') 
  match snippetOpt with
  | Some snippetInfo -> 
      match Data.loadSnippet id r with
      | Some snippet ->
          { Html = snippet
            Details =
              Data.snippets 
              |> Seq.find (fun s -> s.ID = demangleId id)
            Revision =
              match r with 
              | Latest -> snippetInfo.Versions - 1 
              | Revision r -> r }
          |> DotLiquid.page<FormattedSnippet> "snippet.html"
      | None -> invalidSnippetId id
  | None -> invalidSnippetId id

You can find the full template on GitHub. The value of the record is exposed as model and we can access its properties in the template. For example, the heading is generated by <h1>{{ model.Details.Title }}</h1>.

Checking F# code during insertion

The new F# Snippets web site reports all the errors in your F# code on the fly when you are inserting the snippet. Go to the insert snippet page, type some invalid F#, wait a second and you should see the compiler errors and warnings!

The implementation of this uses a simple JavaScript with timer and it calls the /insert/check API end-point implemented by the server. This then returns a simple JSON with a list of the errors and warning.

This is another elegant piece of F# code that uses Suave composable web parts and the JSON type provider from F# Data to generate the JSON response. Check out the following snippet:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 

open FSharp.Data
type Errors = JsonProvider<"""
  [ {"location":[1,1,10,10], "error":true, "message":"sth"} ]""">

let noCache = 
  setHeader "Cache-Control" "no-cache, no-store, must-revalidate"
  >>= setHeader "Pragma" "no-cache"
  >>= setHeader "Expires" "0"
  >>= setMimeType "application/json"

let checkSnippet ctx = async {
  use sr = new StreamReader(new MemoryStream(ctx.request.rawForm))
  let request = sr.ReadToEnd()
  let doc = 
    Literate.ParseScriptString
      (request, "/temp/Snippet.fsx", formatAgent)
  let json = 
    JsonValue.Array
      [| for SourceError((l1,c1),(l2,c2),kind,msg) in doc.Errors ->
         Errors.Root
           ( [| l1; c1; l2; c2 |], 
             (kind = ErrorKind.Error), msg).JsonValue |]
             
  return! ctx |> (noCache >>= Successful.OK(json.ToString()) ) }

There are a few nice things worth mentioning:

• The example shows an interesting use of the JSON type provider. We give it a sample JSON (list with one error), but we're not using it to read data but instead to generate response. As you can ee on line 20, we can then use the provided type Errors.Root to easily build a JSON value representing the error or warning.

• We need to disable all caching in the HTTP response. To do this, we use composition of web parts. We define noCache which sets all the different HTTP headers required for this (lines 6-9) and then we use it when producing the result on line 24.

The compositional nature of Suave means that you can really easily define reusable components and structure your code in the way that works for you. For F# Snippets, I wanted to make the project easy to contribute to, and so there is a fairly large number of small independent files implementing the different components.

Summary

This blog post had two purposes. First, I wanted to share some of the resources that you might find useful if you want to learn about web development with F# using Suave. There are many more information available on the internet, including blog post from Scott Hanselman and a cool series by Claus Sørensen, so my list is just scratching the surface!

My second secret goal was to convince you to contribute to the new F# Snippets project. Writing the prototype was a lot of fun and I think you'd have fun contributing too. There is also a very large number of features that people asked about (commenting, search, clustering, suggesting tags, etc.), so I think anyone will find something interesting. To start with, there are a few high-priority issues that need to be resolved before we can replace the old version.