TP

Creating web sites with Suave How to contribute to F# Snippets

The core of many web sites and web APIs is very simple. Given an HTTP request, produce a HTTP response. In F#, we can represent this as a function with type Request -> Response. To make our server scalable, we should make the function asynchronous to avoid unnecessary blocking of threads. In F#, this can be captured as Request -> Async<Response>. Sounds pretty simple, right? So why are there so many evil frameworks that make simple web programming difficult?

Fortunately, there is a nice F# library called Suave.io that is based exactly on the above idea:

Suave is a simple web development F# library providing a lightweight web server and a set of combinators to manipulate route flow and task composition.

I recently decided to start a new version of the F# Snippets web site and I wanted to keep the implementation functional, simple, cross-platform and easy to contrbute to. I wrote a first prototype of the implementation using Suave and already received a few contributions via pull requests! In this blog post, I'll share a few interesting aspects of the implementation and I'll give you some good pointers where you can learn more about Suave. There is no excuse for not contributing to F# Snippets v2 after reading this blog post!

Getting started with Suave

I recently did a couple of talks about Suave at user groups and conferences and many of them have been recorded. There are also a couple of nice examples online and some good documentation on the official web site. So if you want to learn more about Suave, here are some links for you:

Introducing F# Snippets v2

As already mentioned, I started using Suave for the new version of the F# Snippets web site. The web site is basically a pastebin for F# code snippets. The nice thing is that it uses F# Formatting for formatting the code snippets and generating tool tips. I never released the source code for the old version, because it was simoply too ugly. The new version fixes this!

The previous version of F# Snippets stored all data in an SQL database. When creating the new one, I was wondering what is the best option given the size of the web site. It turns out that the meta-data about all the snippets is small enough to fit in memory (about 1MB in JSON format) and so the new version is a lot simpler.

It keeps the meta-data in memory. The formatted snippets are stored in local file system (when testing things locally) or in Azure blob storage (when running on Azure) - though you can also use Azure storage during development. When the meta-data change, it is also saved to a JSON file in the blob storage (so that it can be reloaded if the application is shut down).

You can find more details in the project architecture section of the project README document.

Interesting Suave snippets

There is a number of things that make Suave really nice to use. As you can have a look at the materials above to learn everything about it, I want to give you just a few examples based on my experience with F# Snippets.

The first nice thing about Suave is that it is a library rather than a framework. This means that you are in control of starting and running the server. This makes it easy to deploy it to Azure, Heroku or anywhere else. In F# Snippets, we have one entry-point in the app.fsx file. This composes the server from individual components.

Composing server from web parts

The following code snippet shows how the server is composed. As you can see, we have functionality for showing the home page, displaying snippets, inserting new snippets, listing snippets and the RSS feed:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
let app = 
  choose 
    [ // When accessing '/' we display the homepage
      path "/" >>= Home.showHome 
      
      // Display snippet (latest, specific version and raw source)
      pathWithId "/%s" (fun id -> Snippet.showSnippet id Latest) 
      pathWithId "/raw/%s" (fun id -> Snippet.showRawSnippet id Latest) 
      pathScan "/%s/%d" 
        (fun (id, r) -> Snippet.showSnippet id (Revision r)) 
      pathScan "/raw/%s/%d"  
        (fun (id, r) -> Snippet.showRawSnippet id (Revision r)) 
      
      // Insert page, with simple REST API to check snippet for errors
      path "/pages/insert" >>= Insert.insertSnippet 
      path "/pages/insert/check" >>= Insert.checkSnippet 
      
      // Listing of snippets by author and by tag
      path "/authors/" >>= Author.showAll 
      pathScan "/authors/%s" Author.showSnippets 
      path "/tags/" >>= Tag.showAll 
      pathScan "/tags/%s" Tag.showSnippets 
      
      // Display RSS feed (allowing number of different path formats)
      ( path "/rss/" <|> path "/rss" <|> 
        path "/pages/Rss" <|> path "/pages/Rss/" ) >>= Rss.getRss 
      
      // Otherwise, try to process the request as a static file
      // (this handles all the CSS and JS files as well as images)
      browseStaticFiles ] 

The choose combinator takes a list of web parts and composes them. A Suave web part is essentially one of those functions from the introduction - web parts can handle requests and produce response. Here, we are building a single web part that goes through the web parts in the list and uses the first one that can handle an incoming request. The path combinator is used to restrict what requests a web part handles - so for example path "/" >>= Home.showHome means that we should display the home page if the request is for the path /. A very nice function is pathScan - it takes an F# format string and builds a web part that recognizes requests to URL with the specified pattern. We can, for example, say pathScan "/raw/%s/%d" to detect URLs such as /raw/cJ/5.

Displaying snippets with DotLiquid

The Suave library does not force you to use any specific templating engine and I actually used Suave for some time with just string concatenation or str.Replace. But if you want to use some templating library, it is really easy to add support for it. To see just how easy, look at my pull request adding support for DotLiquid. We're using DotLiquid in F# snippets, so here is how the code looks.

The code sample below shows how we handle request to display a snippet. We get the snippet ID, get information about it from the meta-data and read the file from storage. If everything succeeds, we create a record FormattedSnippet and pass it to the template loaded from snippet.html:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
type FormattedSnippet =
  { Html : string
    Details : Data.Snippet
    Revision : int }

let showSnippet id r =
  let id' = demangleId id
  let snippetOpt = 
    publicSnippets
    |> Seq.tryFind (fun s -> s.ID = id') 
  match snippetOpt with
  | Some snippetInfo -> 
      match Data.loadSnippet id r with
      | Some snippet ->
          { Html = snippet
            Details =
              Data.snippets 
              |> Seq.find (fun s -> s.ID = demangleId id)
            Revision =
              match r with 
              | Latest -> snippetInfo.Versions - 1 
              | Revision r -> r }
          |> DotLiquid.page<FormattedSnippet> "snippet.html"
      | None -> invalidSnippetId id
  | None -> invalidSnippetId id

You can find the full template on GitHub. The value of the record is exposed as model and we can access its properties in the template. For example, the heading is generated by <h1>{{ model.Details.Title }}</h1>.

Checking F# code during insertion

The new F# Snippets web site reports all the errors in your F# code on the fly when you are inserting the snippet. Go to the insert snippet page, type some invalid F#, wait a second and you should see the compiler errors and warnings!

The implementation of this uses a simple JavaScript with timer and it calls the /insert/check API end-point implemented by the server. This then returns a simple JSON with a list of the errors and warning.

This is another elegant piece of F# code that uses Suave composable web parts and the JSON type provider from F# Data to generate the JSON response. Check out the following snippet:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
open FSharp.Data
type Errors = JsonProvider<"""
  [ {"location":[1,1,10,10], "error":true, "message":"sth"} ]""">

let noCache = 
  setHeader "Cache-Control" "no-cache, no-store, must-revalidate"
  >>= setHeader "Pragma" "no-cache"
  >>= setHeader "Expires" "0"
  >>= setMimeType "application/json"

let checkSnippet ctx = async {
  use sr = new StreamReader(new MemoryStream(ctx.request.rawForm))
  let request = sr.ReadToEnd()
  let doc = 
    Literate.ParseScriptString
      (request, "/temp/Snippet.fsx", formatAgent)
  let json = 
    JsonValue.Array
      [| for SourceError((l1,c1),(l2,c2),kind,msg) in doc.Errors ->
         Errors.Root
           ( [| l1; c1; l2; c2 |], 
             (kind = ErrorKind.Error), msg).JsonValue |]
             
  return! ctx |> (noCache >>= Successful.OK(json.ToString()) ) }

There are a few nice things worth mentioning:

The compositional nature of Suave means that you can really easily define reusable components and structure your code in the way that works for you. For F# Snippets, I wanted to make the project easy to contribute to, and so there is a fairly large number of small independent files implementing the different components.

Summary

This blog post had two purposes. First, I wanted to share some of the resources that you might find useful if you want to learn about web development with F# using Suave. There are many more information available on the internet, including blog post from Scott Hanselman and a cool series by Claus Sørensen, so my list is just scratching the surface!

My second secret goal was to convince you to contribute to the new F# Snippets project. Writing the prototype was a lot of fun and I think you'd have fun contributing too. There is also a very large number of features that people asked about (commenting, search, clustering, suggesting tags, etc.), so I think anyone will find something interesting. To start with, there are a few high-priority issues that need to be resolved before we can replace the old version.

namespace System
namespace System.IO
namespace Suave
module Web

from Suave
module Http

from Suave
module Files

from Suave.Http
module Applicatives

from Suave.Http
module Writers

from Suave.Http
val app : Suave.Types.WebPart

Full name: Fssnip-suave.app
val choose : options:Suave.Types.WebPart list -> Suave.Types.WebPart

Full name: Suave.Http.choose
val path : s:string -> Suave.Types.WebPart

Full name: Suave.Http.Applicatives.path
val id : x:'T -> 'T

Full name: Microsoft.FSharp.Core.Operators.id
val pathScan : pf:PrintfFormat<'a,'b,'c,'d,'t> -> h:('t -> Suave.Types.WebPart) -> Suave.Types.WebPart

Full name: Suave.Http.Applicatives.pathScan
val id : string
val r : int
type FormattedSnippet =
  {Html: string;
   Details: obj;
   Revision: int;}

Full name: Fssnip-suave.FormattedSnippet
FormattedSnippet.Html: string
Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = System.String

Full name: Microsoft.FSharp.Core.string
FormattedSnippet.Details: obj
namespace Microsoft.FSharp.Data
FormattedSnippet.Revision: int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
val showSnippet : id:'a -> r:'b -> 'c

Full name: Fssnip-suave.showSnippet
val id : 'a
val r : 'b
val id' : obj
val snippetOpt : obj option
module Seq

from Microsoft.FSharp.Collections
val tryFind : predicate:('T -> bool) -> source:seq<'T> -> 'T option

Full name: Microsoft.FSharp.Collections.Seq.tryFind
val s : obj
union case Option.Some: Value: 'T -> Option<'T>
val snippetInfo : obj
val snippet : string
val find : predicate:('T -> bool) -> source:seq<'T> -> 'T

Full name: Microsoft.FSharp.Collections.Seq.find
val Latest : 'b
union case Option.None: Option<'T>
Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
type Errors = JsonProvider<...>

Full name: Fssnip-suave.Errors
type JsonProvider

Full name: FSharp.Data.JsonProvider


<summary>Typed representation of a JSON document</summary>
       <param name='Sample'>Location of a JSON sample file or a string containing a sample JSON document</param>
       <param name='SampleList'>If true, sample should be a list of individual samples for the inference.</param>
       <param name='Culture'>The culture used for parsing numbers and dates.</param>
       <param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution)</param>
val noCache : (Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>)

Full name: Fssnip-suave.noCache
val setHeader : key:string -> value:string -> Suave.Types.WebPart

Full name: Suave.Http.Writers.setHeader
val setMimeType : mimeType:string -> Suave.Types.WebPart

Full name: Suave.Http.Writers.setMimeType
val checkSnippet : ctx:Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>

Full name: Fssnip-suave.checkSnippet
val ctx : Suave.Types.HttpContext
val async : AsyncBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
val sr : StreamReader
Multiple items
type StreamReader =
  inherit TextReader
  new : stream:Stream -> StreamReader + 9 overloads
  member BaseStream : Stream
  member Close : unit -> unit
  member CurrentEncoding : Encoding
  member DiscardBufferedData : unit -> unit
  member EndOfStream : bool
  member Peek : unit -> int
  member Read : unit -> int + 1 overload
  member ReadLine : unit -> string
  member ReadToEnd : unit -> string
  ...

Full name: System.IO.StreamReader

--------------------
StreamReader(stream: Stream) : unit
StreamReader(path: string) : unit
StreamReader(stream: Stream, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding) : unit
StreamReader(path: string, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(path: string, encoding: System.Text.Encoding) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit
StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit
Multiple items
type MemoryStream =
  inherit Stream
  new : unit -> MemoryStream + 6 overloads
  member CanRead : bool
  member CanSeek : bool
  member CanWrite : bool
  member Capacity : int with get, set
  member Flush : unit -> unit
  member GetBuffer : unit -> byte[]
  member Length : int64
  member Position : int64 with get, set
  member Read : buffer:byte[] * offset:int * count:int -> int
  ...

Full name: System.IO.MemoryStream

--------------------
MemoryStream() : unit
MemoryStream(capacity: int) : unit
MemoryStream(buffer: byte []) : unit
MemoryStream(buffer: byte [], writable: bool) : unit
MemoryStream(buffer: byte [], index: int, count: int) : unit
MemoryStream(buffer: byte [], index: int, count: int, writable: bool) : unit
MemoryStream(buffer: byte [], index: int, count: int, writable: bool, publiclyVisible: bool) : unit
Suave.Types.HttpContext.request: Suave.Types.HttpRequest
Suave.Types.HttpRequest.rawForm: byte []
val request : string
StreamReader.ReadToEnd() : string
val doc : obj
val json : JsonValue
type JsonValue =
  | String of string
  | Number of decimal
  | Float of float
  | Record of properties: (string * JsonValue) []
  | Array of elements: JsonValue []
  | Boolean of bool
  | Null
  member Request : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> HttpResponse
  member RequestAsync : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> Async<HttpResponse>
  override ToString : unit -> string
  member ToString : saveOptions:JsonSaveOptions -> string
  member WriteTo : w:TextWriter * saveOptions:JsonSaveOptions -> unit
  static member AsyncLoad : uri:string * ?cultureInfo:CultureInfo -> Async<JsonValue>
  static member private JsonStringEncodeTo : w:TextWriter -> value:string -> unit
  static member Load : uri:string * ?cultureInfo:CultureInfo -> JsonValue
  static member Load : reader:TextReader * ?cultureInfo:CultureInfo -> JsonValue
  static member Load : stream:Stream * ?cultureInfo:CultureInfo -> JsonValue
  static member Parse : text:string * ?cultureInfo:CultureInfo -> JsonValue
  static member ParseMultiple : text:string * ?cultureInfo:CultureInfo -> seq<JsonValue>
  static member ParseSample : text:string * ?cultureInfo:CultureInfo -> JsonValue

Full name: FSharp.Data.JsonValue
union case JsonValue.Array: elements: JsonValue [] -> JsonValue
module Successful

from Suave.Http
val OK : a:string -> Suave.Types.WebPart

Full name: Suave.Http.Successful.OK
override JsonValue.ToString : unit -> string
member JsonValue.ToString : saveOptions:JsonSaveOptions -> string

Published: Wednesday, 16 September 2015, 12:26 AM
Author: Tomas Petricek
Typos: Send me a pull request!
Tags: f#, web