Tomas Petricek

Searching for new ways of thinking in programming & working with data

I believe that the most interesting work is not the one solving hard problems, but the one changing how we think about the world. I follow this belief in my work on data science tools, functional programming and F# teaching, in my programming languages research and I try to understand it through philosophy of science.

The Gamma

I'm working on making data-driven storytelling easier, more open and reproducible at the Alan Turing Institute.

Consulting

I'm author of definitive F# books and open-source libraries. I offer my F# training and consulting services as part of fsharpWorks.

Academic

I published papers about theory of context-aware programming languages, type providers, but also philosophy of science.

Tomas Petricek
  • Tomas Petricek
  • Home
  • F# Trainings
  • Talks and books
  • The Gamma
  • Academic

Creating web sites with Suave How to contribute to F# Snippets

The core of many web sites and web APIs is very simple. Given an HTTP request, produce a HTTP response. In F#, we can represent this as a function with type Request -> Response. To make our server scalable, we should make the function asynchronous to avoid unnecessary blocking of threads. In F#, this can be captured as Request -> Async<Response>. Sounds pretty simple, right? So why are there so many evil frameworks that make simple web programming difficult?

Fortunately, there is a nice F# library called Suave.io that is based exactly on the above idea:

Suave is a simple web development F# library providing a lightweight web server and a set of combinators to manipulate route flow and task composition.

I recently decided to start a new version of the F# Snippets web site and I wanted to keep the implementation functional, simple, cross-platform and easy to contrbute to. I wrote a first prototype of the implementation using Suave and already received a few contributions via pull requests! In this blog post, I'll share a few interesting aspects of the implementation and I'll give you some good pointers where you can learn more about Suave. There is no excuse for not contributing to F# Snippets v2 after reading this blog post!

Getting started with Suave

I recently did a couple of talks about Suave at user groups and conferences and many of them have been recorded. There are also a couple of nice examples online and some good documentation on the official web site. So if you want to learn more about Suave, here are some links for you:

  • Channel 9 Interview. Seth Juarez did an interview with me when I was in Redmond and I did a quick 20 minute demo showing how to Deploy an F# Web Application with Suave to Azure. This is the last part of a mini series, so you might also want to check out Making the Case for using F# if you are new to F#, Domain Modeling in F# and Type Providers in F#.

  • NDC Oslo Talk. Next, I talked about Suave at NDC in Oslo. The talk shows two demo application - a web portal showing weather and news and a simple chat written using agents. The talk End-to-end Functional Web Development has been recorded and you can also get the full source code. and slides. It is also worth noting that I'm using the awesome F# Atom plugin in the talk, together with some custom FAKE build scripts to get a nice live reloading when developing the web sites.

  • Community for F# Talk. Henrik Feldt who is one of the Suave contributors did a nice talk Suave from Scratch on the Community for F# channel. This shows many more Suave features and so it is a great follow-up to the above. Also, Henrik is showing Suave on Mac using Xamarin Studio, so you can see that it truly is cross-platform.

  • Web Site and Dojo. For more information, there is a bunch of examples and documentation on the official Suave web site. This includes various ways of deploying Suave applications too. If you then want to get some hands-on experience, try completing the simple Suave Dojo that I put together!

Introducing F# Snippets v2

As already mentioned, I started using Suave for the new version of the F# Snippets web site. The web site is basically a pastebin for F# code snippets. The nice thing is that it uses F# Formatting for formatting the code snippets and generating tool tips. I never released the source code for the old version, because it was simoply too ugly. The new version fixes this!

  • The source code is on GitHub - to run it locally, you'll need to download sample data as discussed in the README.
  • The prototype runs on Azure - this is automatically deployed from the master branch in the GitHub project and it runs as Azure Website.
  • And here is a list of remaining issues before it can replace the old version - the project is quite simple, so this is a great place where you can contribute!

The previous version of F# Snippets stored all data in an SQL database. When creating the new one, I was wondering what is the best option given the size of the web site. It turns out that the meta-data about all the snippets is small enough to fit in memory (about 1MB in JSON format) and so the new version is a lot simpler.

It keeps the meta-data in memory. The formatted snippets are stored in local file system (when testing things locally) or in Azure blob storage (when running on Azure) - though you can also use Azure storage during development. When the meta-data change, it is also saved to a JSON file in the blob storage (so that it can be reloaded if the application is shut down).

You can find more details in the project architecture section of the project README document.

Interesting Suave snippets

There is a number of things that make Suave really nice to use. As you can have a look at the materials above to learn everything about it, I want to give you just a few examples based on my experience with F# Snippets.

The first nice thing about Suave is that it is a library rather than a framework. This means that you are in control of starting and running the server. This makes it easy to deploy it to Azure, Heroku or anywhere else. In F# Snippets, we have one entry-point in the app.fsx file. This composes the server from individual components.

Composing server from web parts

The following code snippet shows how the server is composed. As you can see, we have functionality for showing the home page, displaying snippets, inserting new snippets, listing snippets and the RSS feed:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
let app = 
  choose 
    [ // When accessing '/' we display the homepage
      path "/" >>= Home.showHome 
      
      // Display snippet (latest, specific version and raw source)
      pathWithId "/%s" (fun id -> Snippet.showSnippet id Latest) 
      pathWithId "/raw/%s" (fun id -> Snippet.showRawSnippet id Latest) 
      pathScan "/%s/%d" 
        (fun (id, r) -> Snippet.showSnippet id (Revision r)) 
      pathScan "/raw/%s/%d"  
        (fun (id, r) -> Snippet.showRawSnippet id (Revision r)) 
      
      // Insert page, with simple REST API to check snippet for errors
      path "/pages/insert" >>= Insert.insertSnippet 
      path "/pages/insert/check" >>= Insert.checkSnippet 
      
      // Listing of snippets by author and by tag
      path "/authors/" >>= Author.showAll 
      pathScan "/authors/%s" Author.showSnippets 
      path "/tags/" >>= Tag.showAll 
      pathScan "/tags/%s" Tag.showSnippets 
      
      // Display RSS feed (allowing number of different path formats)
      ( path "/rss/" <|> path "/rss" <|> 
        path "/pages/Rss" <|> path "/pages/Rss/" ) >>= Rss.getRss 
      
      // Otherwise, try to process the request as a static file
      // (this handles all the CSS and JS files as well as images)
      browseStaticFiles ] 

The choose combinator takes a list of web parts and composes them. A Suave web part is essentially one of those functions from the introduction - web parts can handle requests and produce response. Here, we are building a single web part that goes through the web parts in the list and uses the first one that can handle an incoming request. The path combinator is used to restrict what requests a web part handles - so for example path "/" >>= Home.showHome means that we should display the home page if the request is for the path /. A very nice function is pathScan - it takes an F# format string and builds a web part that recognizes requests to URL with the specified pattern. We can, for example, say pathScan "/raw/%s/%d" to detect URLs such as /raw/cJ/5.

Displaying snippets with DotLiquid

The Suave library does not force you to use any specific templating engine and I actually used Suave for some time with just string concatenation or str.Replace. But if you want to use some templating library, it is really easy to add support for it. To see just how easy, look at my pull request adding support for DotLiquid. We're using DotLiquid in F# snippets, so here is how the code looks.

The code sample below shows how we handle request to display a snippet. We get the snippet ID, get information about it from the meta-data and read the file from storage. If everything succeeds, we create a record FormattedSnippet and pass it to the template loaded from snippet.html:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
type FormattedSnippet =
  { Html : string
    Details : Data.Snippet
    Revision : int }

let showSnippet id r =
  let id' = demangleId id
  let snippetOpt = 
    publicSnippets
    |> Seq.tryFind (fun s -> s.ID = id') 
  match snippetOpt with
  | Some snippetInfo -> 
      match Data.loadSnippet id r with
      | Some snippet ->
          { Html = snippet
            Details =
              Data.snippets 
              |> Seq.find (fun s -> s.ID = demangleId id)
            Revision =
              match r with 
              | Latest -> snippetInfo.Versions - 1 
              | Revision r -> r }
          |> DotLiquid.page<FormattedSnippet> "snippet.html"
      | None -> invalidSnippetId id
  | None -> invalidSnippetId id

You can find the full template on GitHub. The value of the record is exposed as model and we can access its properties in the template. For example, the heading is generated by <h1>{{ model.Details.Title }}</h1>.

Checking F# code during insertion

The new F# Snippets web site reports all the errors in your F# code on the fly when you are inserting the snippet. Go to the insert snippet page, type some invalid F#, wait a second and you should see the compiler errors and warnings!

The implementation of this uses a simple JavaScript with timer and it calls the /insert/check API end-point implemented by the server. This then returns a simple JSON with a list of the errors and warning.

This is another elegant piece of F# code that uses Suave composable web parts and the JSON type provider from F# Data to generate the JSON response. Check out the following snippet:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
open FSharp.Data
type Errors = JsonProvider<"""
  [ {"location":[1,1,10,10], "error":true, "message":"sth"} ]""">

let noCache = 
  setHeader "Cache-Control" "no-cache, no-store, must-revalidate"
  >>= setHeader "Pragma" "no-cache"
  >>= setHeader "Expires" "0"
  >>= setMimeType "application/json"

let checkSnippet ctx = async {
  use sr = new StreamReader(new MemoryStream(ctx.request.rawForm))
  let request = sr.ReadToEnd()
  let doc = 
    Literate.ParseScriptString
      (request, "/temp/Snippet.fsx", formatAgent)
  let json = 
    JsonValue.Array
      [| for SourceError((l1,c1),(l2,c2),kind,msg) in doc.Errors ->
         Errors.Root
           ( [| l1; c1; l2; c2 |], 
             (kind = ErrorKind.Error), msg).JsonValue |]
             
  return! ctx |> (noCache >>= Successful.OK(json.ToString()) ) }

There are a few nice things worth mentioning:

  • The example shows an interesting use of the JSON type provider. We give it a sample JSON (list with one error), but we're not using it to read data but instead to generate response. As you can ee on line 20, we can then use the provided type Errors.Root to easily build a JSON value representing the error or warning.

  • We need to disable all caching in the HTTP response. To do this, we use composition of web parts. We define noCache which sets all the different HTTP headers required for this (lines 6-9) and then we use it when producing the result on line 24.

The compositional nature of Suave means that you can really easily define reusable components and structure your code in the way that works for you. For F# Snippets, I wanted to make the project easy to contribute to, and so there is a fairly large number of small independent files implementing the different components.

Summary

This blog post had two purposes. First, I wanted to share some of the resources that you might find useful if you want to learn about web development with F# using Suave. There are many more information available on the internet, including blog post from Scott Hanselman and a cool series by Claus Sørensen, so my list is just scratching the surface!

My second secret goal was to convince you to contribute to the new F# Snippets project. Writing the prototype was a lot of fun and I think you'd have fun contributing too. There is also a very large number of features that people asked about (commenting, search, clustering, suggesting tags, etc.), so I think anyone will find something interesting. To start with, there are a few high-priority issues that need to be resolved before we can replace the old version.

namespace System
namespace System.IO
namespace Suave
module Web

from Suave
module Http

from Suave
module Files

from Suave.Http
module Applicatives

from Suave.Http
module Writers

from Suave.Http
val app : Suave.Types.WebPart

Full name: Fssnip-suave.app
val choose : options:Suave.Types.WebPart list -> Suave.Types.WebPart

Full name: Suave.Http.choose
val path : s:string -> Suave.Types.WebPart

Full name: Suave.Http.Applicatives.path
val id : x:'T -> 'T

Full name: Microsoft.FSharp.Core.Operators.id
val pathScan : pf:PrintfFormat<'a,'b,'c,'d,'t> -> h:('t -> Suave.Types.WebPart) -> Suave.Types.WebPart

Full name: Suave.Http.Applicatives.pathScan
val id : string
val r : int
type FormattedSnippet =
  {Html: string;
   Details: obj;
   Revision: int;}

Full name: Fssnip-suave.FormattedSnippet
FormattedSnippet.Html: string
Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = System.String

Full name: Microsoft.FSharp.Core.string
FormattedSnippet.Details: obj
namespace Microsoft.FSharp.Data
FormattedSnippet.Revision: int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
val showSnippet : id:'a -> r:'b -> 'c

Full name: Fssnip-suave.showSnippet
val id : 'a
val r : 'b
val id' : obj
val snippetOpt : obj option
module Seq

from Microsoft.FSharp.Collections
val tryFind : predicate:('T -> bool) -> source:seq<'T> -> 'T option

Full name: Microsoft.FSharp.Collections.Seq.tryFind
val s : obj
union case Option.Some: Value: 'T -> Option<'T>
val snippetInfo : obj
val snippet : string
val find : predicate:('T -> bool) -> source:seq<'T> -> 'T

Full name: Microsoft.FSharp.Collections.Seq.find
val Latest : 'b
union case Option.None: Option<'T>
Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
type Errors = JsonProvider<...>

Full name: Fssnip-suave.Errors
type JsonProvider

Full name: FSharp.Data.JsonProvider


<summary>Typed representation of a JSON document</summary>
       <param name='Sample'>Location of a JSON sample file or a string containing a sample JSON document</param>
       <param name='SampleList'>If true, sample should be a list of individual samples for the inference.</param>
       <param name='Culture'>The culture used for parsing numbers and dates.</param>
       <param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution)</param>
val noCache : (Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>)

Full name: Fssnip-suave.noCache
val setHeader : key:string -> value:string -> Suave.Types.WebPart

Full name: Suave.Http.Writers.setHeader
val setMimeType : mimeType:string -> Suave.Types.WebPart

Full name: Suave.Http.Writers.setMimeType
val checkSnippet : ctx:Suave.Types.HttpContext -> Async<Suave.Types.HttpContext option>

Full name: Fssnip-suave.checkSnippet
val ctx : Suave.Types.HttpContext
val async : AsyncBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
val sr : StreamReader
Multiple items
type StreamReader =
  inherit TextReader
  new : stream:Stream -> StreamReader + 9 overloads
  member BaseStream : Stream
  member Close : unit -> unit
  member CurrentEncoding : Encoding
  member DiscardBufferedData : unit -> unit
  member EndOfStream : bool
  member Peek : unit -> int
  member Read : unit -> int + 1 overload
  member ReadLine : unit -> string
  member ReadToEnd : unit -> string
  ...

Full name: System.IO.StreamReader

--------------------
StreamReader(stream: Stream) : unit
StreamReader(path: string) : unit
StreamReader(stream: Stream, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding) : unit
StreamReader(path: string, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(path: string, encoding: System.Text.Encoding) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool) : unit
StreamReader(stream: Stream, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit
StreamReader(path: string, encoding: System.Text.Encoding, detectEncodingFromByteOrderMarks: bool, bufferSize: int) : unit
Multiple items
type MemoryStream =
  inherit Stream
  new : unit -> MemoryStream + 6 overloads
  member CanRead : bool
  member CanSeek : bool
  member CanWrite : bool
  member Capacity : int with get, set
  member Flush : unit -> unit
  member GetBuffer : unit -> byte[]
  member Length : int64
  member Position : int64 with get, set
  member Read : buffer:byte[] * offset:int * count:int -> int
  ...

Full name: System.IO.MemoryStream

--------------------
MemoryStream() : unit
MemoryStream(capacity: int) : unit
MemoryStream(buffer: byte []) : unit
MemoryStream(buffer: byte [], writable: bool) : unit
MemoryStream(buffer: byte [], index: int, count: int) : unit
MemoryStream(buffer: byte [], index: int, count: int, writable: bool) : unit
MemoryStream(buffer: byte [], index: int, count: int, writable: bool, publiclyVisible: bool) : unit
Suave.Types.HttpContext.request: Suave.Types.HttpRequest
Suave.Types.HttpRequest.rawForm: byte []
val request : string
StreamReader.ReadToEnd() : string
val doc : obj
val json : JsonValue
type JsonValue =
  | String of string
  | Number of decimal
  | Float of float
  | Record of properties: (string * JsonValue) []
  | Array of elements: JsonValue []
  | Boolean of bool
  | Null
  member Request : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> HttpResponse
  member RequestAsync : uri:string * ?httpMethod:string * ?headers:seq<string * string> -> Async<HttpResponse>
  override ToString : unit -> string
  member ToString : saveOptions:JsonSaveOptions -> string
  member WriteTo : w:TextWriter * saveOptions:JsonSaveOptions -> unit
  static member AsyncLoad : uri:string * ?cultureInfo:CultureInfo -> Async<JsonValue>
  static member private JsonStringEncodeTo : w:TextWriter -> value:string -> unit
  static member Load : uri:string * ?cultureInfo:CultureInfo -> JsonValue
  static member Load : reader:TextReader * ?cultureInfo:CultureInfo -> JsonValue
  static member Load : stream:Stream * ?cultureInfo:CultureInfo -> JsonValue
  static member Parse : text:string * ?cultureInfo:CultureInfo -> JsonValue
  static member ParseMultiple : text:string * ?cultureInfo:CultureInfo -> seq<JsonValue>
  static member ParseSample : text:string * ?cultureInfo:CultureInfo -> JsonValue

Full name: FSharp.Data.JsonValue
union case JsonValue.Array: elements: JsonValue [] -> JsonValue
module Successful

from Suave.Http
val OK : a:string -> Suave.Types.WebPart

Full name: Suave.Http.Successful.OK
override JsonValue.ToString : unit -> string
member JsonValue.ToString : saveOptions:JsonSaveOptions -> string

Published: Tuesday, 15 September 2015, 11:26 PM
Author: Tomas Petricek
Typos: Send me a pull request!
Tags: f#, web

Contact & about

This site is hosted on GitHub and is generated using F# Formatting and DotLiquid. For more info, see the website source on GitHub.

Please submit issues & corrections on GitHub. Use pull requests for minor corrections only.

  • Twitter: @tomaspetricek
  • GitHub: @tpetricek
  • Email me: tomas@tomasp.net

Blog archives

October 2020 (1),  July 2020 (1),  April 2020 (2),  December 2019 (1),  February 2019 (1),  November 2018 (1),  October 2018 (1),  May 2018 (1),  September 2017 (1),  June 2017 (1),  April 2017 (1),  March 2017 (2),  January 2017 (1),  October 2016 (1),  September 2016 (2),  August 2016 (1),  July 2016 (1),  May 2016 (2),  April 2016 (1),  December 2015 (2),  November 2015 (1),  September 2015 (3),  July 2015 (1),  June 2015 (1),  May 2015 (2),  April 2015 (3),  March 2015 (2),  February 2015 (1),  January 2015 (2),  December 2014 (1),  May 2014 (3),  April 2014 (2),  March 2014 (1),  January 2014 (2),  December 2013 (1),  November 2013 (1),  October 2013 (1),  September 2013 (1),  August 2013 (2),  May 2013 (1),  April 2013 (1),  March 2013 (1),  February 2013 (1),  January 2013 (1),  December 2012 (2),  October 2012 (1),  August 2012 (3),  June 2012 (2),  April 2012 (1),  March 2012 (4),  February 2012 (5),  January 2012 (2),  November 2011 (5),  August 2011 (3),  July 2011 (2),  June 2011 (2),  May 2011 (2),  March 2011 (4),  December 2010 (1),  November 2010 (6),  October 2010 (6),  September 2010 (4),  July 2010 (3),  June 2010 (2),  May 2010 (1),  February 2010 (2),  January 2010 (3),  December 2009 (3),  July 2009 (1),  June 2009 (3),  May 2009 (2),  April 2009 (1),  March 2009 (2),  February 2009 (1),  December 2008 (1),  November 2008 (5),  October 2008 (1),  September 2008 (1),  June 2008 (1),  March 2008 (3),  February 2008 (1),  December 2007 (2),  November 2007 (6),  October 2007 (1),  September 2007 (1),  August 2007 (1),  July 2007 (2),  April 2007 (2),  March 2007 (2),  February 2007 (3),  January 2007 (2),  November 2006 (1),  October 2006 (3),  August 2006 (2),  July 2006 (1),  June 2006 (3),  May 2006 (2),  April 2006 (2),  December 2005 (1),  July 2005 (4),  June 2005 (5),  May 2005 (1),  April 2005 (3),  March 2005 (3),  January 2005 (1),  December 2004 (3),  November 2004 (2), 

License

Unless explicitly mentioned, all articles on this site are licensed under Creative Commons Attribution Share Alike. All source code samples are licensed under the MIT License.

CC License logo