TP

Happy New Year 2016 around the World Behind the scenes of my #FsAdvent project

Just like last year and the year before, I wanted to participate in the #FsAdvent event, where someone writes a blog post about something they did with F# during December. Thanks to Sergey Tihon for the organization of the English version and the Japanese F# community for coming up with the idea a few years ago!

As my blog post ended up on 31 December, I wanted to do something that would fit well with the theme of ending of 2015 and starting of the new year 2016 and so I decided to write a little interactive web site that tracks the "Happy New Year" tweets live across the globe. This is partly inspired by Happy New Year Tweets from Twitter in 2014, but rather than analyzing data in retrospect, you can watch 2016 come live!

Without further ado, here are the important links:

Before we get to the technical details, here is a brief screenshot showing the project live:


Overview

On the front-end side, the web site displays three different things - it shows live tweets on a map, it shows live tweets in a feed (below on the right) and it shows a word cloud with most common phrases. Everything is updated live using a three web socket connections with the server.

On the back-end side, the server uses Twitter Streaming API to receive "Happy New Year" tweets as they happen. It then uses various techniques for getting locations of some tweets so that they can appear on the map and it calculates statistics (e.g. for the word cloud) on the fly.

If you look at the source code, pretty much all back-end is implemented in a single F# script file. For the front-end, I didn't do anything fancy and hacked together some JavaScript using the great D3-based Datamaps library for the map.

There are a couple of nice things in the code including the connection to Twitter, F# type providers (as always), agents for reactive programming and Suave web server for implementing web sockets.

Getting a stream of tweets

To get the tweets, I'm using the F# Data Toolbox library, which comes with a nice Twitter API wrapper built using F# type providers. As a single-user application (all is happening on the server), we can directly provider the application access token & secret and connect to the Twitter directly. Then we can use the twitter.Streaming.FilterTweets method to search for tweets that contain any of the known "Happy New Year" phrases:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
let phrases = 
 ["새해 복 많이 받으세요"; "สวัสดีปีใหม่";
  "šťastný nový rok"; "عام سعيد"; (...) ]

let ctx = (Provide key and secrets)
let twitter = Twitter(UserContext(ctx))

// Search for the phrases and start the stream
let search = twitter.Streaming.FilterTweets phrases 
search.Start()

The search.TweetReceived event will be triggered when a new tweet happens. The status object has a bunch of properties (inferred by a type provider). It turns out that event status.Text is optional and so parsing the tweets involves a lot of pattern matching:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
let liveTweets = 
  search.TweetReceived
  |> Observable.choose (fun status -> 
      // Parse the location, if the tweet has it
      let origLocation = parseLocation status.Geo

      // Get user name, text of the tweet and location
      match status.User, status.Text with
      | Some user, Some text ->
        { Tweeted = DateTime.UtcNow; OriginalArea = user.Location
          Text = text;PictureUrl = user.ProfileImageUrl; 
          (Populate other properties) } |> Some
      | _ -> None )

The code is slightly simplified, but it is pretty representative. Now we have a value liveTweets of type IObservable<Tweet> which is an event that is triggered every time we get a new (not completely silly) tweet.

Geolocating tweets and users

The hardest bit turns out to be getting good tweets for the map. Not a lot of tweets come with GPS coordinates and so I had to do a couple of tricks. When more people start tweeting around the New Year, we should be able to use mostly tweets with GPS coordinates, but there are some backup strategies:

  1. If a tweet has GPS coordinates, use this as the location
  2. Every now and then use MapQuest or Bing to geolocate the user based on their location in the profile
  3. If we didn't produce enough tweets using (1) or (2), locate tweet based on the language of the phrase and put it in some place where a previous tweet with the same phrase appeared.

In priciple, geolocating users based on their profile would work good enough, but all the geolocation services have rate limits that are easy to hit when the site is running live and so I added (3) as the last resort. If I had more time, I would probably try to build an index with country and city names, which would likely cover enough tweets (at least from users with a reasonable text in their "location").

Tweets with GPS coordinates

All of the methods report tweets to a "replay" agent (see below) that replays the tweets with a specified delay. This is done using replay.AddEvent at the end of the pipeline. For tweets with GPS coordinates, we simply copy the already provided data to InferredLocation (coordinates) and InferredArea (text):

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
liveTweets
|> Observable.choose (fun tw ->
    match tw.OriginalLocation with
    | Some loc -> 
       (tw.Tweeted.AddSeconds(5.0),
        { tw with 
           InferredArea = Some tw.OriginalArea 
           InferredLocation = Some loc }) |> Some
    | _ -> None)
|> Observable.add replay.AddEvent

Geolocating tweets using Bing and MapQuest

For locating tweets based on the user's location, we will be calling Bing and MapQuest APIs. This is done using type providers (see below) and wrapped in a nice MapQuest.locate and Bing.locate functions. We also need to limit rate at which we use these - the following geolocates one tweet per 5 seconds using MapQuest:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
liveTweets
|> Observable.limitRate 5000
|> Observable.mapAsyncIgnoreErrors (fun tw -> async {
    let! located = MapQuest.locate tw.OriginalArea
    return located |> Option.map (fun (area, loc) ->
      tw.Tweeted.AddSeconds(10.0),
      { tw with 
         InferredLocation = Some loc; 
         InferredArea = Some area }) })
|> Observble.choose id
|> Observable.add replay.AddEvent

Time zones and geolocating with type providers

As in every F# project, I'm making a heavy use of F# Data type providers when calling REST-based geolocation services. As a bonus, I also needed to find time zones of countries of the world, which can be done by extracting the information from List of time zones by country Wikipedia page using the HTML type provider.

Extracting time zone information

The HTML type provider gives us access to the tables on the Wikipedia page and so we can get the country and time zones just by writing r.Country and r.''Time Zone'' (using backticks to wrap the space). As far as I know, Datamaps does not easily let me display multiple time zones per country and so I just pick the middle time zone:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 
type TimeZones = HtmlProvider<"http://.../List_of_time_zones_by_country">
let reg = Regex("""UTC([\+\-][0-9][0-9]\:[0-9][0-9])?""")

let timeZones = 
 [ for r in TimeZones.GetSample().Tables.Table1.Rows do
    let tz = r.``Time Zone``.Replace("−", "-")
    let matches = reg.Matches(tz)
    if matches.Count > 0 then
      yield r.Country, matches.[matches.Count/2].Value ]

There are a few explicitly defined countries in the actual source code for countries where the middle time zone is very wrong and for countries that are named differently on Wikipedia.

Geolocating using MapQuest

Both Bing and MapQuest provide a nice REST end-point that we can call using the JSON type provider. To compose the sample URL, we need to use the Literal attribute and append a key (which is stored in a separate config file). The JSON type provider infers the type from the response and gives us nice typed access to the results:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
// Use JSON provider to get a type for calling the API
let [<Literal>] MapQuestSample = 
  "http://mapquestapi.com/geocoding/v1/address?location=Prague"
type MapQuest = JsonProvider<MapQuestSample>

let locate (place:string) = 
  let url = 
    "http://www.mapquestapi.com/geocoding/v1/address?key=" +
      Config.MapQuestKey + "&location=" + (HttpUtility.UrlEncode place)
  MapQuest.Load(url).Results
  |> Seq.choose (fun loc ->
      // Pick the first returned location if there were any
      if loc.Locations.Length = 0 then None
      else Some(loc, loc.Locations.[0]) )
  |> Seq.map (fun (info, loc) ->
      // Return the location with lattitude and longitude
      info.ProvidedLocation.Location, 
        (loc.LatLng.Lat, loc.LatLng.Lng) )

As usual, using the JSON type provider for calling REST APIs makes things very easy. The Results property is inferred to be an array of records and information such as loc.LatLng.Lat is also statically typed.

Reactive programming with F# agents

The project does quite a lot of interesting reactive event processing. In F#, you can, of course, use Reactive Extensions (Rx), but I always found Rx a bit hard to use because they lack simple underlying primitives (more about this in my rant on library design). F# comes with a simple set of primitives in the Observable module which covers some 80% of what you need and you can easily implement additional primitives using F# agents.

For example, the following is a simple agent that I wrote to limit the rate of requests. The idea is that the agent will emit an event it receives and then it will ignore all other events for the specified number of milliseconds:

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
/// Limits the rate of emitted messages to at most 
/// one per the specified number of milliseconds
type RateLimitAgent<'T>(timeout) = 
  let event = Event<'T>()
  let agent = MailboxProcessor.Start(fun inbox -> 
    // We remember the last time we emitted a message
    let rec loop (lastMessageTime:DateTime) = async {
      let! e = inbox.Receive()
      let now = DateTime.UtcNow
      // If we waited long enough, report the event
      // otherwise ignore it and wait some more
      let ms = (now - lastMessageTime).TotalMilliseconds
      if ms > timeout then
        event.Trigger(e)
        return! loop now
      else 
        return! loop lastMessageTime }
    loop DateTime.MinValue )

  /// Triggered when an event happens
  member x.EventOccurred = event.Publish
  /// Send an event to the agent
  member x.AddEvent(event) = agent.Post(event)

Agents are the much needed lower level primitive of the Reactive Extensions. You can quite easily express any logic you need using just a state machine encoded as a recursive asynchronous loop. The implementation then wraps the agent in a higher-level primitive Observable.limitRate that was used in the earlier snippet.

Handling websockets with Suave

One more nice thing in the project is the handling of web sockets. The server serves static files from the web sub-directory, but it also communicates with the front-end via three web sockets (for the map, feed and wordcloud). When a client connects, we simply want to start sending updates to it from one of the IObservable<T> events that we defined earlier (e.g. by serializing tweets from liveTweets as JSON).

To do this, I first defined a helper socketOfObservable, which uses Suave's socket { .. } computation builder and repeatedly awaits an update from the specified updates and reports it to via the socket:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
let socketOfObservable 
    updates (webSocket:WebSocket) ctx = socket {
  while true do
    // Wait for the next update from the source
    let! update = updates |> Async.AwaitObservable 
                          |> SocketOp.ofAsync
    // Report it to the front-end over the wire!
    do! webSocket.send Text (UTF8.bytes update) true }

The main server is then composed from a number of web parts - the first three handle the communication via web sockets, the fourth one returns information about time zones that we downloaded from Wikipedia and the last two serve static files:

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
let part =
  choose 
    [ path "/maptweets" >>= handShake (socketOfObservable mapTweets)
      path "/feedtweets" >>= handShake (socketOfObservable feedTweets)
      path "/frequencies" >>= handShake (socketOfObservable phraseUpdates)
      path "/zones" >>= Successful.OK timeZonesJson
      path "/">>= Files.browseFile root "index.html" 
      Files.browse root ]

Summary

The main part of the project in app.fsx is some 350 lines long and I find it pretty amazing how much you can do in this small number of lines. If you're writing a project like this in F#, you get to use a number of nice libraries including F# Data Toolbox for the Twitter API, Suave.io for the web server and F# Data type providers for calling REST APIs. Finally, I deployed the service using Azure VM, but you could also use MBrace which can host web servers in a cluster, or any other hosting - all the libraries I'm using are cross-platform.

If you're reading this around December 31, 2015 then definitely check out the project running live. I didn't plan to turn this into a reusable application, but who knows! :-) If you want to use it for tracking tweets related to some other events you can find the full source on GitHub under the Apache license - and also get in touch if you have some interesting use for this work!

namespace System
namespace System.Web
namespace System.Collections
namespace System.Collections.Generic
Multiple items
namespace FSharp

--------------------
namespace Microsoft.FSharp
Multiple items
namespace FSharp.Data

--------------------
namespace Microsoft.FSharp.Data
namespace FSharp.Data.Toolbox
namespace FSharp.Data.Toolbox.Twitter
module AsyncHelpers
namespace System.Text
namespace System.Text.RegularExpressions
namespace Suave
module Web

from Suave
module Http

from Suave
module Applicatives

from Suave.Http
namespace Suave.Sockets
namespace Suave.Sockets.Control
module AsyncSocket

from Suave.Sockets
module WebSocket

from Suave
namespace Suave.Utils
type Tweet =
  {Tweeted: DateTime;
   Text: string;
   OriginalArea: string;
   UserName: string;
   UserScreenName: string;
   PictureUrl: string;
   OriginalLocation: (decimal * decimal) option;
   Phrase: int;
   IsRetweet: bool;
   GeoLocationSource: string;
   ...}

Full name: Happy-new-year-tweets.Tweet


 Information we collect about tweets. The `Inferred` fields are calculated later
 by geolocating the user, all other information is filled when tweet is received
Tweet.Tweeted: DateTime
Multiple items
type DateTime =
  struct
    new : ticks:int64 -> DateTime + 10 overloads
    member Add : value:TimeSpan -> DateTime
    member AddDays : value:float -> DateTime
    member AddHours : value:float -> DateTime
    member AddMilliseconds : value:float -> DateTime
    member AddMinutes : value:float -> DateTime
    member AddMonths : months:int -> DateTime
    member AddSeconds : value:float -> DateTime
    member AddTicks : value:int64 -> DateTime
    member AddYears : value:int -> DateTime
    ...
  end

Full name: System.DateTime

--------------------
DateTime()
   (+0 other overloads)
DateTime(ticks: int64) : unit
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int, kind: DateTimeKind) : unit
   (+0 other overloads)
Multiple items
Tweet.Text: string

--------------------
namespace System.Text
Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = String

Full name: Microsoft.FSharp.Core.string
Tweet.OriginalArea: string
Tweet.UserName: string
Tweet.UserScreenName: string
Tweet.PictureUrl: string
Tweet.OriginalLocation: (decimal * decimal) option
type 'T option = Option<'T>

Full name: Microsoft.FSharp.Core.option<_>
Multiple items
val decimal : value:'T -> decimal (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.decimal

--------------------
type decimal = Decimal

Full name: Microsoft.FSharp.Core.decimal

--------------------
type decimal<'Measure> = decimal

Full name: Microsoft.FSharp.Core.decimal<_>
Tweet.Phrase: int
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
Tweet.IsRetweet: bool
type bool = Boolean

Full name: Microsoft.FSharp.Core.bool
Tweet.GeoLocationSource: string
Tweet.InferredArea: string option
Tweet.InferredLocation: (decimal * decimal) option
val root : string

Full name: Happy-new-year-tweets.root
namespace System.IO
type Path =
  static val DirectorySeparatorChar : char
  static val AltDirectorySeparatorChar : char
  static val VolumeSeparatorChar : char
  static val InvalidPathChars : char[]
  static val PathSeparator : char
  static member ChangeExtension : path:string * extension:string -> string
  static member Combine : [<ParamArray>] paths:string[] -> string + 3 overloads
  static member GetDirectoryName : path:string -> string
  static member GetExtension : path:string -> string
  static member GetFileName : path:string -> string
  ...

Full name: System.IO.Path
IO.Path.Combine([<ParamArray>] paths: string []) : string
IO.Path.Combine(path1: string, path2: string) : string
IO.Path.Combine(path1: string, path2: string, path3: string) : string
IO.Path.Combine(path1: string, path2: string, path3: string, path4: string) : string
val phrases : string list

Full name: Happy-new-year-tweets.phrases
"manigong bagong taon"; "Срећна Нова година"; "честита нова година"; "selamat tahun baru";
  "С Новым Годом"; "あけまして おめでとう ございます"; "新年快乐"; "Щасливого Нового Року"; "שנה טובה"
  "yeni yılınız kutlu olsun"; "feliz año nuevo"; "happy new year"; "Καλή Χρονιά";"godt nyttår"
  "bon any nou"; "felice anno nuovo"; "sretna nova godina"; "godt nytår"; "gelukkig nieuwjaar"
  "Frohes neues Jahr"; "urte berri on"; "bonne année"; "boldog új évet"; "gott nytt år"
  "szczęśliwego nowego roku"; "blwyddyn newydd dda"; "feliz ano novo"; "sugeng warsa enggal"
val ctx : TwitterUserContext

Full name: Happy-new-year-tweets.ctx
{ ConsumerKey = Config.TwitterKey; ConsumerSecret = Config.TwitterSecret;
     AccessToken = Config.TwitterAccessToken; AccessSecret = Config.TwitterAccessSecret }
val twitter : Twitter

Full name: Happy-new-year-tweets.twitter
Multiple items
type Twitter =
  new : context:TwitterContext -> Twitter
  member RequestRawData : url:string * query:(string * string) list -> string
  member Connections : Connections
  member Search : Search
  member Streaming : Streaming
  member Timelines : Timelines
  member Users : Users
  static member Authenticate : consumer_key:string * consumer_secret:string -> TwitterConnector
  static member AuthenticateAppOnly : consumer_key:string * consumer_secret:string -> Twitter
  static member TwitterWeb : unit -> WebBrowser

Full name: FSharp.Data.Toolbox.Twitter.Twitter

--------------------
new : context:TwitterContext -> Twitter
union case TwitterContext.UserContext: TwitterUserContext -> TwitterContext
val search : TwitterStream<JsonProvider<...>.Root>

Full name: Happy-new-year-tweets.search
property Twitter.Streaming: Streaming
member Streaming.FilterTweets : keywords:seq<string> -> TwitterStream<JsonProvider<...>.Root>
abstract member TwitterStream.Start : unit -> unit
val liveTweets : IObservable<Tweet>

Full name: Happy-new-year-tweets.liveTweets
property TwitterStream.TweetReceived: IEvent<JsonProvider<...>.Root>
Multiple items
module Observable

from AsyncHelpers

--------------------
module Observable

from Microsoft.FSharp.Control
val choose : chooser:('T -> 'U option) -> source:IObservable<'T> -> IObservable<'U>

Full name: Microsoft.FSharp.Control.Observable.choose
val status : JsonProvider<...>.Root
val origLocation : (decimal * decimal) option
property JsonProvider<...>.Root.Geo: Option<JsonProvider<...>.Geo>
property JsonProvider<...>.Root.User: Option<JsonProvider<...>.User>
property JsonProvider<...>.Root.Text: Option<string>
union case Option.Some: Value: 'T -> Option<'T>
val user : JsonProvider<...>.User
val text : string
property DateTime.UtcNow: DateTime
Multiple items
union case Opcode.Text: Opcode

--------------------
namespace System.Text
UserName = user.Name; UserScreenName = user.ScreenName;
          OriginalLocation = origLocation; Phrase = getPhrase text
          InferredArea = None; InferredLocation = None;
          IsRetweet = isRT status; GeoLocationSource = "NA"
union case Option.None: Option<'T>
val tw : Tweet
val loc : decimal * decimal
DateTime.AddSeconds(value: float) : DateTime
val add : callback:('T -> unit) -> source:IObservable<'T> -> unit

Full name: Microsoft.FSharp.Control.Observable.add
union case AlternativeSourceAgentMessage.AddEvent: 'T -> AlternativeSourceAgentMessage<'T>
val limitRate : milliseconds:int -> source:IObservable<'a> -> IObservable<'a>

Full name: AsyncHelpers.Observable.limitRate


 Limits the rate of emitted messages to at most one per the specified number of milliseconds
val mapAsyncIgnoreErrors : f:('a -> Async<'b>) -> source:IObservable<'a> -> IObservable<'b>

Full name: AsyncHelpers.Observable.mapAsyncIgnoreErrors


 Behaves like `Observable.map`, but does not stop when error happens
val async : AsyncBuilder

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.async
val located : (string * (decimal * decimal)) option
Multiple items
module Option

from Suave.Utils

--------------------
module Option

from Microsoft.FSharp.Core
val map : mapping:('T -> 'U) -> option:'T option -> 'U option

Full name: Microsoft.FSharp.Core.Option.map
val area : string
val choose : options:Types.WebPart list -> Types.WebPart

Full name: Suave.Http.choose
val id : x:'T -> 'T

Full name: Microsoft.FSharp.Core.Operators.id
type TimeZones = HtmlProvider<...>

Full name: Happy-new-year-tweets.TimeZones
type HtmlProvider

Full name: FSharp.Data.HtmlProvider


<summary>Typed representation of an HTML file.</summary>
           <param name='Sample'>Location of an HTML sample file or a string containing a sample HTML document.</param>
           <param name='PreferOptionals'>When set to true, inference will prefer to use the option type instead of nullable types, `double.NaN` or `""` for missing values. Defaults to false.</param>
           <param name='IncludeLayoutTables'>Includes tables that are potentially layout tables (with cellpadding=0 and cellspacing=0 attributes)</param>
           <param name='MissingValues'>The set of strings recogized as missing values. Defaults to `NaN,NA,#N/A,:,-,TBA,TBD`.</param>
           <param name='Culture'>The culture used for parsing numbers and dates. Defaults to the invariant culture.</param>
           <param name='Encoding'>The encoding used to read the sample. You can specify either the character set name or the codepage number. Defaults to UTF8 for files, and to ISO-8859-1 the for HTTP requests, unless `charset` is specified in the `Content-Type` response header.</param>
           <param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution).</param>
           <param name='EmbeddedResource'>When specified, the type provider first attempts to load the sample from the specified resource
              (e.g. 'MyCompany.MyAssembly, resource_name.html'). This is useful when exposing types generated by the type provider.</param>
"https://en.wikipedia.org/wiki/List_of_time_zones_by_country"
val reg : Regex

Full name: Happy-new-year-tweets.reg
Multiple items
type Regex =
  new : pattern:string -> Regex + 1 overload
  member GetGroupNames : unit -> string[]
  member GetGroupNumbers : unit -> int[]
  member GroupNameFromNumber : i:int -> string
  member GroupNumberFromName : name:string -> int
  member IsMatch : input:string -> bool + 1 overload
  member Match : input:string -> Match + 2 overloads
  member Matches : input:string -> MatchCollection + 1 overload
  member Options : RegexOptions
  member Replace : input:string * replacement:string -> string + 5 overloads
  ...

Full name: System.Text.RegularExpressions.Regex

--------------------
Regex(pattern: string) : unit
Regex(pattern: string, options: RegexOptions) : unit
val timeZones : (string * string) list

Full name: Happy-new-year-tweets.timeZones
val r : HtmlProvider<...>.Table1.Row
HtmlProvider<...>.GetSample() : HtmlProvider<...>
val tz : string
val matches : MatchCollection
Regex.Matches(input: string) : MatchCollection
Regex.Matches(input: string, startat: int) : MatchCollection
property MatchCollection.Count: int
property HtmlProvider<...>.Table1.Row.Country: string
Multiple items
type LiteralAttribute =
  inherit Attribute
  new : unit -> LiteralAttribute

Full name: Microsoft.FSharp.Core.LiteralAttribute

--------------------
new : unit -> LiteralAttribute
val MapQuestSample : string

Full name: Happy-new-year-tweets.MapQuestSample
"http://www.mapquestapi.com/geocoding/v1/address?location=Prague&key=" + Config.MapQuestKey
type MapQuest = JsonProvider<...>

Full name: Happy-new-year-tweets.MapQuest
type JsonProvider

Full name: FSharp.Data.JsonProvider


<summary>Typed representation of a JSON document.</summary>
       <param name='Sample'>Location of a JSON sample file or a string containing a sample JSON document.</param>
       <param name='SampleIsList'>If true, sample should be a list of individual samples for the inference.</param>
       <param name='RootName'>The name to be used to the root type. Defaults to `Root`.</param>
       <param name='Culture'>The culture used for parsing numbers and dates. Defaults to the invariant culture.</param>
       <param name='Encoding'>The encoding used to read the sample. You can specify either the character set name or the codepage number. Defaults to UTF8 for files, and to ISO-8859-1 the for HTTP requests, unless `charset` is specified in the `Content-Type` response header.</param>
       <param name='ResolutionFolder'>A directory that is used when resolving relative file references (at design time and in hosted execution).</param>
       <param name='EmbeddedResource'>When specified, the type provider first attempts to load the sample from the specified resource
          (e.g. 'MyCompany.MyAssembly, resource_name.json'). This is useful when exposing types generated by the type provider.</param>
       <param name='InferTypesFromValues'>If true, turns on additional type inference from values.
          (e.g. type inference infers string values such as "123" as ints and values constrained to 0 and 1 as booleans.)</param>
val locate : place:string -> seq<string * (decimal * decimal)>

Full name: Happy-new-year-tweets.locate
val place : string
val url : string
module Config
val MapQuestKey : string

Full name: Config.MapQuestKey
Multiple items
type HttpUtility =
  new : unit -> HttpUtility
  static member HtmlAttributeEncode : s:string -> string + 1 overload
  static member HtmlDecode : s:string -> string + 1 overload
  static member HtmlEncode : s:string -> string + 2 overloads
  static member JavaScriptStringEncode : value:string -> string + 1 overload
  static member ParseQueryString : query:string -> NameValueCollection + 1 overload
  static member UrlDecode : str:string -> string + 3 overloads
  static member UrlDecodeToBytes : str:string -> byte[] + 3 overloads
  static member UrlEncode : str:string -> string + 3 overloads
  static member UrlEncodeToBytes : str:string -> byte[] + 3 overloads
  ...

Full name: System.Web.HttpUtility

--------------------
HttpUtility() : unit
HttpUtility.UrlEncode(bytes: byte []) : string
HttpUtility.UrlEncode(str: string) : string
HttpUtility.UrlEncode(str: string, e: Encoding) : string
HttpUtility.UrlEncode(bytes: byte [], offset: int, count: int) : string
JsonProvider<...>.Load(uri: string) : JsonProvider<...>.Root


Loads JSON from the specified uri

JsonProvider<...>.Load(reader: IO.TextReader) : JsonProvider<...>.Root


Loads JSON from the specified reader

JsonProvider<...>.Load(stream: IO.Stream) : JsonProvider<...>.Root


Loads JSON from the specified stream
module Seq

from Microsoft.FSharp.Collections
val choose : chooser:('T -> 'U option) -> source:seq<'T> -> seq<'U>

Full name: Microsoft.FSharp.Collections.Seq.choose
val loc : JsonProvider<...>.Result
property JsonProvider<...>.Result.Locations: JsonProvider<...>.Location []
property Array.Length: int
val map : mapping:('T -> 'U) -> source:seq<'T> -> seq<'U>

Full name: Microsoft.FSharp.Collections.Seq.map
val info : JsonProvider<...>.Result
val loc : JsonProvider<...>.Location
property JsonProvider<...>.Result.ProvidedLocation: JsonProvider<...>.ProvidedLocation
property JsonProvider<...>.ProvidedLocation.Location: string
property JsonProvider<...>.Location.LatLng: JsonProvider<...>.LatLng
property JsonProvider<...>.LatLng.Lat: decimal
property JsonProvider<...>.LatLng.Lng: decimal
Multiple items
type RateLimitAgent<'T> =
  new : timeout:float -> RateLimitAgent<'T>
  member AddEvent : event:'T -> unit
  member EventOccurred : IEvent<'T>

Full name: Happy-new-year-tweets.RateLimitAgent<_>


 Limits the rate of emitted messages to at most
 one per the specified number of milliseconds


--------------------
new : timeout:float -> RateLimitAgent<'T>
val timeout : float
val event : Event<'T>
Multiple items
module Event

from Microsoft.FSharp.Control

--------------------
type Event<'T> =
  new : unit -> Event<'T>
  member Trigger : arg:'T -> unit
  member Publish : IEvent<'T>

Full name: Microsoft.FSharp.Control.Event<_>

--------------------
type Event<'Delegate,'Args (requires delegate and 'Delegate :> Delegate)> =
  new : unit -> Event<'Delegate,'Args>
  member Trigger : sender:obj * args:'Args -> unit
  member Publish : IEvent<'Delegate,'Args>

Full name: Microsoft.FSharp.Control.Event<_,_>

--------------------
new : unit -> Event<'T>

--------------------
new : unit -> Event<'Delegate,'Args>
val agent : MailboxProcessor<'T>
Multiple items
type MailboxProcessor<'Msg> =
  interface IDisposable
  new : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:CancellationToken -> MailboxProcessor<'Msg>
  member Post : message:'Msg -> unit
  member PostAndAsyncReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> Async<'Reply>
  member PostAndReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> 'Reply
  member PostAndTryAsyncReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> Async<'Reply option>
  member Receive : ?timeout:int -> Async<'Msg>
  member Scan : scanner:('Msg -> Async<'T> option) * ?timeout:int -> Async<'T>
  member Start : unit -> unit
  member TryPostAndReply : buildMessage:(AsyncReplyChannel<'Reply> -> 'Msg) * ?timeout:int -> 'Reply option
  ...

Full name: Microsoft.FSharp.Control.MailboxProcessor<_>

--------------------
new : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:Threading.CancellationToken -> MailboxProcessor<'Msg>
static member MailboxProcessor.Start : body:(MailboxProcessor<'Msg> -> Async<unit>) * ?cancellationToken:Threading.CancellationToken -> MailboxProcessor<'Msg>
val inbox : MailboxProcessor<'T>
val loop : (DateTime -> Async<'a>)
val lastMessageTime : DateTime
val e : 'T
member MailboxProcessor.Receive : ?timeout:int -> Async<'Msg>
val now : DateTime
val ms : float
member Event.Trigger : arg:'T -> unit
field DateTime.MinValue
val x : RateLimitAgent<'T>
member RateLimitAgent.EventOccurred : IEvent<'T>

Full name: Happy-new-year-tweets.RateLimitAgent`1.EventOccurred


 Triggered when an event happens
property Event.Publish: IEvent<'T>
member RateLimitAgent.AddEvent : event:'T -> unit

Full name: Happy-new-year-tweets.RateLimitAgent`1.AddEvent


 Send an event to the agent
val event : 'T
member MailboxProcessor.Post : message:'Msg -> unit
val socketOfObservable : updates:IObservable<string> -> webSocket:WebSocket -> ctx:'a -> Async<Choice<unit,Error>>

Full name: Happy-new-year-tweets.socketOfObservable
val updates : IObservable<string>
val webSocket : WebSocket
Multiple items
module WebSocket

from Suave

--------------------
type WebSocket =
  new : connection:Connection -> WebSocket
  member read : unit -> Async<Choice<(Opcode * byte [] * bool),Error>>
  member send : opcode:Opcode -> bs:byte [] -> fin:bool -> Async<Choice<unit,Error>>

Full name: Suave.WebSocket.WebSocket

--------------------
new : connection:Connection -> WebSocket
val ctx : 'a
val socket : SocketMonad

Full name: Suave.Sockets.Control.SocketMonad.socket
val update : string
Multiple items
module Async

from Suave.Utils

--------------------
type Async
static member AsBeginEnd : computation:('Arg -> Async<'T>) -> ('Arg * AsyncCallback * obj -> IAsyncResult) * (IAsyncResult -> 'T) * (IAsyncResult -> unit)
static member AwaitEvent : event:IEvent<'Del,'T> * ?cancelAction:(unit -> unit) -> Async<'T> (requires delegate and 'Del :> Delegate)
static member AwaitIAsyncResult : iar:IAsyncResult * ?millisecondsTimeout:int -> Async<bool>
static member AwaitTask : task:Task -> Async<unit>
static member AwaitTask : task:Task<'T> -> Async<'T>
static member AwaitWaitHandle : waitHandle:WaitHandle * ?millisecondsTimeout:int -> Async<bool>
static member CancelDefaultToken : unit -> unit
static member Catch : computation:Async<'T> -> Async<Choice<'T,exn>>
static member FromBeginEnd : beginAction:(AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg:'Arg1 * beginAction:('Arg1 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg1:'Arg1 * arg2:'Arg2 * beginAction:('Arg1 * 'Arg2 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromBeginEnd : arg1:'Arg1 * arg2:'Arg2 * arg3:'Arg3 * beginAction:('Arg1 * 'Arg2 * 'Arg3 * AsyncCallback * obj -> IAsyncResult) * endAction:(IAsyncResult -> 'T) * ?cancelAction:(unit -> unit) -> Async<'T>
static member FromContinuations : callback:(('T -> unit) * (exn -> unit) * (OperationCanceledException -> unit) -> unit) -> Async<'T>
static member Ignore : computation:Async<'T> -> Async<unit>
static member OnCancel : interruption:(unit -> unit) -> Async<IDisposable>
static member Parallel : computations:seq<Async<'T>> -> Async<'T []>
static member RunSynchronously : computation:Async<'T> * ?timeout:int * ?cancellationToken:CancellationToken -> 'T
static member Sleep : millisecondsDueTime:int -> Async<unit>
static member Start : computation:Async<unit> * ?cancellationToken:CancellationToken -> unit
static member StartAsTask : computation:Async<'T> * ?taskCreationOptions:TaskCreationOptions * ?cancellationToken:CancellationToken -> Task<'T>
static member StartChild : computation:Async<'T> * ?millisecondsTimeout:int -> Async<Async<'T>>
static member StartChildAsTask : computation:Async<'T> * ?taskCreationOptions:TaskCreationOptions -> Async<Task<'T>>
static member StartImmediate : computation:Async<unit> * ?cancellationToken:CancellationToken -> unit
static member StartWithContinuations : computation:Async<'T> * continuation:('T -> unit) * exceptionContinuation:(exn -> unit) * cancellationContinuation:(OperationCanceledException -> unit) * ?cancellationToken:CancellationToken -> unit
static member SwitchToContext : syncContext:SynchronizationContext -> Async<unit>
static member SwitchToNewThread : unit -> Async<unit>
static member SwitchToThreadPool : unit -> Async<unit>
static member TryCancelled : computation:Async<'T> * compensation:(OperationCanceledException -> unit) -> Async<'T>
static member CancellationToken : Async<CancellationToken>
static member DefaultCancellationToken : CancellationToken

Full name: Microsoft.FSharp.Control.Async

--------------------
type Async<'T>

Full name: Microsoft.FSharp.Control.Async<_>
static member Async.AwaitObservable : ev1:IObservable<'T1> -> Async<'T1>


 Creates an asynchronous workflow that will be resumed when the
 specified observables produces a value. The workflow will return
 the value produced by the observable.
Multiple items
module SocketOp

from Suave.Sockets

--------------------
type SocketOp<'a> = Async<Choice<'a,Error>>

Full name: Suave.Sockets.SocketOp<_>
val ofAsync : a:Async<'a> -> SocketOp<'a>

Full name: Suave.Sockets.SocketOp.ofAsync
member WebSocket.send : opcode:Opcode -> bs:byte [] -> fin:bool -> Async<Choice<unit,Error>>
module UTF8

from Suave.Utils
val bytes : s:string -> byte []

Full name: Suave.Utils.UTF8.bytes
val part : Types.WebPart

Full name: Happy-new-year-tweets.part
val path : s:string -> Types.WebPart

Full name: Suave.Http.Applicatives.path
val handShake : continuation:(WebSocket -> Types.HttpContext -> SocketOp<unit>) -> ctx:Types.HttpContext -> Async<Types.HttpContext option>

Full name: Suave.WebSocket.handShake
module Successful

from Suave.Http
val OK : a:string -> Types.WebPart

Full name: Suave.Http.Successful.OK
module Files

from Suave.Http
val browseFile : rootPath:string -> fileName:string -> Types.WebPart

Full name: Suave.Http.Files.browseFile
val browse : rootPath:string -> Types.WebPart

Full name: Suave.Http.Files.browse

Published: Wednesday, 30 December 2015, 6:09 PM
Author: Tomas Petricek
Typos: Send me a pull request!
Tags: f#, data journalism, thegamma, data science, visualization