Offline POST requests with Elm and Service Worker

Err disconnected logos

Elm developers with an interest in making full use of the web platform might be disappointed to find that many of the shinier features are not yet supported. If that’s the case with a feature you’d like to make use of, you might want to look at communicating between Elm and JavaScript using ports.

In this post, we’re going to explore how ports might be used when working with Service Worker and Background Sync. We’ll look to extend a small application which automates the insertion of markdown-formatted image tags into a textarea, such that it no longer breaks when the connection is out or when an upload fails for some other reason.

Initial application
Offline-ready with Service Worker
Improving offline behaviour
Retries with asynchronous work

Each iteration includes an interactive demo, but whether using these or trying this out for real, please note that Background Sync is currently not widely supported. Try Chrome!

Initial application

Our starting point is a simple Elm application consisting of a textarea, and file input. When a user selects a file, we’ll upload it (faked for simplicity’s sake in these demos), then insert a markdown image tag at the location of the cursor:

Insert image

In this starter application, we have a simple model storing the current state of the textarea. When a user selects a file, we send it to JavaScript and query for the current cursor position, before sending both back to Elm. Here we update the model with the image tag at the correct position.

Sadly, this does not work if the user is offline, or their upload fails for some other reason. Our end goal is something which behaves roughly like the above, from the user’s point of view, but allows them to continue editing text and inserting images while offline. On top of this, if they have a connection but the upload still fails, we should look to retry the attempt using Background Sync.

Let’s go!

Offline-ready with Service Worker

For the initial service worker setup we’re going to borrow an approach from Phil Nash on the Twilio blog, and rather than repeat Phil’s description verbatim I’ll skip ahead to the interaction between this setup and Elm. Just know that we’re using the idb library to simplify interaction with IndexedDB, our persistence mechanism, and a store variable wrapping interaction with the particular database instance.

Once we’ve imported idb and set store, we can look at hooking up our service worker, so we only ask Elm to attempt an upload when online. In our initial application, seen above, the upload attempt is triggered when Elm receives a file and cursor position over the receiveCursorForFile port. In order to make this work offline, we’ll want to make an update such that this action is triggered by the service worker’s sync handler.

In order to do this, we need only make changes outside of Elm, for now. Where we previously subscribed to and sent messages across ports, let’s wrap the subscription in a service worker registration callback—we can’t do anything useful with files received here until the service worker is ready. We’ll drop the sending of the file and cursor back to Elm for now, but come back to that later on:

app.js

1
2
3
4
5
6
7
8
9
10
11
12
 var app = Elm.Main.embed(elmDiv);

-app.ports.askCursorForFile.subscribe(function(payload) {
-  var textarea = elmDiv.querySelector("textarea");
-  var cursorPosition = textarea.selectionStart;
-  app.ports.receiveCursorForFile.send({"cursor": cursorPosition, "file": payload});
+navigator.serviceWorker.register("sw.js").then(function(reg) {
+  app.ports.askCursorForFile.subscribe(function(payload) {
+    var textarea = elmDiv.querySelector("textarea");
+    var cursorPosition = textarea.selectionStart;
+  });
 });

Now, let’s store the file data received in messages on this subscription, along with the cursor position we calculate, in IndexedDB. With that done, we can use it’s return value (the row id) in a call to the service worker’s sync handler:

1
2
3
4
5
6
7
     var cursorPosition = textarea.selectionStart;
+    store.outbox("readwrite").then(function(outbox) {
+      var row = {"cursor": cursorPosition, "file": payload}
+      return outbox.put(row)
+    }).then(function(id) {
+      return reg.sync.register(id);
+    });

Next, we’ll look at the service worker itself. We need it to handle this sync event, where things get interesting.

Typically, this event handler might be expected to retrieve the data it needs and then call fetch, or similar, attempting the request. In our case though, we want Elm to be responsible for attempting the upload.

As the service worker isn’t aware of the Elm application, it can’t send a message across its ports, triggering this upload directly. Instead, we’ll send a message back to the service worker’s clients.

Additionally, we’re going to use clients.claim() in the worker’s activate handler. We want this as otherwise the worker won’t take control of the page on its initial load—and so we won’t see any links inserted at all!

Here’s how that looks:

sw.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
importScripts("/js/idb.js");
importScripts("/js/store.js");

self.addEventListener("activate", function(event) {
  event.waitUntil(self.clients.claim());
});

self.addEventListener("sync", function(event) {
  upload(event.tag)
});

function upload(rowId) {
  return store.outbox("readonly").then(function(outbox) {
    return outbox.get(parseInt(rowId));
  }).then(function(row) {
    clients.matchAll().then(function(clients) {
      channel = new MessageChannel();
      clients.forEach(function(client) {
        client.postMessage(row, [channel.port1]);
      });
    });
  });
}

Now, back in our main JavaScript, we can listen for messages, and upon receiving one, forward the data on through the receiveCursorForFile port we removed earlier:

app.js

1
2
3
4
5
6
7
 app.ports.askCursorForFile.subscribe(function(payload) {
 //...
 });
+
+navigator.serviceWorker.addEventListener("message", function(event) {
+  app.ports.receiveCursorForFile.send(event.data);
+});

Try it out! Go offline, enter a file, then come back online again. After a brief pause, the link should appear.

Insert image

You might notice though that the approach has a major flaw: when offline, you might try to insert 2 links. Then at the point of coming back online, by the time of the 2nd insertion the textarea state has changed, and so the stored cursor position may no longer be correct. We get similar behaviour if we edit text while waiting for an upload. Next we’ll look to resolve this.

Improving offline behaviour

In order to fix this, we need to minimize the time passing between a file being chosen and the insertion of a tag in the textarea. One way for us to do so is by inserting an initial tag as soon as we know the cursor position while deferring the much slower upload until later. Then, once we have this, we’ll update the model once more with the URL.

To make this a little simpler, we can switch to using markdown’s reference links format for the inserted tags. This will mean rather than searching through the text for a specific tag to update once we have the URL, we can simply append the URL value to the current state:

1
2
3
4
5
6
7
Some really interesting text that should be supplemented with an image.

![][1]

Lovely picture that.

[1]: http://example.com/image.png

With this strategy, the flow of data might look like so:

User selects a file;
We send a message to JavaScript, including the encoded file;
This message is persisted to IndexedDB, then JavaScript sends the ID and cursor position back to Elm. We update the state with the 1st part of the reference link, at this position;
Later, the upload actually occurs. Once complete we simply append the corresponding [id]: URL definition to the textarea.

Let’s give this a shot. First, we’ll look at handling the initial link insertion, prior to hitting the service worker, by moving the receiveCursorForFile send earlier in the process.

app.js

1
2
3
4
5
6
7
8
9
10
11
12
         return outbox.put(row)
       }).then(function(id) {
+        var cursorId = {"id": id, "cursor": cursorPosition}
+        app.ports.receiveCursorForFile.send(cursorId);
+
         return reg.sync.register(id);
       });
     });

     navigator.serviceWorker.addEventListener("message", function(event) {
-      app.ports.receiveCursorForFile.send(event.data);
     })

With this change, receiveCursorForFile sends the ID and cursor, rather than the file and cursor as previously. Back in our Elm application, let’s update the alias we’ll use when decoding the received message, to reflect this change:

Model.elm

1
2
3
4
5
6
-type alias CursorFile =
+type alias CursorId =
     { cursor : Int
-    , file : NativeFile
+    , id : Int
+    }

Now we can look at updating the tag insertion in Main. Where we previously went straight on to perform an upload when receiving this message, we now want to quickly update the model based only upon the received cursor position and ID:

Main.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-        InsertImgTag cursorForFile ->
-            let
-                cmd =
-                    cursorForFile
-                        |> Json.Decode.decodeValue Decoders.cursorFileDecoder
-                        |> uploadFile
-            in
-                model ! [ cmd ]
+        InsertImgTag cursorForId ->
+            let
+                cursorId =
+                    cursorForId
+                        |> Json.Decode.decodeValue Decoders.cursorIdDecoder
+            in
+                (Model.insertImgTag model cursorId) ! []

We need to adapt Model.insertImgTag for this new reality. We now need it to handle the result of decoding the message coming across the port, instead of an integer representing the cursor position. Assuming this decode is successful, we put the 1st part of our reference tag at the cursor’s position:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
-insertImgTag : Model -> Int -> String -> Model
-insertImgTag model cursor url =
-    let
-        toPrepend =
-            String.left cursor model.textAreaContents
-
-        toInsert =
-            imgTag url
-
-        toAppend =
-            String.dropLeft cursor model.textAreaContents
-
-        newState =
-            String.concat [ toPrepend, toInsert, toAppend ]
-    in
-        { model | textAreaContents = newState }
+insertImgTag : Model -> Result String CursorId -> Model
+insertImgTag model cursorId =
+    case cursorId of
+        Ok value ->
+            let
+                toPrepend =
+                    String.left value.cursor model.textAreaContents
+
+                toInsert =
+                    referenceTagInitial value.id
+
+                toAppend =
+                    String.dropLeft value.cursor model.textAreaContents
+
+                newState =
+                    String.concat [ toPrepend, toInsert, toAppend ]
+            in
+                { model | textAreaContents = newState }
+
+        _ ->
+            model


-imgTag : String -> String
-imgTag url =
-    "![](" ++ url ++ ")"
+referenceTagInitial : Int -> String
+referenceTagInitial id =
+    "![][" ++ toString id ++ "]"

Now, we can move on to handle the upload. We’ll need to send a message across a new port asking Elm to do so, triggered by the service worker sending a message to its clients, as previously:

app.js

1
2
3
     navigator.serviceWorker.addEventListener("message", function(event) {
+      app.ports.receiveTryUploadForFile.send(event.data);
     });

We need to define this new port:

Ports.elm

1
2
3
4
 port receiveCursorForFile : (Value -> msg) -> Sub msg
+
+
+port receiveTryUploadForFile : (Value -> msg) -> Sub msg

And now, let’s subscribe to this, and add a new AppendImgUrl message handler, which decodes the received payload, and triggers the upload:

Main.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
+        AppendImgUrl fileForId ->
+            let
+                cmd =
+                    fileForId
+                        |> Json.Decode.decodeValue Decoders.fileIdDecoder
+                        |> uploadFile
+            in
+                model ! [ cmd ]

# ...

-uploadFile : Result String CursorFile -> Cmd Msg
-uploadFile cursorFile =
-    case cursorFile of
+uploadFile : Result String FileId -> Cmd Msg
+uploadFile result =
+    case result of
         Ok fileId ->
             let
                 uploadResponse =
                     "http://example.com/image.png"

                 request =
                     Task.succeed uploadResponse
             in
-                Task.attempt (UploadComplete value.cursor) request
+                Task.attempt (UploadComplete fileId.id) request

         Err error ->
             Cmd.none


 subscriptions model =
     Sub.batch
         [ Ports.receiveCursorForFile InsertImgTag
+        , Ports.receiveTryUploadForFile AppendImgUrl
         ]

Next, we’ll fix the UploadComplete message handler where we update the model’s state, to make use of the passed in ID, instead of the cursor position, which it will use to append the 2nd part of a reference link to the textarea’s contents:

1
2
3
4
-        UploadComplete cursor (Ok result) ->
+        UploadComplete id (Ok result) ->
-            (Model.insertImgTag model cursor result) ! []
+            (Model.appendImgUrl model id result) ! []

The final changes are in the Model module. We just need to define the type which links a file with an ID and the new appendImgUrl function:

Model.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
     = TextEntered String
     | Files (List NativeFile)
     | InsertImgTag Json.Encode.Value
     | UploadComplete Int (Result Http.Error String)
+    | AppendImgUrl Json.Encode.Value

# ...

+type alias FileId =
+    { file : NativeFile
+    , id : Int
}

# ...

+appendImgUrl : Model -> Int -> String -> Model
+appendImgUrl model id url =
+    let
+        newState =
+            model.textAreaContents
+                ++ "\n["
+                ++ toString id
+                ++ "]: "
+                ++ url
+    in
+        { model | textAreaContents = newState }

We need some changes to our decoders to hook this up also. Elm decoders are a long article of their own so I won’t go into the details here, but for completeness, here are the changes:

Decoders.elm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
-cursorFileDecoder : Decoder CursorFile
-cursorFileDecoder =
-    Json.Decode.map2 CursorFile
+cursorIdDecoder : Decoder CursorId
+cursorIdDecoder =
+    Json.Decode.map2 CursorId
         (Json.Decode.field "cursor" Json.Decode.int)
+        (Json.Decode.field "id" Json.Decode.int)
+
+
+fileIdDecoder : Decoder FileId
+fileIdDecoder =
+    Json.Decode.map2 FileId
         (Json.Decode.field "file" nativeFileDecoder)
+        (Json.Decode.field "id" Json.Decode.int)

With this done, we should be in a much better position. Go offline, and try entering a few links at various positions in the textarea. All being well, we should see strings of the form ![][999] entered before you have a chance to do anything else.

Once you’ve placed a few of these, go online again to see the [999]: http://example.com/image.png strings appended.

Insert image

This is pretty good, but we can take it one step further. While this implementation handles offline submissions, it doesn’t handle online submissions which fail.

Retries with asynchronous work

Fortunately, Background Sync has our back here. By calling waitUntil() in our service worker’s sync handler, and returning a resolved or rejected Promise here, we can indicate whether the work is done, or needs retrying.

The difficulty we have around this is down to the asynchronous nature of passing messages across Elm ports. Once our service worker has asked for an upload attempt to occur, it will receive no response, and so has no idea whether or not the attempt has succeeded, and so whether it should schedule a retry.

We can look to get around this by indicating the state of a given upload in IndexedDB, and by assuming the upload has failed at the 1st attempt. Once our upload has succeeded, we’ll update the state in IndexedDB to represent this, then should a retry event occur, we can return a resolved Promise before attempting the upload again, like so:

User selects a file;
We store the file and cursor in IndexedDB, insert a reference link at the current position, and ask the service worker to attempt an upload, as previously;
Our worker’s sync event calls event.waitUntil(), which wraps our logic determining whether or not work is complete;
Here, we check the row’s status first. Initially, it is set to pending, so we continue, passing a message to back to the client asking it to attempt an upload, before returning a rejected Promise;
A retry is scheduled;
Meanwhile, our upload succeeds, so Elm sends a message to app.js, where we update the row’s status to be succeeded;
Sometime later, the service worker tries again. It checks the row’s status once more, sees succeeded, and so returns a resolved Promise without retrying the upload.

So first of all, let’s set the initial state of a stored file in IndexedDB to pending:

app.js

1
2
3
4
5
6
7
8
9
       store.outbox("readwrite").then(function(outbox) {
-        var row = {"retry", "cursor": cursorPosition, "file": payload}
+        var row = {
+          "cursor": cursorPosition,
+          "file": payload,
+          "status": "pending"
+        }
         return outbox.put(row)
       }).then(function(id) {

Next, we’ll set up the service worker to retry uploads. Where we previously attempted an upload directly in the sync handler, we now wrap this in a waitUntil() call. This is the part of the Background Sync API that will handle retries for us. In order to get that working, we only need to return a resolved promise for the success case, or a rejected one when we want a retry.

So in our case, we first check the row’s status. If this is succeeded, we’ve already completed the upload, so resolve now. Otherwise, we perform the same actions as previously—sending a message to our client and triggering an upload attempt, before rejecting the Promise.

sw.js

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
 self.addEventListener("sync", function(event) {
-  upload(event.tag)
+  event.waitUntil(upload(event.tag));
 });

 function upload(rowId) {
   return store.outbox("readonly").then(function(outbox) {
     return outbox.get(parseInt(rowId));
   }).then(function(row) {
-    clients.matchAll().then(function(clients) {
-      channel = new MessageChannel();
-      clients.forEach(function(client) {
-        client.postMessage(row, [channel.port1]);
-      });
+    if (row.status == "succeeded") {
+      return Promise.resolve();
+    } else {
+      askForUpload(row);
+      return Promise.reject();
+    }
+  });
+}
+
+function askForUpload(row) {
+  clients.matchAll().then(function(clients) {
+    channel = new MessageChannel();
+    clients.forEach(function(client) {
+      client.postMessage(row, [channel.port1]);
     });
   });
 }

The last change we need is to set the succeeded status on upload complete. We’ll use ports to do so again, which shouldn’t take too much work to hook up. First, let’s send a message across a new port once an upload is done:

Ports.elm

1
+port uploadSuccessful : Int -> Cmd msg

Main.elm

1
2
3
4
5
6
7
         UploadComplete id (Ok result) ->
-            (Model.appendImgUrl model id result) ! []
+            let
+                cmd =
+                    Ports.uploadSuccessful (id)
+            in
+                (Model.appendImgUrl model id result) ! [ cmd ]

Now all we need is to handle this message, updating the status to succeeded once it comes through:

app.js

1
2
3
4
5
6
7
8
+    app.ports.uploadSuccessful.subscribe(function(id) {
+      store.outbox("readwrite").then(function(outbox) {
+        outbox.get(id).then(function(row) {
+          row.status = "succeeded";
+          outbox.put(row);
+        });
+      });
+    });

Give it a go! As our fake uploads will never fail, you perhaps wont see much happen here. In Chrome’s developer tools though, we should see the changes.

Select a file with WiFi off, and you’ll see a new entry in the uploads database, with state pending. Now turn WiFi back on, and you should see the URL portion of the link appended, while the row in IndexedDB is updated to succeeded.

Insert image

Wrapping up

That’s it! We’ve successfully adapted our Elm application to handle offline upload attempts and retries of failures.

This seems to work pretty well, but our implementation perhaps has some room for improvement:

The message passing between Service Worker, our JavaScript client, and Elm makes this a rather involved process. I wonder whether Elm is the best choice for this sort of app.
If the user has more than one tab open on the editor screen, we have no way of telling which should have a tag inserted—we currently ask all clients to attempt an upload.
If the user closes the editor while we have retries scheduled, they will still be attempted by the service worker.
Background Sync’s retries will only happen twice more. Perhaps we should look to trigger sync again if the third attempt fails;
The file handling could be improved: it currently accepts any type of file, regardless of whether it’s actually an image and it is not cleared after an upload, meaning if the same file is uploaded twice in a row, the 2nd does nothing.
Storing entire files in IndexedDB is going to get large quick. We should remove them once uploaded, minimum, and perhaps think about whether to do so in other case also.

Regardless, it was a fun exercise. If you’re keen to see a more fully worked example, I’ve made an implementation public on GitHub. I look forward to seeing how Elm exposes this functionality in the future!