Unexpectedly today, I got progress displays working for uploads via WebDAV.
The roadblock had been that the interface of for uploading to S3 and WebDAV
is something like ByteString -> IO a. Which doesn't provide any hook to
update a progress display as the ByteString is consumed.
My solution to this was to create a hGetContentsObserved, that's similar
to hGetContents, but after reading each 64kb chunk of data from the
Handle to populate the ByteString, it runs some observing action. So when
the ByteString is streamed, as each chunk is consumed, the observer
runs. I was able to get this to run in constant space, despite not having
access to some of the ByteString internals that hGetContents is built
with.
So, a little scary, but nice. I am curious if there's not a better way to solve this problem hidden in a library somewhere. Perhaps it's another thing that conduit solves somehow? Because if there's not, my solution probably deserves to be put into a library. Any Haskell folk know?
Used above to do progress displays for uploads to S3. Also did progress display to console on download from S3. Now very close to being done with progressbars. Finally. Only bup and hook remotes need progress work.
Reworked the core crypto interface, to better support streaming data through gpg. This allowed fixing both the directory and webdav special remotes to not buffer whole files in memory when retrieving them as chunks from the remote.
Spent some time dealing with API changes in Yesod and Conduit. Some of them annoyingly gratuitous.