git-annex cannot connect to freenet cloud (webdav)

Please describe the problem.

git-annex's WebDAV support does not like (aka it does not work) the WebDAV server of the freenet cloud.

What steps will reproduce the problem?

My first attempt was:

WEBDAV_USERNAME='XXX' WEBDAV_PASSWORD='XXX' git annex initremote webdav type=webdav url='https://webmail.freenet.de/webdav' encryption=none
initremote webdav (testing WebDAV server...) 
git-annex: WebDAV test failed: StatusCodeException (Status {statusCode = 401, statusMessage = "Unauthorized"}) [("Date","Fri, 04 Dec 2015 12:20:57 GMT"),("Server","Apache/2.2.16 (Debian)"),("WWW-Authenticate","Basic realm=\"MD-Cloud\""),
("Vary","Accept-Encoding"),("Content-Encoding","gzip"),("Content-Length","20"),("Connection","close"),("Content-Type","text/html; charset=iso-8859-15"),("X-Response-Body-Start",""),("X-Request-URL","MKCOL https://webmail.freenet.de:443/webdav/tmp")]
(CJ {expose = []}): user error failed
git-annex: initremote: 1 failed

Ok this fails (what is the error?). However, it does create a folder "tmp" in the "cloud". A second attempt yields another error:

WEBDAV_USERNAME='XXX' WEBDAV_PASSWORD='XXX' git annex initremote webdav type=webdav url='https://webmail.freenet.de/webdav' encryption=none
initremote webdav (testing WebDAV server...) 
git-annex: WebDAV test failed: StatusCodeException (Status {statusCode = 501, statusMessage = "Not Implemented"}) [("Date","Fri, 04 Dec 2015 12:21:22 GMT"),("Server","Apache/2.2.16 (Debian)"),("Content-Length","349"),("Connection","close"),
("Content-Type","application/xml; charset=utf-8"),("X-Response-Body-Start","<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<d:error xmlns:d=\"DAV:\" xmlns:s=\"http://sabredav.org/ns\">\n  <s:exception>Sabre\\DAV\\Exception\\NotImplemented</s:exception>\n  
<s:message>This server is not compatible with OS/X finder. Consider using a different WebDAV client or webserver.</s:message>\n  <s:sabredav-version>1.8.6</s:sabredav-version>\n</d:error>\n"),("X-Request-URL","PUT https://webmail.freenet.de:443/webdav
/tmp/git-annex-test")] (CJ {expose = []}): user error failed
git-annex: initremote: 1 failed

which is I guess the same. The WebDAV server does support writing and locking files. I tried writing using davfs2 and locking-unlocking using cadaver. I guess in the end that it's the server's fault, but it would be great to know what exactly fails at this point :).

What version of git-annex are you using? On what operating system?

git-annex version: 5.20151116-gbe86081

Gentoo Linux

Have you had any luck using git-annex before? (Sometimes we get tired of reading bug reports all day and a lil' positive end note does wonders)

First time trying WebDAV support.

RSS Atom

comment 1

ok, I could find the source of the problem: they use sabredav as WebDAV server and sabredav does not support chunked transfers:

    // Intercepting the Finder problem
    if (($expected = $request->getHeader('X-Expected-Entity-Length')) && $expected > 0) {

        /*
        Many webservers will not cooperate well with Finder PUT requests,
        because it uses 'Chunked' transfer encoding for the request body.

        The symptom of this problem is that Finder sends files to the
        server, but they arrive as 0-length files in PHP.

        If we don't do anything, the user might think they are uploading
        files successfully, but they end up empty on the server. Instead,
        we throw back an error if we detect this.

        The reason Finder uses Chunked, is because it thinks the files
        might change as it's being uploaded, and therefore the
        Content-Length can vary.

        Instead it sends the X-Expected-Entity-Length header with the size
        of the file at the very start of the request. If this header is set,
        but we don't get a request body we will fail the request to
        protect the end-user.
        */

        // Only reading first byte
        $firstByte = fread($body, 1);
        if (strlen($firstByte) !== 1) {
            throw new Exception\Forbidden('This server is not compatible with OS/X finder. Consider using a different WebDAV client or webserver.');
        }

        ...

Although, I did not told git-annex to chunk the transfer :-/, because I did not append a 'chunk' parameter. Any ideas how to fix that?

Comment by sts — Mon Dec 7 08:12:26 2015

Remove comment

comment 2

The first failure is git-annex sending MKCOL (make directory basically). The server fails with "Unauthorized". You say it also made the directory. That's got to be a bug in the server, no? It can't sanely have an authorization problem and also go on and do the unathorized action. (Sounds rather like a security hole..)

As to the PUT failure, the chunked transfer encoding mentioned in that comment is a regular part of the HTTP protocol (this is not connected to git-annex's own chunking). https://en.wikipedia.org/wiki/Chunked_transfer_encoding

Looks like this PHP webdav server might be delegating the actual HTTP to whatever web server it's running on somehow. Since chunked transfer encoding might not be supported by some web server, they are left trying to detect that. I don't know if their check for that is accurate.

As to the implementation in git-annex, Network.Http.Client.RequestBodyStreamChunked is documented to be the only thing that causes a chunked request body to be sent, and git-annex is using RequestBodyLBS instead. Unless the documentation is wrong (and I also looked at the http-client source code and the documentation seems accurate), I am doubtful that the chunked transfer encoding is actually being used by git-annex. If eg a protocol dump shows that it is in fact using chunked transfer encoding (ie, contains "Transfer-Encoding: chunked"), that would be grounds to file a bug on the http-client library.

Aah, but.. git-annex is sending an empty file. And the webdav server's check consists of reading 1 byte. Of course there's not a byte to read if an empty file is being sent! So that code you showed is certianly buggy.

I've changed git-annex to send a non-empty file when testing the webdav server to work around this.

Comment by joey — Fri Dec 11 15:08:30 2015

Remove comment

comment 3

hey joey!

thanks, it (kinda) worked :). At least the initialization works now pretty fine. But uploading files does not work as expected. It does upload some data, but does not move the data to the corresponding folder (from 'tmp' to $hash):

git annex copy --debug --to webdav 1354580391258.jpg
[2015-12-20 20:31:36.985579] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","git-annex"]
[2015-12-20 20:31:36.990098] process done ExitSuccess
[2015-12-20 20:31:36.990584] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","show-ref","--hash","refs/heads/git-annex"]
[2015-12-20 20:31:36.993635] process done ExitSuccess
[2015-12-20 20:31:36.994311] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","log","refs/heads/git-annex..2ff0c487bc1c92c11a45131b55f6fb2ba034071d","-n1","--pretty=%H"]
[2015-12-20 20:31:36.998675] process done ExitSuccess
[2015-12-20 20:31:36.999747] chat: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","cat-file","--batch"]
[2015-12-20 20:31:37.003483] read: git ["--git-dir=.git","--work-tree=.","--literal-pathspecs","ls-files","--cached","-z","--","1354580391258.jpg"]
copy 1354580391258.jpg (checking webdav...) (to webdav...) 
100%        564.6KB/s 0s
DAV failure: 409 "Conflict"
failed
git-annex: copy: 1 failed

In the 'tmp'-folder on the server I can find the file ('SHA256E-s1156230--...'), but this file is not 100% identical to the one I uploaded. Well, it seems that the first byte is missing at least if I compare the files via vimdiff:

0000000: d8ff e000 104a 4649 4600 0101 0100 4800  .....JFIF.....H.               |  0000000: ffd8 ffe0 0010 4a46 4946 0001 0101 0048  ......JFIF.....H               
0000010: 4800 00ff db00 4300 0101 0101 0101 0101  H.....C.........               |  0000010: 0048 0000 ffdb 0043 0001 0101 0101 0101  .H.....C........               
0000020: 0101 0101 0102 0203 0202 0202 0204 0303  ................               |  0000020: 0101 0101 0101 0202 0302 0202 0202 0403  ................               
0000030: 0203 0504 0505 0504 0404 0506 0706 0505  ................               |  0000030: 0302 0305 0405 0505 0404 0405 0607 0605  ................               
0000040: 0706 0404 0609 0607 0808 0808 0805 0609  ................               |  0000040: 0507 0604 0406 0906 0708 0808 0808 0506  ................               
0000050: 0a09 080a 0708 0808 ffdb 0043 0101 0101  ...........C....               |  0000050: 090a 0908 0a07 0808 08ff db00 4301 0101  ............C...               
0000060: 0202 0204 0202 0408 0504 0508 0808 0808  ................               |  0000060: 0102 0202 0402 0204 0805 0405 0808 0808  ................

As you can see, on the left side (aka 'the uploaded file') the first two hexadecimal values are missing in comparison to the right side (the source), so the first byte is missing.

I guess in the end it is still the same problem, right :)?

Comment by sts — Sun Dec 20 19:49:26 2015

Remove comment

comment 4

As to where the first byte went, it seems rather suggestive that the part of the server's code you pasted earlier reads out $firstByte. Perhaps it forgot to preserve that first byte in the file.

Which is horrible, but then there's the "Conflict" error. All I can make of this is that the server is probably rejecting the MOVE request for some reason. That might be reasonable if the destination file already existed, but I don't think it does in this case, so who knows.

Only thing I'm sure of is, I never want to entrust this particular webdav server with any of my data!

Comment by joey — Wed Jan 20 19:22:36 2016

Remove comment

Add a comment