Git push freezes at writing objects and gives 504 error

Question

Problem

Git hangs while writing objects for a 81 kB repo over HTTPS (SSH is not allowed). The specific error message from Nginx's access.log is as follow:

POST /Hello-World/.git/git-receive-pack HTTP/1.1" 504 183 "-" "git/2.28.0.windows.1"

Background

I host Nginx and Git servers that are inside a mounted Debian image. Important to note: my www for all my website files are mounted separately as read-only. Also my Git root folder is inside www (i.e., www/git).

sudo mount -o loop,offset=1 raspbian-stretch-lite.img /mnt/
sudo mount --bind www/ /mnt/usr/share/nginx/www/
sudo mount -o bind,remount,ro /mnt/usr/share/nginx/www

My Nginx conf is as follows:

server {
  listen 443 ssl;
  server_name git.domain.com;
  ssl_certificate fullchain.pem;
  ssl_certificate_key privkey.pem;

  # fast clone static files
  location ~ \.git/objects/(?:[0-9a-f]+/[0-9a-f]+|pack/pack-[0-9a-f]+\.(?:pack|idx))$ {
    root /usr/share/nginx/www/git;
    limit_except GET HEAD {
      allow all;
    }
  }

  location ~ ^.*\.git/(HEAD|info/refs|objects/info/.*|git-(upload|receive)-pack) {
    root /usr/share/nginx/www/git;
    auth_basic "Restricted";
    auth_basic_user_file /usr/share/nginx/www/auth;
    include fastcgi_params;
    fastcgi_param SCRIPT_FILENAME /usr/lib/git-core/git-http-backend;
    fastcgi_param GIT_HTTP_EXPORT_ALL "";
    fastcgi_param GIT_PROJECT_ROOT $realpath_root;
    fastcgi_param REMOTE_USER $remote_user;
    fastcgi_param PATH_INFO $uri;
    fastcgi_param unix:/var/fcgiwrap.socket;
  }

  location /gitweb.cgi {
    root /usr/share/nginx/www/gitweb;
    include fastcgi_params:
    gzip off;
    fastcgi_param SCRIPT_NAME $uri:
    fastcgi_param GITWEB_CONFIG $realpath_root/gitweb.conf;
    fastcgi_pass unix:/var/fcgiwrap.socket;
  }

  location / {
    root /usr/share/nginx/www/gitweb;
    index gitweb.cgi;
  }
}

If I visit https://git.domain.com from browser, I get a Gitweb GUI. I clone with git clone https://git.domain.com/Hello-World/.git and all is well. But when I try to push using git push, it hangs at writing object step, eventually giving me a gateway timeout error. Interestingly, every time this happens, both https://git.domain.com Gitweb page and Git backend shut down, forcing me to kill and restart my fcgiwrap process.

My guess

I've tried git config --global http.postBuffer 524288000 to no avail. Since I'm authenticating (by typing my username and password), git-receive-pack should be allowed, right?

I'm assuming this has something to do with git folder not having write permission? Perhaps my Nginx config's regex is not correct?

Edits:

Normally errors from Git would go to stderr, which is usually written into a log by the web server. Is there such a log on your system, and if so, what does it contain?

The error I get in my command prompt from client-side is as follows:

(base) C:\Users\pairwiseseq\Hello-World>git push
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Writing objects: 100% (3/3), 258 bytes | 258.00 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0), pack-reused 0
error: RPC failed; HTTP 504 curl 22 The requested URL returned error: 504
fatal: the remote end hung up unexpectedly
fatal: the remote end hung up unexpectedly
Everything up-to-date

Also from Nginx's error.log:

*9 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 123.456.7.890, server: git.domain.com, request: "POST /Hello-World/.git/git-receive-pack HTTP/1.1", upstream: "fastcgi://unix:/var/fcgiwrap.socket", host: "git.domain.com"

strace for fcgiwrap process:

read(3, "\1\1\0\1\0\10\0\0\0\1\1\0\0\0\0\0\1\4\0\1\0041\7\0\f\0QUERY_STRING\16\4REQUEST_METHODPOST\f&CONTENT_TYPEapplication/x-git-receive-pack-request\16\3CONTENT_LENGTH452\v\"SCRIPT_NAME/Hello-World/.git/git-receive-pack\v\"REQUEST_URI/Hello-World/.git/git-receive-pack\f\"DOCUMENT_URI/Hello-World/.git/git-receive-pack\r\30DOCUMENT_ROOT/usr/share/nginx/www/git\17\10SERVER_PROTOCOLHTTP/1.1\16\5REQUEST_SCHEMEhttps\5\2HTTPSon\21\7GATEWAY_INTERFACECGI/1.1\17\fSERVER_SOFTWAREnginx/1.10.3\v\rREMOTE_ADDR123.456.7.890\v\5REMOTE_PORT39118\v\fSERVER_ADDR123.456.7.12\v\3SERVER_PORT443\v\20SERVER_NAMEgit.domain.com\17\3REDIRECT_STATUS200\17\"SCRIPT_FILENAME/usr/lib/git-core/git-http-backend\23\0GIT_HTTP_EXPORT_ALL\20\30GIT_PROJECT_ROOT/usr/share/nginx/www/git\v\10REMOTE_USERusername\t\"PATH_INFO/Hello-World/.git/git-receive-pack\24\4HTTP_ACCEPT_ENCODINGgzip\v\10HTTP_PRAGMAno-cache\17\31HTTP_USER_AGENTJGit/5.1.2.201810061102-r\22*HTTP_AUTHORIZATIONBasic cGc5eno0c2g6WWw5V1RpdnVrWlE2QTRNRw==\21&HTTP_CONTENT_TYPEapplication/x-git-receive-pack-request\v%HTTP_ACCEPTapplication/x-git-receive-pack-result\25\4HTTP_CONTENT_ENCODINGgzip\23\3HTTP_CONTENT_LENGTH452\t\20HTTP_HOSTgit.domain.com\17\nHTTP_CONNECTIONKeep-Alive\0\0\0\0\0\0\0\1\4\0\1\0\0\0\0\1\5\0\1\1\304\4\0\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0\0\0\0\0\1\5\0\1\0\0\0\0", 8192) = 1584
write(5, "\37\213\10\0\0\0\0\0\0\0\1\255\1R\37600b97fd1a60b01f91b314f59955a4e4d4e80d8edf11d f1a5d26dcb87f31b127cd397894aae43f86f362c refs/heads/master\0report-status delete-refs ofs-delta side-band-64k agent=JGit/5.1.2.201810061102-r0000PACK\0\0\0\2\0\0\0\3\234\fx\234\225\313\301\r\3020\f@\321{\246\310\2\225\354&v\32\t!\6\340\306\4I\354\10\16\220*r\367\247\214\300\345\35\276\364m\252zT&\26\222\36\326\232\2\244J\25\267\225\2651\"K<\273f\255\215\334^\246~\314\247.X\30*`\317X\3\306N9\23\225\250Q\242n \233JG\24W\16{\216\351m\234\\~\336\36\307\354\245\351r/\273\215}\tW\217\fk\f\t3\373\5\10\300\265\361~\277\314\364\337\317}\1q~<[\242\2x\234340031Q\10rut\361ue\210\233\301z.Q\341Ks\234\307\375\0O\311\233\5\355N\241:\0\257\246\v\211=x\234\363H\315\311\311WHLO\314\314S\344\2\0 d\4@\4{\334\"\301R\v\30\337\271\275\n\203\3349{D\255S;9\301\247\200\255\1\0\0", 452) = 452
read(6, "Expires: Fri, 01 Jan 1980 00:00:00 GMT\r\nPragma: no-cache\r\nCache-Control: no-cache, max-age=0, must-revalidate\r\nContent-Type: application/x-git-receive-pack-result\r\n\r\n", 4096) = 165

After this, fcgiwrap just completely hangs. I can neither clone nor push at this point and have to spawn a new fcgiwrap process.

Answer 1

Have you tried the answer from here?

git config --add remote.origin.proxy ""

Also there is an answer here with people complaining of 504 errors with a "push".

I also got HTTP 504 errors (gateway timeout) with push at first; for me, this was caused by a permission problem: fcgiwrap ran as user www-data, but the git repository directories were owned by another user.