Dr Max Grossmann

The world-historical mess of using proxies in Go (and other languages)

Posted 2026-07-22 · Last updated 2026-10-04 ·

Two of my favorite pieces of software are written in Go: Syncthing, which does continuous file synchronization, and restic, which does encrypted, deduplicated backups. Both are truly, truly excellent — some of the best software ever written: reliable, well documented, and maintained with obvious care. This post is not about a flaw in either of them. It is about what happens when you try to run Go programs over a proxy — and about a mess that sits in Go's libraries, beneath both. And lest anyone think this is a Go problem: further down, we take a detour into Python, where Amazon's own AWS CLI manages to make things worse.

Why you would proxy Syncthing at all

It can be a good idea to run Syncthing over a proxy such as Tor. Your Syncthing device ID is not exactly a secret: you hand it to everyone you sync with. Anyone who ever knew your device ID can query Syncthing's global discovery servers for the addresses your device announces and can therefore permanently track your IP address and also know where you are. Routing Syncthing through Tor decouples your device ID from your physical location. I currently use Syncthing only with my own devices, but for you, that may matter (e.g., if you use Syncthing to securely collaborate with coauthors and share data, which I have done in the past).

Syncthing gets proxying right

Syncthing supports this out of the box via environment variables. In a systemd unit:

[Service]
Environment="ALL_PROXY=socks5://127.0.0.1:9050"
Environment="ALL_PROXY_NO_FALLBACK=1"
Environment="NO_PROXY=192.168.0.0/16,10.0.0.0/8,172.16.0.0/12"

Three things are going on here:

Under the hood, Syncthing builds its dialing on golang.org/x/net/proxy, and credit where credit is due: this is a clean, well-behaved little library. It reads ALL_PROXY and NO_PROXY in both spellings, and — best of all — when given a hostname, its SOCKS5 implementation sends that hostname to the proxy (SOCKS5 address type 0x03) instead of resolving it locally, so name resolution happens on the far side of the proxy. That is exactly the right default: DNS leaks are the classic way to ruin an otherwise proxied setup. In curl's terminology this is socks5h:// behavior; the library treats socks5:// and socks5h:// identically, and in the safe direction.

So far, so pleasant. Now suppose that, out of curiosity, you try the same trick with another Go program you trust. Say, restic.

The world-historical mess

Set ALL_PROXY=socks5://127.0.0.1:9050 for restic and run a backup to an S3 bucket. It completes flawlessly. There is just one small problem: none of that traffic went through your proxy. No error, no warning — your backup traffic, and your DNS lookups, went out over the plain network.

This is not restic's fault. Restic does the textbook-correct thing: it builds its HTTP transport with Proxy: http.ProxyFromEnvironment, the standard mechanism that virtually every Go HTTP client uses. The problem is what that mechanism actually reads.

Go has two entirely separate libraries for reading proxy configuration from the environment, and they disagree about which variables exist:

So whether ALL_PROXY does anything depends entirely on which Go program you happen to be running, and you cannot tell without reading its source. The Go team is aware: issue #31813, filed in 2019 by Brad Fitzpatrick himself, proposes that net/http.DefaultTransport honor ALL_PROXY. As of this writing it is still open. Related warts go back even longer: #16715 (for an unregistered non-SOCKS scheme, FromURL returns unknown scheme, while FromEnvironment swallows that error and silently returns the direct dialer) and #13456 (for a while, only the lowercase all_proxy spelling was read).

For restic specifically, the fix is to use the variables that net/http does read:

HTTPS_PROXY=socks5h://127.0.0.1:9050

(plus HTTP_PROXY if your backend is reached over plain HTTP).

Meanwhile, in Python: the AWS CLI

Having fixed restic, you will at some point want to poke at that same S3 bucket with Amazon's own AWS CLI, which is written in Python. My own use case is mundane: on dodgy connections, I route some traffic through a VPS I trust, using a plain ssh -D12345 vps SOCKS tunnel courtesy of OpenSSH. So you set HTTPS_PROXY as you did for restic, now pointing it at the SSH tunnel:

HTTPS_PROXY=socks5h://127.0.0.1:12345

But the AWS CLI greets you with this:

Failed to connect to proxy URL: "http://socks5h://127.0.0.1:12345"

Yes, really: http://socks5h://. The AWS CLI v2 ships its own vendored copy of botocore, whose ProxyConfiguration._fix_proxy_url “fixes” any proxy URL that does not begin with http: or https: by gluing http:// to its front. It then tries to connect to the abomination it has just built, fails, and reports the abomination back to you as if you had typed it.

Repairing the glue would not help, either. The proxy URL ends up in urllib3's ProxyManager, which speaks HTTP and HTTPS proxies only and rejects everything else with ProxySchemeUnknown. urllib3 does support SOCKS, through a separate SOCKSProxyManager that needs the optional PySocks package — but the AWS CLI uses neither. The world's most prominent cloud command-line tool simply does not do SOCKS, and instead of saying so, it invents a URL.

In my test, the malformed proxy URL failed before reaching S3. But urllib3 parses it as an HTTP proxy at socks5h:80, so this is not guaranteed to fail closed if that hostname resolves.

ALL_PROXY, on the other hand, is the restic trap all over again, with a different culprit. Botocore collects proxies with Python's urllib.request.getproxies(), which dutifully picks up every *_PROXY variable, ALL_PROXY included — and files it under the key all, as it should. Botocore then looks up a proxy by the scheme of the request URL, https, finds nothing, and connects directly. I tested this with AWS CLI 2.23.6: with ALL_PROXY=socks5h://127.0.0.1:1, pointing at a port where nothing listens, aws s3 cp cheerfully completes a download over the open network. No error, no warning. In the same environment, curl refuses to connect, as it should.

The fix: put an HTTP proxy in front of the SOCKS proxy

If the AWS CLI insists on an HTTP proxy, give it one that forwards everything to the SOCKS tunnel. OpenSSH cannot do this by itself: -D is its only proxy, and it speaks SOCKS only. Privoxy (apt install privoxy on Debian) can. Privoxy is an ad blocker at heart, but stripped down, it makes a perfectly dumb pipe. This is my complete /etc/privoxy/config:

confdir /etc/privoxy
logdir /var/log/privoxy
logfile logfile

listen-address 127.0.0.1:8118

forward-socks5t / 127.0.0.1:12345 .

toggle 0
enable-remote-toggle 0
enable-remote-http-toggle 0
enable-edit-actions 0
accept-intercepted-requests 0
forwarded-connect-retries 0
keep-alive-timeout 5
socket-timeout 300

Line by line, for the parts that matter:

Then:

sudo privoxy --config-test /etc/privoxy/config
sudo systemctl restart privoxy
export HTTPS_PROXY=http://127.0.0.1:8118
export HTTP_PROXY=http://127.0.0.1:8118

To check that it works, compare your apparent IP address three ways (ip.mg.sb is a service I run that returns the caller's address):

curl -s --noproxy '*' https://ip.mg.sb                  # direct: your own address
curl -s -x socks5h://127.0.0.1:12345 https://ip.mg.sb   # via ssh -D: the VPS
curl -s -x http://127.0.0.1:8118 https://ip.mg.sb       # via Privoxy: must also be the VPS

Note that this tests Privoxy, not the AWS CLI.

If your SOCKS proxy is Tor, skip all of this: Tor can expose an HTTP CONNECT proxy itself. Add HTTPTunnelPort to your torrc and point HTTPS_PROXY at it. I haven’t tried that, though.

Takeaways

First: ALL_PROXY is a coin toss. Go's standard HTTP transport ignores it, and so does the AWS CLI. Set HTTP_PROXY and HTTPS_PROXY instead. What those may point to differs, too: Go happily takes socks5h://, while the AWS CLI needs an actual HTTP proxy, such as Privoxy in front of your SOCKS tunnel. There is no single setting that does the right thing everywhere.

Second: never just assume that setting a proxy variable did anything. The simplest test is to point the variable at a port where nothing is listening and check that the program now fails. That is how I caught the AWS CLI: a download that should have been impossible went through without a hitch.

Third: if the proxy is security-critical, look up how the specific program dials out. Even a program that honors ALL_PROXY may, like Syncthing without ALL_PROXY_NO_FALLBACK, treat it as a suggestion rather than a rule.

None of this diminishes restic or Syncthing — they remain some of the finest software in existence, and Syncthing in particular went out of its way to get proxying right. The mess lives a layer below them, in a standard library and an extension library that have disagreed about the environment for a decade. And Go is merely the most interesting offender, not the only one: Amazon, with all the resources in the world, ships a CLI that ignores ALL_PROXY in silence and greets a SOCKS URL by turning it into http://socks5h://. Decades after the first *_proxy variable, there is still no shared understanding of what any of them mean. World-historical, indeed.