Cloudflare Argo Tunnel allows you to expose your web server to the internet without having to open routes in your firewall or setup dedicated routes. This guide will outline how you can setup Cloudflare Argo Tunnel for private encrypted HTTP/2 connection to your origin web server or web application on CentOS 7 for Centmin Mod LEMP stack users.
The below instructions for setting up a Cloudflare Argo Tunnel HTTP/2 or QUIC UDP based connection to a self hosted application/origin server is tailored to Centmin Mod LEMP stack users running CentOS 7 running CSF Firewall. The guide will be setting up Cloudflare Argo Tunnels using Named Tunnels and a Ingress Rule based configuration which allow you map traffic for multiple hostnames and services within a single Argo Tunnel.
Cloudflare Argo Tunnels Now Cloudflare Tunnels
Update as of April 15, 2021, Cloudflare Argo Tunnels has been renamed Cloudflare Tunnels to reflect that Cloudflare Tunnels are now free to all Cloudflare users regardless of their Cloudflare subscription plan. Personally, I’d still recommend you sign up for Cloudflare For Teams free subscription as that offering provides more than just Cloudflare Tunnels which you can advantage of.
What Is Cloudflare Argo Tunnel
Cloudflare Argo Tunnel is a private encrypted HTTP/2 or QUIC UDP based connection between your web server and Cloudflare which makes it so that only traffic that routes through Cloudflare can reach your server or web application without needing a publicly routable IP address.
An example request follows these steps:
- A visitor makes a request to tunnel.yourdomain.com.
- The DNS lookup resolves to a Cloudflare network address.
- The visitor connects to the closest Cloudflare edge PoP via Anycast.
- Cloudflare routes the visitor through a special PoP-to-PoP route called Argo Smart Routing and connects them to a Cloudflare edge PoP that has an established persistent connection to the daemon (
cloudflared
) running on the visitor’s web server. - The request is routed to the
cloudflared
instance running on your server. The connection betweencloudflared
and the Cloudflare edge is a long-lived persistent HTTP/2 or QUIC UDP configurable connection encrypted with TLS. To keep the connection alive,cloudflared
sends a heartbeat to the edge in the form of a ping frame over HTTP/2. If the connection is dropped, thecloudflared
client re-establishes the connection with Cloudflare.cloudflared
connects to Cloudflare on port 7844. All packets between Cloudflare and the tunneled web server use stream multiplexing over HTTP/2 or QUIC UDP. In HTTP/2, each request/response pair is called a Stream and given a unique Stream ID so that these streams can be “multiplexed” or sent asynchronously over the same connection. QUIC UDP improves on HTTP/2 connections by solving Head of Line (HoL) blocking as well making it possible for Cloudflare to supportcloudflared
tocloudflared
communication. - The Argo Tunnel client forwards the request to your web service.
Contents
This guide will outline both manual (with cloudflared
) and automated methods (via Cloudflare API with Cloudflare API Tokens) and is based on official Cloudflare documentation and Cloudflare Argo Tunnel FAQ documentation.
- Sign Up For Cloudflare For Teams Free Subscription
- CSF Firewall
- Create Centmin Mod Nginx Vhost Sites
- Manual Argo Tunnel Setup with cloudflared
- Automated Argo Tunnel Setup with Cloudflare API
- Cloudflare Zero Trust Terminal Browser
- Listing Argo Tunnels Created
- Centmin Mod Nginx Access Logs
- Cloudflared Log Rotation
- Cloudflared System Settings
- Cloudflare Authenticated Origin Pull Incompatibility
- Cloudflare Restrict IPs
- Cloudflare Tunnel http2 vs quic protocol connections
Sign Up For Cloudflare For Teams Free Subscription
The easiest way for your Cloudflare Account to get access to free Argo Tunnel feature is to sign up for Cloudflare For Teams free subscription which bundles Cloudflare Gateway, Cloudflare Access, Cloudflare Browser Isolation and Cloudflare Argo Tunnels together and is available as add-ons to your existing Cloudflare subscription. The official Cloudflare For Teams documentation is available here.
To sign up for the free Cloudflare For Teams subscription, go to the Cloudflare Team dashboard link at https://dash.teams.cloudflare.com/ and log into your Cloudflare Account and go through the Cloudflare For Teams onboarding process. You’d want to ensure your Cloudflare account’s billing details for credit card and billing address are updated if you choose a paid subscription instead of a free one.
Then go through the Cloudflare For Teams onboarding process
Once payment is processed, you’ll be greeted with the welcome page and can check your Account Billing section to check that your on the correct Cloudflare For Teams plan subscription.
CSF Firewall
This is a one time task. Place in /etc/csf/csf.allow
allow file whitelisting for Cloudflare route1/2 hostname’s IP addresses to allow egress TCP traffic on destination port 7844 as per Cloudflare Argo Tunnel FAQ documentation.
First command backs up /etc/csf/csf.allow
and then appends to csf.allow the CSF Firewall allow list to CF Argo Tunnel IPs for destination port 7844 and port 443.
cp -a /etc/csf/csf.allow /etc/csf/csf.allow.backup.before-argo-tunnel cat >> /etc/csf/csf.allow << EOF tcp|out|d=7844|d=198.41.192.7 tcp|out|d=7844|d=198.41.192.47 tcp|out|d=7844|d=198.41.192.107 tcp|out|d=7844|d=198.41.192.167 tcp|out|d=7844|d=198.41.192.227 tcp|out|d=7844|d=198.41.200.193 tcp|out|d=7844|d=198.41.200.233 tcp|out|d=7844|d=198.41.200.13 tcp|out|d=7844|d=198.41.200.53 tcp|out|d=7844|d=198.41.200.113 tcp|out|d=443|d=104.19.193.29 tcp|out|d=443|d=104.19.192.29 EOF
restart CSF Firewall
csf -ra
Create Centmin Mod Nginx Vhost Sites
You can use Centmin Mod’s centmin.sh menu option 2, 22 or nv command line to create Nginx vhost sites if they haven’t already been created. Examples including enabling free Letsencrypt SSL certificate support during Nginx vhost site creation here.
touch /etc/centminmod/custom_config.inc echo "LETSENCRYPT_DETECT='y'" >> /etc/centminmod/custom_config.inc
nv command line
nv Usage: /usr/bin/nv [-d yourdomain.com] [-s y|n|yd|le|led|lelive|lelived] [-u ftpusername] -d yourdomain.com or subdomain.yourdomain.com -s ssl self-signed create = y or n or https only vhost = yd -s le - letsencrypt test cert or led test cert with https default -s lelive - letsencrypt live cert or lelived live cert with https default -u your FTP username example: /usr/bin/nv -d yourdomain.com -s y -u ftpusername /usr/bin/nv -d yourdomain.com -s n -u ftpusername /usr/bin/nv -d yourdomain.com -s yd -u ftpusername /usr/bin/nv -d yourdomain.com -s le -u ftpusername /usr/bin/nv -d yourdomain.com -s led -u ftpusername /usr/bin/nv -d yourdomain.com -s lelive -u ftpusername /usr/bin/nv -d yourdomain.com -s lelived -u ftpusername
create self-signed SSL certificate with non-HTTPS + HTTPS vhosts
/usr/bin/nv -d tun.domain.com -s y -u ftpusername
create self-signed SSL certificate with HTTPS only vhosts
/usr/bin/nv -d tun.domain.com -s yd -u ftpusername
create free Letsencrypt SSL certificate with non-HTTPS + HTTPS vhosts
/usr/bin/nv -d tun.domain.com -s lelive -u ftpusername
create free Letsencrypt SSL certificate with HTTPS only vhosts
/usr/bin/nv -d tun.domain.com -s lelived -u ftpusername
Manual Argo Tunnel Setup with cloudflared
You can manually using cloudflared binary to setup an Argo Tunnel
Step 1. Install cloudflared Binary
You can download latest cloudflared
binary from here.
wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared chmod +x /usr/local/bin/cloudflared cloudflared update
Authenticate cloudflared using your Cloudflare Account log as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/setup
cloudflared tunnel login
Verify cloudflared version installed
cloudflared --version cloudflared version 2021.2.1 (built 2021-02-04-1528 UTC)
Step 2. Create Argo Tunnel
- You can name your Argo Tunnels whatever you want, here we’ll use the intended hostname/subdomain as the name of the Argo Tunnel.
- Pipe the tunnel create command output into a log file named
$hostname-tunnel-create.log
so can use the log file to find the tunnel id and credential file needed to install cloudflared as a service and assign them to variables. - You’ll need to have
jq
installed which is installed by default on Centmin Mod LEMP stacks. If not you can install it viayum -y install jq
.
hostname=tun.domain.com cloudflared tunnel create $hostname | tee $hostname-tunnel-create.log tunnelid=$(cloudflared tunnel list -o json | jq -r --arg h $hostname '.[] | select(.name == $h) | .id') credfile="/root/.cloudflared/${tunnelid}.json"
Step 3. Creating cloudflared YAML Config File
Create the cloudflared YAML config file at ~/.cloudflared/config.yml
by populating the variables below. This configuration uses Cloudflare Named Tunnels and Ingress Rules. Updated: October 14, 2021 to add originServerName to originRequest
section.
hostname=tun.domain.com localhost=https://localhost:443 metrics=localhost:5432 cftag='cmm=test' cfpid='/var/run/cmm-test-argo.pid' cfupdatettl='24h'
Make a copy of generated cert.pem
file as /root/.cloudflared/cert-${hostname}.pem
which will be used for origincert configuration value in tunnel’s YAML config file.
cp -a /root/.cloudflared/cert.pem /root/.cloudflared/cert-${hostname}.pem
Creating ~/.cloudflared/config.yml
file with variables you assigned. Updated: October 21, 2021 – Cloudflare now supports Cloudflare Tunnel connections over QUIC protocol via a UDP listener on the Cloudflare edge server side. So you can now choose between protocol: http2
or protocol: quic
in your configuration file. See the protocol argument for configuring Cloudflare Tunnel. If using protocol: quic
, ensure your system default UDP receive buffer size is optimally configured.
cat > ~/.cloudflared/config.yml <<EOF tunnel: $tunnelid credentials-file: $credfile origincert: /root/.cloudflared/cert-${hostname}.pem protocol: http2 originRequest: connectTimeout: 30s metrics: $metrics #tag: $cftag pidfile: $cfpid autoupdate-freq: $cfupdatettl loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: $hostname service: $localhost originRequest: connectTimeout: 10s originServerName: $hostname noTLSVerify: true - service: http_status:404 EOF
The resulting ~/.cloudflared/config.yml
file will contain
tunnel: your_tunnelid credentials-file: /root/.cloudflared/your_tunnelid.json origincert: /root/.cloudflared/cert-tun.domain.com.pem protocol: http2 originRequest: connectTimeout: 30s metrics: localhost:5432 #tag: cmm=test pidfile: /var/run/cmm-test-argo.pid autoupdate-freq: 24h loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: tun.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: tun.domain.com noTLSVerify: true - service: http_status:404
On SystemD based system like CentOS 7, you can query your cloudflared
daemon logs using journalctl
.
journalctl -u cloudflared --no-pager | sed -e "s|$(hostname)|hostname|g"
For instance, you can verify the location of your working service config.xml
file and which protocol cloudflared
daemon is connecting to Cloudflare edge servers with – either HTTP/2
or QUIC
.
Oct 20 17:00:12 hostname systemd[1]: Stopped Argo Tunnel. Oct 20 17:00:12 hostname systemd[1]: Starting Argo Tunnel... Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Starting tunnel tunnelID=bbcb1xxx-3a7b-4f69-914c-xxx Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Version 2021.10.3 Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF GOOS: linux, GOVersion: devel +a84af465cb Mon Aug 9 10:31:00 2021 -0700, GoArch: amd64 Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Settings: map[autoupdate-freq:24h0m0s config:/etc/cloudflared/config.yml Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Generated Connector ID: 062e30ae-c7a9-xxxx-b7e2-xxxx Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Initial protocol quic
With Argo Tunnel Ingress rules, you can setup other Cloudflare domain zone sites within the same Cloudflare Account to go through this single Cloudflare Argo Tunnel by adding the additional domain/subdomains i.e. hostname.domain.com and host.example.com as CNAME DNS records pointing to the same tunnelid.cfargotunnel.com
target. Then update ~/.cloudflared/config.yml
config file to:
tunnel: your_tunnelid credentials-file: /root/.cloudflared/your_tunnelid.json origincert: /root/.cloudflared/cert-tun.domain.com.pem protocol: http2 originRequest: connectTimeout: 30s metrics: localhost:5432 #tag: cmm=blog pidfile: /var/run/cmm-test-argo.pid autoupdate-freq: 24h loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: tun.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: tun.domain.com noTLSVerify: true - hostname: hostname.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: hostname.domain.com noTLSVerify: true - hostname: host.example.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: host.example.com noTLSVerify: true - service: http_status:404
I commented out the tag:
setting for now as manually running the cloudflared argo tunnel gives an error right now. Update: it seems the tag argument currently isn’t meant to do anything.
cloudflared tunnel --config ~/.cloudflared/config.yml run tun.domain.com expected string slice found string for tag
Validate ingress rules
cloudflared tunnel ingress validate Validating rules from /root/.cloudflared/config.yml OK
Step 4. Create Cloudflare CNAME DNS Record To Route Argo Tunnel
As per documentation https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/routing-to-tunnel/dns, you can create the CNAME DNS record via command line. This will only work for the Cloudflare site zone that you authenticated the initial cloudflared login setup for in Step 1. Other Cloudflare site zones you intend to add to the Argo Tunnel will have to have their CNAME DNS records added either manually or via Cloudflare DNS API.
cloudflared tunnel route dns <UUID or NAME> www.app.com
In this case
cloudflared tunnel route dns tun.domain.com tun.domain.com
or using $tunnelid
variable you populated in step 3 above.
cloudflared tunnel route dns $tunnelid tun.domain.com
This will then create the CNAME DNS record for tun.domain.com
to point to tunnelid.cfargotunnel.com
target if the CNAME doesn’t already exist. If the CNAME already exists, you’ll get an error and need to manually edit and update the existing CNAME DNS record.
Step 5. Install cloudflared Service on CentOS 7
Install cloudflared service, start it and ensure it starts on server reboots. Running service install command will create and configure systemd service file at /etc/systemd/system/cloudflared.service
and seems the config file at ~/.cloudflared/config.yml
is copied and read from /etc/cloudflared/config.yml
afterwards.
[Unit] Description=Argo Tunnel After=network.target [Service] TimeoutStartSec=0 Type=notify ExecStart=/usr/local/bin/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate tunnel run Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target
You can check the cloudflared log file at /var/log/cloudflared.log
.
cloudflared service install service cloudflared start chkconfig cloudflared on
Check status
systemctl status cloudflared.service
or
service cloudflared status
If you already had previously installed cloudflared service, you’ll get an error message:
cloudflared service install Possible conflicting configuration in /root/.cloudflared/config.yml and /etc/cloudflared/config.yml. Either remove /etc/cloudflared/config.yml or run `cloudflared --config /etc/cloudflared/config.yml service install`
Then start and enable services to auto update the cloudflared binary.
systemctl start cloudflared-update.timer systemctl enable cloudflared-update.timer systemctl status cloudflared-update.timer
You can then run the systemctl list-timers
command to list when cloudflared
will next update:
systemctl list-timers cloudflared-update.timer --all NEXT LEFT LAST PASSED UNIT ACTIVATES Tue 2021-10-26 00:00:00 UTC 4h 20min left n/a n/a cloudflared-update.timer cloudflared-update.service
Automated Argo Tunnel Setup with Cloudflare API
Argo Tunnels can also be created via Cloudflare API. This below outlined steps could be scripted for automatic Cloudflare Argo Tunnel creation and setup.
Step 1. Create Cloudflare API Token with Argo Tunnel Write (Edit) Permission
Create a Cloudflare API Token with write permissions = Edit at the Cloudflare account level and DNS edit permissions at zone level.
If you forget or lose your generated Cloudflare API Token, you can go to API Token listing page and in menu drop down (3 dots) link, select Roll to regenarate a new Cloudflare API Token.
Step 2. Install cloudflared
If you have yet to install and authenticate cloudflared, you can do the one time task and install it via curl:
curl -4s https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.tgz | tar xzC /usr/local/bin cloudflared update
Authenticate cloudflared using your Cloudflare Account log as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/setup
cloudflared tunnel login
Verify cloudflared version installed
cloudflared --version cloudflared version 2021.2.1 (built 2021-02-04-1528 UTC)
Step 3. Create Argo Tunnel
Create Argo Tunnel via Cloudflare API with session variables populated where tunnelname
is your desired Argo Tunnel hostname/subdomain.com and Cloudflare Account Id and Coudflare Zone ID are available on your Cloudflare site’s zone overview page right column and Cloudflare API Token is the one you generated from previous step. The Cloudflare API curl command output is saved to $tunnelname-tunnel-create.log
log so we can get the Argo Tunnel’s tunnel id.
cfaccountid='your_cf_account_id' cfapitoken='your_cf_api_token' cfzoneid='your_cf_zone_id' # tunnel name = hostname tunnelname=tun.domain.com tunnelsecret=$(openssl rand -hex 16 | base64) curl -4sX POST "https://api.cloudflare.com/client/v4/accounts/$cfaccountid/tunnels" \ -H "Authorization: Bearer $cfapitoken" \ -H "Content-Type: application/json" \ --data "{\"name\":\"$tunnelname\",\"tunnel_secret\":\"$tunnelsecret\"}" | jq | tee $tunnelname-tunnel-create.log
command output:
{ "id": "34c4cadb-2edc-47ac-a682-xxxxxxxx", "created_at": "2021-02-08T17:41:25.125890Z", "deleted_at": null, "name": "tun.domain.com", "connections": [] }
Get tunnel id from $tunnelname-tunnel-create.log
log
tunnelid=$(jq -r '.id' $tunnelname-tunnel-create.log)
Step 4. Create Argo Tunnel CNAME DNS Record
tunnelid=$(jq -r '.id' $tunnelname-tunnel-create.log) curl -4sX POST "https://api.cloudflare.com/client/v4/zones/$cfzoneid/dns_records" \ -H "Authorization: Bearer $cfapitoken" \ -H "Content-Type: application/json" \ --data "{\"type\":\"CNAME\",\"name\":\"$tunnelname\",\"content\":\"$tunnelid.cfargotunnel.com\",\"proxied\":true}" | jq | tee $tunnelname-tunnel-cname.log
command output:
{ "result": { "id": "9b848b83cc6dfb46169cf8a9a9d9446b", "zone_id": "your_cf_zone_id", "zone_name": "domain.com", "name": "tun2.domain.com", "type": "CNAME", "content": "34c4cadb-2edc-47ac-a682-xxxxxxxx.cfargotunnel.com", "proxiable": true, "proxied": true, "ttl": 1, "locked": false, "meta": { "auto_added": false, "managed_by_apps": false, "managed_by_argo_tunnel": false, "source": "primary" }, "created_on": "2021-02-08T18:04:17.990511Z", "modified_on": "2021-02-08T18:04:17.990511Z" }, "success": true, "errors": [], "messages": [] }
This will then create the CNAME DNS record for tun2.domain.com
to point to tunnelid.cfargotunnel.com
target if the CNAME doesn’t already exist. If the CNAME already exists, you’ll get an error and need to manually edit and update the existing CNAME DNS record.
Step 5. Create Argo Tunnel Credentials JSON File
echo "{ \"AccountTag\": \"$cfaccountid\", \"TunnelSecret\": \"$tunnelsecret\", \"TunnelID\": \"$tunnelid\", \"TunnelName\": \"$tunnelname\" }" | jq -c | tee ~/.cloudflared/$tunnelid.json
contents of ~/.cloudflared/$tunnelid.json
cat ~/.cloudflared/$tunnelid.json | jq { "AccountTag": "your_cf_account_id", "TunnelSecret": "ZTdkYTRlZmJjZDRmNDRlZjRiYzFjN2UxNGI0NzgyMTIK", "TunnelID": "34c4cadb-2edc-47ac-a682-xxxxxxxx", "TunnelName": "tun2.domain.com" }
Step 6. Create Argo Tunnel YAML Config File
Create the cloudflared YAML config file at ~/.cloudflared/config.yml
by populating the variables below. This configuration uses Cloudflare Named Tunnels and Ingress Rules. Updated: October 14, 2021 to add originServerName to originRequest
section.
hostname=tun2.domain.com credfile="/root/.cloudflared/${tunnelid}.json" localhost=https://localhost:443 metrics=localhost:5432 cftag='cmm=test' cfpid='/var/run/cmm-test-argo.pid' cfupdatettl='24h'
Make a copy of generated cert.pem
file as /root/.cloudflared/cert-${hostname}.pem
which will be used for origincert configuration value in tunnel’s YAML config file.
cp -a /root/.cloudflared/cert.pem /root/.cloudflared/cert-${hostname}.pem
Creating ~/.cloudflared/config.yml
file with variables you assigned. Updated: October 21, 2021 – Cloudflare now supports Cloudflare Tunnel connections over QUIC protocol via a UDP listener on the Cloudflare edge server side. So you can now choose between protocol: http2
or protocol: quic
in your configuration file. See the protocol argument for configuring Cloudflare Tunnel. If using protocol: quic
, ensure your system default UDP receive buffer size is optimally configured.
cat > ~/.cloudflared/config.yml <<EOF tunnel: $tunnelid credentials-file: $credfile origincert: /root/.cloudflared/cert-${hostname}.pem protocol: http2 originRequest: connectTimeout: 30s metrics: $metrics #tag: $cftag pidfile: $cfpid autoupdate-freq: $cfupdatettl loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: $hostname service: $localhost originRequest: connectTimeout: 10s originServerName: $hostname noTLSVerify: true - service: http_status:404 EOF
The resulting ~/.cloudflared/config.yml
file will contain
tunnel: your_tunnelid credentials-file: /root/.cloudflared/your_tunnelid.json origincert: /root/.cloudflared/cert-tun.domain.com.pem protocol: http2 originRequest: connectTimeout: 30s metrics: localhost:5432 #tag: cmm=test pidfile: /var/run/cmm-test-argo.pid autoupdate-freq: 24h loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: tun2.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: tun2.domain.com noTLSVerify: true - service: http_status:404
On SystemD based system like CentOS 7, you can query your cloudflared
daemon logs using journalctl
.
journalctl -u cloudflared --no-pager | sed -e "s|$(hostname)|hostname|g"
For instance, you can verify the location of your working service config.xml
file and which protocol cloudflared
daemon is connecting to Cloudflare edge servers with – either HTTP/2
or QUIC
.
Oct 20 17:00:12 hostname systemd[1]: Stopped Argo Tunnel. Oct 20 17:00:12 hostname systemd[1]: Starting Argo Tunnel... Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Starting tunnel tunnelID=bbcb1xxx-3a7b-4f69-914c-xxx Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Version 2021.10.3 Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF GOOS: linux, GOVersion: devel +a84af465cb Mon Aug 9 10:31:00 2021 -0700, GoArch: amd64 Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Settings: map[autoupdate-freq:24h0m0s config:/etc/cloudflared/config.yml Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Generated Connector ID: 062e30ae-c7a9-xxxx-b7e2-xxxx Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Initial protocol quic
With Argo Tunnel Ingress rules, you can setup other Cloudflare domain zone sites within the same Cloudflare Account to go through this single Cloudflare Argo Tunnel by adding the additional domain/subdomains i.e. hostname.domain.com and host.example.com as CNAME DNS records pointing to the same tunnelid.cfargotunnel.com
target. Then update ~/.cloudflared/config.yml
config file to:
tunnel: your_tunnelid credentials-file: /root/.cloudflared/your_tunnelid.json origincert: /root/.cloudflared/cert-tun2.domain.com.pem protocol: http2 originRequest: connectTimeout: 30s metrics: localhost:5432 #tag: cmm=blog pidfile: /var/run/cmm-test-argo.pid autoupdate-freq: 24h loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: tun2.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: tun2.domain.com noTLSVerify: true - hostname: hostname.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: hostname.domain.com noTLSVerify: true - hostname: host.example.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: host.example.com noTLSVerify: true - service: http_status:404
I commented out the tag:
setting for now as manually running the cloudflared argo tunnel gives an error right now. Update: it seems the tag argument currently isn’t meant to do anything.
cloudflared tunnel --config ~/.cloudflared/config.yml run tun.domain.com expected string slice found string for tag
Validate ingress rules
cloudflared tunnel ingress validate Validating rules from /root/.cloudflared/config.yml OK
Step 7. Install cloudflared Service
Install cloudflared service, start it and ensure it starts on server reboots. Running service install command will create and configure systemd service file at /etc/systemd/system/cloudflared.service
and seems the config file at ~/.cloudflared/config.yml
is copied and read from /etc/cloudflared/config.yml
afterwards.
[Unit] Description=Argo Tunnel After=network.target [Service] TimeoutStartSec=0 Type=notify ExecStart=/usr/local/bin/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate tunnel run Restart=on-failure RestartSec=5s [Install] WantedBy=multi-user.target
You can check the cloudflared log file at /var/log/cloudflared.log
.
cloudflared service install service cloudflared start chkconfig cloudflared on
Check status
systemctl status cloudflared.service
or
service cloudflared status
If you already had previously installed cloudflared service, you’ll get an error message:
cloudflared service install Possible conflicting configuration in /root/.cloudflared/config.yml and /etc/cloudflared/config.yml. Either remove /etc/cloudflared/config.yml or run `cloudflared --config /etc/cloudflared/config.yml service install`
Then start and enable services to auto update the cloudflared binary.
systemctl start cloudflared-update.timer systemctl enable cloudflared-update.timer systemctl status cloudflared-update.timer
You can then run the systemctl list-timers
command to list when cloudflared
will next update:
systemctl list-timers cloudflared-update.timer --all NEXT LEFT LAST PASSED UNIT ACTIVATES Tue 2021-10-26 00:00:00 UTC 4h 20min left n/a n/a cloudflared-update.timer cloudflared-update.service
Cloudflare Zero Trust Terminal Browser
Cloudflare updated Cloudflare Tunnels to support Cloudflare Zero Trust Terminal Browser access allowing you to utilise Cloudflare For Team’s Access to create a secure browser rendered SSH terminal to connect to your servers. The instructions are documented here.
To continue with the above examples of an Argo Tunnel Ingress rules multi host/origin configuration with ~/.cloudflared/config.yml
config file modified to add the following:
- hostname: yourssh.domain.com service: ssh://localhost:22
setting up your desired browser SSH terminal hostname as yourssh.domain.com
which would be accessed at https://yourssh.domain.com
so that the full ~/.cloudflared/config.yml
config file looks like below example:
tunnel: your_tunnelid credentials-file: /root/.cloudflared/your_tunnelid.json origincert: /root/.cloudflared/cert-tun2.domain.com.pem protocol: http2 originRequest: connectTimeout: 30s metrics: localhost:5432 #tag: cmm=blog pidfile: /var/run/cmm-test-argo.pid autoupdate-freq: 24h loglevel: info logfile: /var/log/cloudflared.log ingress: - hostname: tun2.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: tun2.domain.com noTLSVerify: true - hostname: hostname.domain.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: hostname.domain.com noTLSVerify: true - hostname: host.example.com service: https://localhost:443 originRequest: connectTimeout: 10s originServerName: host.example.com noTLSVerify: true - hostname: yourssh.domain.com service: ssh://localhost:22 - service: http_status:404
Then manually setup the Cloudflare CNAME DNS record, yourssh.domain.com
pointing to the same tunnelid.cfargotunnel.com
target you initially created.
Then restart cloudflared service
service cloudflared restart
Then log into your Cloudflare For Teams dashboard and go to Cloudflare Access section to set up a self-hosted app as outlined here and here (skip advanced step 3 and only do step 1 for setup of Self Hosted App and step 2 to add a policy to control who accesses your app at https://yourssh.domain.com
).
The key setting for Zero Trust browser terminals is to enable a new setting for Enable browser rendering:
Then accessing the browser terminal at https://yourssh.domain.com
where you’d be greeted with your Cloudflare Access policy for login and be prompted to enter username and password for your SSH server or use shorted lived SSH certificates for Public Key Authentication.
Using Github as an authentication method to access the browser terminal URL.
SSH username and password prompts
Once logged in, you’re able to access the browser terminal for the server.
Listing Argo Tunnels Created
Above method used via cloudflared
or Cloudflare API will not list the created Argo Tunnel in Cloudflare dashboard’s Traffic > Argo Tunnel section in web GUI. Update October 19, 2021: Cloudflare For Teams GUI dashboard now allows you to list the Cloudflare Tunnels you have created using the above method.
To list your created Argo Tunnels use either command:
cloudflared tunnel list
or
cloudflared tunnel list -o json
Or via the new Cloudflare For Teams Access Tunnel GUI dashboard:
Centmin Mod Nginx Access Logs
Argo Tunnel by default won’t pass on the visitor’s real IP address to Centmin Mod Nginx. Instead it will show up like
tail -1 /home/nginx/domains/tun.domain.com/log/access.log 127.0.0.1 - - [07/Feb/2021:22:39:12 +0000] "GET /?test HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36 OPR/74.0.3911.75"
Centmin Mod Nginx config file at /usr/local/nginx/conf/nginx.conf
has additional Nginx log format options which can log Argo Tunnel received visitor’s real IP addresses via $http_x_forwarded_for
field.
Example for log format named cf_custom4
log_format cf_custom4 '$remote_addr - $remote_user [$time_local] $request ' '"$status" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio" "$brotli_ratio"' ' "$connection" "$connection_requests" "$request_time" $http_cf_ray ' '$ssl_protocol $ssl_cipher $http_content_length $http_content_encoding $request_length';
You can alter and create your own custom log formats too.
To enable this, you need to edit Nginx vhost config files include file /usr/local/nginx/conf/cloudflare.conf
in /usr/local/nginx/conf/conf.d/tun.domain.com.ssl.conf
and/or /usr/local/nginx/conf/conf.d/tun.domain.com.conf
. This enables Cloudflare’s documented restoration of real visitor IP addresses.
# uncomment cloudflare.conf include if using cloudflare for # server and/or vhost site include /usr/local/nginx/conf/cloudflare.conf;
And also add a second access log line assigning the log format named cf_custom4
access_log /home/nginx/domains/tun.domain.com/log/cf-ssl-access.log cf_custom4;
to existing one so it looks like
access_log /home/nginx/domains/tun.domain.com/log/access.log combined buffer=256k flush=5m; error_log /home/nginx/domains/tun.domain.com/log/error.log; access_log /home/nginx/domains/tun.domain.com/log/cf-access.log cf_custom4;
Then restart Nginx
service nginx restart
Or via Centmin Mod command shortcut
ngxrestart
Visit Argo Tunnel site and then check the new access log and see the $http_x_forwarded_for
field record the real visitor IP = 122.xxx.xxx.xxx
tail -1 /home/nginx/domains/tun.domain.com/log/cf-access.log 127.0.0.1 - - [07/Feb/2021:22:39:12 +0000] GET /?test HTTP/1.1 "200" 2112 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36 OPR/74.0.3911.75" "122.xxx.xxx.xxx" "3.14" "-" "54043" "3" "0.000" 61f323cc591c32a4-ORD TLSv1.3 TLS_AES_256_GCM_SHA384 - - 1303
Cloudflared Log Rotation
The above cloudflared config.yml
configuration file specifies logging at logfile: /var/log/cloudflared.log
. To manage the log size, you’d also need to setup log rotation profile at /etc/logrotate.d/cloudflared
containing the follow:
/var/log/cloudflared.log { daily dateext missingok rotate 8 maxsize 100M compress delaycompress copytruncate notifempty missingok }
Then the /etc/logrotate.d/cloudflared
will rotate daily automatically.
Debug logrotate run:
logrotate -d /etc/logrotate.d/cloudflared reading config file /etc/logrotate.d/cloudflared Allocating hash table for state file, size 15360 B Handling 1 logs rotating pattern: /var/log/cloudflared.log after 1 days (8 rotations) empty log files are not rotated, log files >= 104857600 are rotated earlier, old logs are removed considering log /var/log/cloudflared.log log needs rotating rotating log /var/log/cloudflared.log, log->rotateCount is 8 dateext suffix '-20211021' glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]' glob finding logs to compress failed glob finding old rotated logs failed copying /var/log/cloudflared.log to /var/log/cloudflared.log-20211021 truncating /var/log/cloudflared.log
Forced a manual logrotate run:
logrotate -f /etc/logrotate.d/cloudflared
Then check your /var/log directory
ls -lahrt /var/log | grep cloudflared -rw-r--r-- 1 root root 435M Oct 21 22:54 cloudflared.log-20211021 -rw-r--r-- 1 root root 0 Oct 21 22:54 cloudflared.log
Cloudflared System Settings
Cloudflare Tunnel’s cloudflared
daemon defaults to http2
protocol persistent connections. However, it can now be manually switched to using quic
protocol persistent connections via it’s config.yml
config file setup outlined above. cloudflared
uses quic-go library it seems and wants to use a UDP receive buffer size of 2MB. On Centmin Mod LEMP stack systems this is not a problem as initial installs of Centmin Mod already increase the buffer size out the box.
// DesiredReceiveBufferSize is the kernel UDP receive buffer size that we'd like to use. const DesiredReceiveBufferSize = (1 << 20) * 2 // 2 MB
However, on non-Centmin Mod LEMP systems, you may need to raise your default UDP receive buffer size as outlined at https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for more optimal transfer performance when using for your Cloudflare Tunnel when configured to use quic
protocol persistent connections. Otherwise, you may encounter this error where 208 kiB is usually the default Linux receive buffer size which via SO_RCVBUF gets doubled by the Linux Kernel to 416 kiB. But it isn’t enough as quic-go wants to ideally use 2MB.
failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Cloudflare Authenticated Origin Pull Incompatibility
Cloudflare Argo Tunnel is incompatible with Cloudflare Authenticated Origin Pull as they both do the same thing to prevent visitor access that bypasses Cloudflare proxy and tries HTTP access via the server’s real IP access – just done in different ways. So if you have manually enabled Cloudflare Authenticated Origin Pull certificate configuration in your Nginx vhost config file at /usr/local/nginx/conf/conf.d/yourdomain.com.conf
and/or /usr/local/nginx/conf/conf.d/yourdomain.com.ssl.conf
, you will need to disable them again by commenting out the relevant lines using a hash #
in front of the 2nd and 3rd lines below and then restart Nginx service. By default Cloudflare Authenticated Origin Pull is disabled with a hash in front usually.
# cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/ # ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/yourdomain.com/origin.crt; # ssl_verify_client on;
Cloudflare Restrict IPs
The final step once Cloudflare Tunnels is working is to restrict HTTP (port 80) and HTTPS (port 443) access on your origin server to just Cloudflare client/edge server requests so that no other non-Cloudflare requests can reach your origin server. You do this by restricting at origin server firewall level only requests made by Cloudflare servers. As this guide is for Centmin Mod LEMP stack users who use CSF Firewall, then to do this step just remove port 80
and 443
from CSF Firewall’s /etc/csf/csf.conf
(backup file before editing) config file for TCP_IN
and TCP6_IN
comma separated list and then restart CSF Firewall.
csf -ra
Centmin Mod initial install and above CSF Firewall whitelist configuration would have already taken care of steps to allow Cloudflare only requests at CSF Firewall level. Removing the port 80
and 443
from CSF Firewall’s /etc/csf/csf.conf
config file would take care of the remaining step to disallow non-Cloudflare requests from hitting the origin server on those ports.
If you need to reverse this change, just re-add port 80
and 443
to CSF Firewall’s /etc/csf/csf.conf
settings for TCP_IN
and TCP6_IN
comma separated list and then restart CSF Firewall again.
If you’re using a non-Centmin Mod or non-CSF Firewall setup, then you’d have to deal with iptables based restrictions outlined by the Cloudflare support article here.
Cloudflare Tunel http2 vs quic protocol connections
Cloudflare Tunnels now support either HTTP/2 or QUIC protocol based persistent connections from Cloudflare edge server to local origin cloudflared daemon instance which sits as a reverse proxy in front of your local origin server.
I ran some loader.io load tests comparing Cloudflare Tunnel protocol connection methods http2
vs quic
for both Cloudflare CDN cached and also for Cloudflare CDN bypass cached configurations. As you can see cache bypassed Cloudflare Tunnel with QUIC protocol based persistent connections did slightly better than with the default HTTP/2 protocol based persistent connections with ~16.6% better average response times.
Below are the configurations tested:
- CF Tunnel
http2
protocol – CF Cache Bypassed – 500 users with HTTP gzip compressed requests – cache bypassed means testing origin server and thus the Cloudflare edge server to local cloudflared daemon instance connection. - CF Tunnel
quic
protocol – CF Cache Bypassed – 500 users with HTTP gzip compressed requests – cache bypassed means testing origin server and thus the Cloudflare edge server to local cloudflared daemon instance connection. - CF Tunnel
http2
protocol – CF Cached – 1000 users with HTTP gzip compressed requests – cached means testing Cloudflare CDN Cache - CF Tunnel
quic
protocol – CF Cached – 1000 users with HTTP gzip compressed requests – cached means testing Cloudflare CDN Cache
This WordPress blog is hosted in US West Coast, San Jose on a US$5/month – 1 CPU, 1GB memory based KVM VPS server at Upcloud.com running my optimized Centmin Mod LEMP stack on CentOS 7 64bit OS with fully optimized full HTML page caching at Centmin Mod Nginx level and origin server PHP-FPM is compiled with Profile Guide Optimization training for specific WordPress PHP usage loads to further boost PHP performance (PHP 8 vs 7 with/without PGO).
lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 1 On-line CPU(s) list: 0 Thread(s) per core: 1 Core(s) per socket: 1 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 85 Model name: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz Stepping: 4 CPU MHz: 2992.968 BogoMIPS: 5985.93 Hypervisor vendor: KVM Virtualization type: full L1d cache: 32K L1i cache: 32K L2 cache: 4096K L3 cache: 16384K NUMA node0 CPU(s): 0 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat umip pku ospke md_clear spec_ctrl intel_stibp
Tabulated results:
CF Tunnel http2 protocol – CF Cache Bypassed – 500 users
CF Tunnel quic protocol – CF Cache Bypassed – 500 users
CF Tunnel http2 protocol – CF Cached – 1000 users
CF Tunnel quic protocol – CF Cached – 1000 users
The origin WordPress server’s CPU and memory usage statistics for both Tunnel HTTP/2 and QUIC protocol persistent connections. The HTTP/2 is the higher CPU usage peak and the QUIC UDP based protocol connection test is the lower CPU usage peak.