How To Setup Cloudflare Argo Tunnel On CentOS 7

Cloudflare Argo Tunnel allows you to expose your web server to the internet without having to open routes in your firewall or setup dedicated routes. This guide will outline how you can setup Cloudflare Argo Tunnel for private encrypted HTTP/2 connection to your origin web server or web application on CentOS 7 for Centmin Mod LEMP stack users.

The below instructions for setting up a Cloudflare Argo Tunnel HTTP/2 or QUIC UDP based connection to a self hosted application/origin server is tailored to Centmin Mod LEMP stack users running CentOS 7 running CSF Firewall. The guide will be setting up Cloudflare Argo Tunnels using Named Tunnels and a Ingress Rule based configuration which allow you map traffic for multiple hostnames and services within a single Argo Tunnel.

Cloudflare Argo Tunnels Now Cloudflare Tunnels

Update as of April 15, 2021, Cloudflare Argo Tunnels has been renamed Cloudflare Tunnels to reflect that Cloudflare Tunnels are now free to all Cloudflare users regardless of their Cloudflare subscription plan. Personally, I’d still recommend you sign up for Cloudflare For Teams free subscription as that offering provides more than just Cloudflare Tunnels which you can advantage of.

What Is Cloudflare Argo Tunnel

Cloudflare Argo Tunnel is a private encrypted HTTP/2 or QUIC UDP based connection between your web server and Cloudflare which makes it so that only traffic that routes through Cloudflare can reach your server or web application without needing a publicly routable IP address.

An example request follows these steps:

  • A visitor makes a request to tunnel.yourdomain.com.
  • The DNS lookup resolves to a Cloudflare network address.
  • The visitor connects to the closest Cloudflare edge PoP via Anycast.
  • Cloudflare routes the visitor through a special PoP-to-PoP route called Argo Smart Routing and connects them to a Cloudflare edge PoP that has an established persistent connection to the daemon (cloudflared) running on the visitor’s web server.
  • The request is routed to the cloudflared instance running on your server. The connection between cloudflared and the Cloudflare edge is a long-lived persistent HTTP/2 or QUIC UDP configurable connection encrypted with TLS. To keep the connection alive, cloudflared sends a heartbeat to the edge in the form of a ping frame over HTTP/2. If the connection is dropped, the cloudflared client re-establishes the connection with Cloudflare. cloudflared connects to Cloudflare on port 7844. All packets between Cloudflare and the tunneled web server use stream multiplexing over HTTP/2 or QUIC UDP. In HTTP/2, each request/response pair is called a Stream and given a unique Stream ID so that these streams can be “multiplexed” or sent asynchronously over the same connection. QUIC UDP improves on HTTP/2 connections by solving Head of Line (HoL) blocking as well making it possible for Cloudflare to support cloudflared to cloudflared communication.
  • The Argo Tunnel client forwards the request to your web service.

Cloudflare Argo Tunnel Connection

Contents

This guide will outline both manual (with cloudflared) and automated methods (via Cloudflare API with Cloudflare API Tokens) and is based on official Cloudflare documentation and Cloudflare Argo Tunnel FAQ documentation.

Sign Up For Cloudflare For Teams Free Subscription

The easiest way for your Cloudflare Account to get access to free Argo Tunnel feature is to sign up for Cloudflare For Teams free subscription which bundles Cloudflare Gateway, Cloudflare Access, Cloudflare Browser Isolation and Cloudflare Argo Tunnels together and is available as add-ons to your existing Cloudflare subscription. The official Cloudflare For Teams documentation is available here.

To sign up for the free Cloudflare For Teams subscription, go to the Cloudflare Team dashboard link at https://dash.teams.cloudflare.com/ and log into your Cloudflare Account and go through the Cloudflare For Teams onboarding process. You’d want to ensure your Cloudflare account’s billing details for credit card and billing address are updated if you choose a paid subscription instead of a free one.

Cloudflare Billing

Then go through the Cloudflare For Teams onboarding process

Cloudflare For Teams onboarding process

Cloudflare For Teams onboarding process

Cloudflare For Teams onboarding process

Cloudflare For Teams onboarding process

Once payment is processed, you’ll be greeted with the welcome page and can check your Account Billing section to check that your on the correct Cloudflare For Teams plan subscription.

Cloudflare For Teams onboarded

Cloudflare For Teams Billing page

CSF Firewall

This is a one time task. Place in /etc/csf/csf.allow allow file whitelisting for Cloudflare route1/2 hostname’s IP addresses to allow egress TCP traffic on destination port 7844 as per Cloudflare Argo Tunnel FAQ documentation.

First command backs up /etc/csf/csf.allow and then appends to csf.allow the CSF Firewall allow list to CF Argo Tunnel IPs for destination port 7844 and port 443.

cp -a /etc/csf/csf.allow /etc/csf/csf.allow.backup.before-argo-tunnel
cat >> /etc/csf/csf.allow << EOF
tcp|out|d=7844|d=198.41.192.7
tcp|out|d=7844|d=198.41.192.47
tcp|out|d=7844|d=198.41.192.107
tcp|out|d=7844|d=198.41.192.167
tcp|out|d=7844|d=198.41.192.227
tcp|out|d=7844|d=198.41.200.193
tcp|out|d=7844|d=198.41.200.233
tcp|out|d=7844|d=198.41.200.13
tcp|out|d=7844|d=198.41.200.53
tcp|out|d=7844|d=198.41.200.113
tcp|out|d=443|d=104.19.193.29
tcp|out|d=443|d=104.19.192.29
EOF

restart CSF Firewall

csf -ra

Create Centmin Mod Nginx Vhost Sites

You can use Centmin Mod’s centmin.sh menu option 2, 22 or nv command line to create Nginx vhost sites if they haven’t already been created. Examples including enabling free Letsencrypt SSL certificate support during Nginx vhost site creation here.

touch /etc/centminmod/custom_config.inc
echo "LETSENCRYPT_DETECT='y'" >> /etc/centminmod/custom_config.inc

nv command line

nv

Usage: /usr/bin/nv [-d yourdomain.com] [-s y|n|yd|le|led|lelive|lelived] [-u ftpusername]

  -d  yourdomain.com or subdomain.yourdomain.com
  -s  ssl self-signed create = y or n or https only vhost = yd
  -s  le - letsencrypt test cert or led test cert with https default
  -s  lelive - letsencrypt live cert or lelived live cert with https default
  -u  your FTP username

  example:

  /usr/bin/nv -d yourdomain.com -s y -u ftpusername
  /usr/bin/nv -d yourdomain.com -s n -u ftpusername
  /usr/bin/nv -d yourdomain.com -s yd -u ftpusername
  /usr/bin/nv -d yourdomain.com -s le -u ftpusername
  /usr/bin/nv -d yourdomain.com -s led -u ftpusername
  /usr/bin/nv -d yourdomain.com -s lelive -u ftpusername
  /usr/bin/nv -d yourdomain.com -s lelived -u ftpusername

create self-signed SSL certificate with non-HTTPS + HTTPS vhosts

/usr/bin/nv -d tun.domain.com -s y -u ftpusername

create self-signed SSL certificate with HTTPS only vhosts

/usr/bin/nv -d tun.domain.com -s yd -u ftpusername

create free Letsencrypt SSL certificate with non-HTTPS + HTTPS vhosts

/usr/bin/nv -d tun.domain.com -s lelive -u ftpusername

create free Letsencrypt SSL certificate with HTTPS only vhosts

/usr/bin/nv -d tun.domain.com -s lelived -u ftpusername

Manual Argo Tunnel Setup with cloudflared

You can manually using cloudflared binary to setup an Argo Tunnel

Step 1. Install cloudflared Binary

You can download latest cloudflared binary from here.

wget https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64 -O /usr/local/bin/cloudflared
chmod +x /usr/local/bin/cloudflared
cloudflared update

Authenticate cloudflared using your Cloudflare Account log as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/setup

cloudflared tunnel login

Verify cloudflared version installed

cloudflared --version
cloudflared version 2021.2.1 (built 2021-02-04-1528 UTC)

Step 2. Create Argo Tunnel

  • You can name your Argo Tunnels whatever you want, here we’ll use the intended hostname/subdomain as the name of the Argo Tunnel.
  • Pipe the tunnel create command output into a log file named $hostname-tunnel-create.log so can use the log file to find the tunnel id and credential file needed to install cloudflared as a service and assign them to variables.
  • You’ll need to have jq installed which is installed by default on Centmin Mod LEMP stacks. If not you can install it via yum -y install jq.
hostname=tun.domain.com
cloudflared tunnel create $hostname | tee $hostname-tunnel-create.log
tunnelid=$(cloudflared tunnel list -o json | jq -r --arg h $hostname '.[] | select(.name == $h) | .id')
credfile="/root/.cloudflared/${tunnelid}.json"

Step 3. Creating cloudflared YAML Config File

Create the cloudflared YAML config file at ~/.cloudflared/config.yml by populating the variables below. This configuration uses Cloudflare Named Tunnels and Ingress Rules. Updated: October 14, 2021 to add originServerName to originRequest section.

hostname=tun.domain.com
localhost=https://localhost:443
metrics=localhost:5432
cftag='cmm=test'
cfpid='/var/run/cmm-test-argo.pid'
cfupdatettl='24h'

Make a copy of generated cert.pem file as /root/.cloudflared/cert-${hostname}.pem which will be used for origincert configuration value in tunnel’s YAML config file.

cp -a /root/.cloudflared/cert.pem /root/.cloudflared/cert-${hostname}.pem

Creating ~/.cloudflared/config.yml file with variables you assigned. Updated: October 21, 2021 – Cloudflare now supports Cloudflare Tunnel connections over QUIC protocol via a UDP listener on the Cloudflare edge server side. So you can now choose between protocol: http2 or protocol: quic in your configuration file. See the protocol argument for configuring Cloudflare Tunnel. If using protocol: quic, ensure your system default UDP receive buffer size is optimally configured.

cat > ~/.cloudflared/config.yml <<EOF
tunnel: $tunnelid
credentials-file: $credfile
origincert: /root/.cloudflared/cert-${hostname}.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: $metrics
#tag: $cftag
pidfile: $cfpid
autoupdate-freq: $cfupdatettl
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: $hostname
    service: $localhost
    originRequest:
      connectTimeout: 10s
      originServerName: $hostname
      noTLSVerify: true
  - service: http_status:404
EOF

The resulting ~/.cloudflared/config.yml file will contain

tunnel: your_tunnelid
credentials-file: /root/.cloudflared/your_tunnelid.json
origincert: /root/.cloudflared/cert-tun.domain.com.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: localhost:5432
#tag: cmm=test
pidfile: /var/run/cmm-test-argo.pid
autoupdate-freq: 24h
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: tun.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: tun.domain.com
      noTLSVerify: true
  - service: http_status:404

On SystemD based system like CentOS 7, you can query your cloudflared daemon logs using journalctl.

journalctl -u cloudflared --no-pager | sed -e "s|$(hostname)|hostname|g"

For instance, you can verify the location of your working service config.xml file and which protocol cloudflared daemon is connecting to Cloudflare edge servers with – either HTTP/2 or QUIC.

Oct 20 17:00:12 hostname systemd[1]: Stopped Argo Tunnel.
Oct 20 17:00:12 hostname systemd[1]: Starting Argo Tunnel...
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Starting tunnel tunnelID=bbcb1xxx-3a7b-4f69-914c-xxx
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Version 2021.10.3
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF GOOS: linux, GOVersion: devel +a84af465cb Mon Aug 9 10:31:00 2021 -0700, GoArch: amd64
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Settings: map[autoupdate-freq:24h0m0s config:/etc/cloudflared/config.yml 
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Generated Connector ID: 062e30ae-c7a9-xxxx-b7e2-xxxx
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Initial protocol quic

With Argo Tunnel Ingress rules, you can setup other Cloudflare domain zone sites within the same Cloudflare Account to go through this single Cloudflare Argo Tunnel by adding the additional domain/subdomains i.e. hostname.domain.com and host.example.com as CNAME DNS records pointing to the same tunnelid.cfargotunnel.com target. Then update ~/.cloudflared/config.yml config file to:

tunnel: your_tunnelid
credentials-file: /root/.cloudflared/your_tunnelid.json
origincert: /root/.cloudflared/cert-tun.domain.com.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: localhost:5432
#tag: cmm=blog
pidfile: /var/run/cmm-test-argo.pid
autoupdate-freq: 24h
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: tun.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: tun.domain.com
      noTLSVerify: true
  - hostname: hostname.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: hostname.domain.com
      noTLSVerify: true
  - hostname: host.example.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: host.example.com
      noTLSVerify: true
  - service: http_status:404

I commented out the tag: setting for now as manually running the cloudflared argo tunnel gives an error right now. Update: it seems the tag argument currently isn’t meant to do anything.

cloudflared tunnel --config ~/.cloudflared/config.yml run tun.domain.com
expected string slice found string for tag

Validate ingress rules

cloudflared tunnel ingress validate
Validating rules from /root/.cloudflared/config.yml
OK

Step 4. Create Cloudflare CNAME DNS Record To Route Argo Tunnel

As per documentation https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/routing-to-tunnel/dns, you can create the CNAME DNS record via command line. This will only work for the Cloudflare site zone that you authenticated the initial cloudflared login setup for in Step 1. Other Cloudflare site zones you intend to add to the Argo Tunnel will have to have their CNAME DNS records added either manually or via Cloudflare DNS API.

cloudflared tunnel route dns <UUID or NAME> www.app.com

In this case

cloudflared tunnel route dns tun.domain.com tun.domain.com

or using $tunnelid variable you populated in step 3 above.

cloudflared tunnel route dns $tunnelid tun.domain.com

This will then create the CNAME DNS record for tun.domain.com to point to tunnelid.cfargotunnel.com target if the CNAME doesn’t already exist. If the CNAME already exists, you’ll get an error and need to manually edit and update the existing CNAME DNS record.

Step 5. Install cloudflared Service on CentOS 7

Install cloudflared service, start it and ensure it starts on server reboots. Running service install command will create and configure systemd service file at /etc/systemd/system/cloudflared.service and seems the config file at ~/.cloudflared/config.yml is copied and read from /etc/cloudflared/config.yml afterwards.

[Unit]
Description=Argo Tunnel
After=network.target

[Service]
TimeoutStartSec=0
Type=notify
ExecStart=/usr/local/bin/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate tunnel run
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

You can check the cloudflared log file at /var/log/cloudflared.log.

cloudflared service install
service cloudflared start
chkconfig cloudflared on

Check status

systemctl status cloudflared.service

or

service cloudflared status

If you already had previously installed cloudflared service, you’ll get an error message:

cloudflared service install
Possible conflicting configuration in /root/.cloudflared/config.yml and /etc/cloudflared/config.yml. Either remove /etc/cloudflared/config.yml or run `cloudflared --config /etc/cloudflared/config.yml service install`

Then start and enable services to auto update the cloudflared binary.

systemctl start cloudflared-update.timer
systemctl enable cloudflared-update.timer
systemctl status cloudflared-update.timer

You can then run the systemctl list-timers command to list when cloudflared will next update:

systemctl list-timers cloudflared-update.timer --all
NEXT                         LEFT          LAST PASSED UNIT                     ACTIVATES
Tue 2021-10-26 00:00:00 UTC  4h 20min left n/a  n/a    cloudflared-update.timer cloudflared-update.service

Automated Argo Tunnel Setup with Cloudflare API

Argo Tunnels can also be created via Cloudflare API. This below outlined steps could be scripted for automatic Cloudflare Argo Tunnel creation and setup.

Step 1. Create Cloudflare API Token with Argo Tunnel Write (Edit) Permission

Create a Cloudflare API Token with write permissions = Edit at the Cloudflare account level and DNS edit permissions at zone level.

Cloudflare Argo Tunnel API Token permissions

If you forget or lose your generated Cloudflare API Token, you can go to API Token listing page and in menu drop down (3 dots) link, select Roll to regenarate a new Cloudflare API Token.

Cloudflare API Token roll generate

Step 2. Install cloudflared

If you have yet to install and authenticate cloudflared, you can do the one time task and install it via curl:

curl -4s https://bin.equinox.io/c/VdrWdbjqyF/cloudflared-stable-linux-amd64.tgz | tar xzC /usr/local/bin
cloudflared update

Authenticate cloudflared using your Cloudflare Account log as per https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup/setup

cloudflared tunnel login

Verify cloudflared version installed

cloudflared --version
cloudflared version 2021.2.1 (built 2021-02-04-1528 UTC)

Step 3. Create Argo Tunnel

Create Argo Tunnel via Cloudflare API with session variables populated where tunnelname is your desired Argo Tunnel hostname/subdomain.com and Cloudflare Account Id and Coudflare Zone ID are available on your Cloudflare site’s zone overview page right column and Cloudflare API Token is the one you generated from previous step. The Cloudflare API curl command output is saved to $tunnelname-tunnel-create.log log so we can get the Argo Tunnel’s tunnel id.

cfaccountid='your_cf_account_id'
cfapitoken='your_cf_api_token'
cfzoneid='your_cf_zone_id'
# tunnel name = hostname
tunnelname=tun.domain.com
tunnelsecret=$(openssl rand -hex 16 | base64)

curl -4sX POST "https://api.cloudflare.com/client/v4/accounts/$cfaccountid/tunnels" \
     -H "Authorization: Bearer $cfapitoken" \
     -H "Content-Type: application/json" \
     --data "{\"name\":\"$tunnelname\",\"tunnel_secret\":\"$tunnelsecret\"}" | jq | tee $tunnelname-tunnel-create.log

command output:

{
  "id": "34c4cadb-2edc-47ac-a682-xxxxxxxx",
  "created_at": "2021-02-08T17:41:25.125890Z",
  "deleted_at": null,
  "name": "tun.domain.com",
  "connections": []
}

Get tunnel id from $tunnelname-tunnel-create.log log

tunnelid=$(jq -r '.id' $tunnelname-tunnel-create.log)

Step 4. Create Argo Tunnel CNAME DNS Record

tunnelid=$(jq -r '.id' $tunnelname-tunnel-create.log)

curl -4sX POST "https://api.cloudflare.com/client/v4/zones/$cfzoneid/dns_records" \
     -H "Authorization: Bearer $cfapitoken" \
     -H "Content-Type: application/json" \
     --data "{\"type\":\"CNAME\",\"name\":\"$tunnelname\",\"content\":\"$tunnelid.cfargotunnel.com\",\"proxied\":true}" | jq | tee $tunnelname-tunnel-cname.log

command output:

{
  "result": {
    "id": "9b848b83cc6dfb46169cf8a9a9d9446b",
    "zone_id": "your_cf_zone_id",
    "zone_name": "domain.com",
    "name": "tun2.domain.com",
    "type": "CNAME",
    "content": "34c4cadb-2edc-47ac-a682-xxxxxxxx.cfargotunnel.com",
    "proxiable": true,
    "proxied": true,
    "ttl": 1,
    "locked": false,
    "meta": {
      "auto_added": false,
      "managed_by_apps": false,
      "managed_by_argo_tunnel": false,
      "source": "primary"
    },
    "created_on": "2021-02-08T18:04:17.990511Z",
    "modified_on": "2021-02-08T18:04:17.990511Z"
  },
  "success": true,
  "errors": [],
  "messages": []
}

This will then create the CNAME DNS record for tun2.domain.com to point to tunnelid.cfargotunnel.com target if the CNAME doesn’t already exist. If the CNAME already exists, you’ll get an error and need to manually edit and update the existing CNAME DNS record.

Step 5. Create Argo Tunnel Credentials JSON File

echo "{
  \"AccountTag\": \"$cfaccountid\",
  \"TunnelSecret\": \"$tunnelsecret\",
  \"TunnelID\": \"$tunnelid\",
  \"TunnelName\": \"$tunnelname\"
}" | jq -c | tee ~/.cloudflared/$tunnelid.json

contents of ~/.cloudflared/$tunnelid.json

cat ~/.cloudflared/$tunnelid.json | jq

{
  "AccountTag": "your_cf_account_id",
  "TunnelSecret": "ZTdkYTRlZmJjZDRmNDRlZjRiYzFjN2UxNGI0NzgyMTIK",
  "TunnelID": "34c4cadb-2edc-47ac-a682-xxxxxxxx",
  "TunnelName": "tun2.domain.com"
}

Step 6. Create Argo Tunnel YAML Config File

Create the cloudflared YAML config file at ~/.cloudflared/config.yml by populating the variables below. This configuration uses Cloudflare Named Tunnels and Ingress Rules. Updated: October 14, 2021 to add originServerName to originRequest section.

hostname=tun2.domain.com
credfile="/root/.cloudflared/${tunnelid}.json"
localhost=https://localhost:443
metrics=localhost:5432
cftag='cmm=test'
cfpid='/var/run/cmm-test-argo.pid'
cfupdatettl='24h'

Make a copy of generated cert.pem file as /root/.cloudflared/cert-${hostname}.pem which will be used for origincert configuration value in tunnel’s YAML config file.

cp -a /root/.cloudflared/cert.pem /root/.cloudflared/cert-${hostname}.pem

Creating ~/.cloudflared/config.yml file with variables you assigned. Updated: October 21, 2021 – Cloudflare now supports Cloudflare Tunnel connections over QUIC protocol via a UDP listener on the Cloudflare edge server side. So you can now choose between protocol: http2 or protocol: quic in your configuration file. See the protocol argument for configuring Cloudflare Tunnel. If using protocol: quic, ensure your system default UDP receive buffer size is optimally configured.

cat > ~/.cloudflared/config.yml <<EOF
tunnel: $tunnelid
credentials-file: $credfile
origincert: /root/.cloudflared/cert-${hostname}.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: $metrics
#tag: $cftag
pidfile: $cfpid
autoupdate-freq: $cfupdatettl
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: $hostname
    service: $localhost
    originRequest:
      connectTimeout: 10s
      originServerName: $hostname
      noTLSVerify: true
  - service: http_status:404
EOF

The resulting ~/.cloudflared/config.yml file will contain

tunnel: your_tunnelid
credentials-file: /root/.cloudflared/your_tunnelid.json
origincert: /root/.cloudflared/cert-tun.domain.com.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: localhost:5432
#tag: cmm=test
pidfile: /var/run/cmm-test-argo.pid
autoupdate-freq: 24h
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: tun2.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: tun2.domain.com
      noTLSVerify: true
  - service: http_status:404

On SystemD based system like CentOS 7, you can query your cloudflared daemon logs using journalctl.

journalctl -u cloudflared --no-pager | sed -e "s|$(hostname)|hostname|g"

For instance, you can verify the location of your working service config.xml file and which protocol cloudflared daemon is connecting to Cloudflare edge servers with – either HTTP/2 or QUIC.

Oct 20 17:00:12 hostname systemd[1]: Stopped Argo Tunnel.
Oct 20 17:00:12 hostname systemd[1]: Starting Argo Tunnel...
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Starting tunnel tunnelID=bbcb1xxx-3a7b-4f69-914c-xxx
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Version 2021.10.3
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF GOOS: linux, GOVersion: devel +a84af465cb Mon Aug 9 10:31:00 2021 -0700, GoArch: amd64
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Settings: map[autoupdate-freq:24h0m0s config:/etc/cloudflared/config.yml 
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Generated Connector ID: 062e30ae-c7a9-xxxx-b7e2-xxxx
Oct 20 17:00:12 hostname cloudflared[59286]: 2021-10-20T17:00:12Z INF Initial protocol quic

With Argo Tunnel Ingress rules, you can setup other Cloudflare domain zone sites within the same Cloudflare Account to go through this single Cloudflare Argo Tunnel by adding the additional domain/subdomains i.e. hostname.domain.com and host.example.com as CNAME DNS records pointing to the same tunnelid.cfargotunnel.com target. Then update ~/.cloudflared/config.yml config file to:

tunnel: your_tunnelid
credentials-file: /root/.cloudflared/your_tunnelid.json
origincert: /root/.cloudflared/cert-tun2.domain.com.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: localhost:5432
#tag: cmm=blog
pidfile: /var/run/cmm-test-argo.pid
autoupdate-freq: 24h
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: tun2.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: tun2.domain.com
      noTLSVerify: true
  - hostname: hostname.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: hostname.domain.com
      noTLSVerify: true
  - hostname: host.example.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: host.example.com
      noTLSVerify: true
  - service: http_status:404

I commented out the tag: setting for now as manually running the cloudflared argo tunnel gives an error right now. Update: it seems the tag argument currently isn’t meant to do anything.

cloudflared tunnel --config ~/.cloudflared/config.yml run tun.domain.com
expected string slice found string for tag

Validate ingress rules

cloudflared tunnel ingress validate
Validating rules from /root/.cloudflared/config.yml
OK

Step 7. Install cloudflared Service

Install cloudflared service, start it and ensure it starts on server reboots. Running service install command will create and configure systemd service file at /etc/systemd/system/cloudflared.service and seems the config file at ~/.cloudflared/config.yml is copied and read from /etc/cloudflared/config.yml afterwards.

[Unit]
Description=Argo Tunnel
After=network.target

[Service]
TimeoutStartSec=0
Type=notify
ExecStart=/usr/local/bin/cloudflared --config /etc/cloudflared/config.yml --no-autoupdate tunnel run
Restart=on-failure
RestartSec=5s

[Install]
WantedBy=multi-user.target

You can check the cloudflared log file at /var/log/cloudflared.log.

cloudflared service install
service cloudflared start
chkconfig cloudflared on

Check status

systemctl status cloudflared.service

or

service cloudflared status

If you already had previously installed cloudflared service, you’ll get an error message:

cloudflared service install
Possible conflicting configuration in /root/.cloudflared/config.yml and /etc/cloudflared/config.yml. Either remove /etc/cloudflared/config.yml or run `cloudflared --config /etc/cloudflared/config.yml service install`

Then start and enable services to auto update the cloudflared binary.

systemctl start cloudflared-update.timer
systemctl enable cloudflared-update.timer
systemctl status cloudflared-update.timer

You can then run the systemctl list-timers command to list when cloudflared will next update:

systemctl list-timers cloudflared-update.timer --all
NEXT                         LEFT          LAST PASSED UNIT                     ACTIVATES
Tue 2021-10-26 00:00:00 UTC  4h 20min left n/a  n/a    cloudflared-update.timer cloudflared-update.service

Cloudflare Zero Trust Terminal Browser

Cloudflare updated Cloudflare Tunnels to support Cloudflare Zero Trust Terminal Browser access allowing you to utilise Cloudflare For Team’s Access to create a secure browser rendered SSH terminal to connect to your servers. The instructions are documented here.

To continue with the above examples of an Argo Tunnel Ingress rules multi host/origin configuration with ~/.cloudflared/config.yml config file modified to add the following:

  - hostname: yourssh.domain.com
    service: ssh://localhost:22

setting up your desired browser SSH terminal hostname as yourssh.domain.com which would be accessed at https://yourssh.domain.com so that the full ~/.cloudflared/config.yml config file looks like below example:

tunnel: your_tunnelid
credentials-file: /root/.cloudflared/your_tunnelid.json
origincert: /root/.cloudflared/cert-tun2.domain.com.pem
protocol: http2
originRequest:
  connectTimeout: 30s

metrics: localhost:5432
#tag: cmm=blog
pidfile: /var/run/cmm-test-argo.pid
autoupdate-freq: 24h
loglevel: info
logfile: /var/log/cloudflared.log

ingress:
  - hostname: tun2.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: tun2.domain.com
      noTLSVerify: true
  - hostname: hostname.domain.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: hostname.domain.com
      noTLSVerify: true
  - hostname: host.example.com
    service: https://localhost:443
    originRequest:
      connectTimeout: 10s
      originServerName: host.example.com
      noTLSVerify: true
  - hostname: yourssh.domain.com
    service: ssh://localhost:22
  - service: http_status:404

Then manually setup the Cloudflare CNAME DNS record, yourssh.domain.com pointing to the same tunnelid.cfargotunnel.com target you initially created.

Then restart cloudflared service

service cloudflared restart

Then log into your Cloudflare For Teams dashboard and go to Cloudflare Access section to set up a self-hosted app as outlined  here and here (skip advanced step 3 and only do step 1 for setup of Self Hosted App and step 2 to add a policy to control who accesses your app at https://yourssh.domain.com).

The key setting for Zero Trust browser terminals is to enable a new setting for Enable browser rendering:

Cloudflare Access Enable browser rendering

Then accessing the browser terminal at https://yourssh.domain.com where you’d be greeted with your Cloudflare Access policy for login and be prompted to enter username and password for your SSH server or use shorted lived SSH certificates for Public Key Authentication.

Using Github as an authentication method to access the browser terminal URL.

Cloudflare For Team's Access Github authentication

SSH username and password prompts

Cloudflare Zero Trust browser terminal login

Cloudflare Zero Trust browser terminal login

Once logged in, you’re able to access the browser terminal for the server.

Cloudflare Zero Trust browser terminal for SSH access to server

Listing Argo Tunnels Created

Above method used via cloudflared or Cloudflare API will not list the created Argo Tunnel in Cloudflare dashboard’s Traffic > Argo Tunnel section in web GUI. Update October 19, 2021: Cloudflare For Teams GUI dashboard now allows you to list the Cloudflare Tunnels you have created using the above method.

To list your created Argo Tunnels use either command:

cloudflared tunnel list

or

cloudflared tunnel list -o json

Or via the new Cloudflare For Teams Access Tunnel GUI dashboard:

Cloudflare For Teams Access Tunnel GUI dashboard

Centmin Mod Nginx Access Logs

Argo Tunnel by default won’t pass on the visitor’s real IP address to Centmin Mod Nginx. Instead it will show up like

tail -1 /home/nginx/domains/tun.domain.com/log/access.log 
127.0.0.1 - - [07/Feb/2021:22:39:12 +0000] "GET /?test HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36 OPR/74.0.3911.75"

Centmin Mod Nginx config file at /usr/local/nginx/conf/nginx.conf has additional Nginx log format options which can log Argo Tunnel received visitor’s real IP addresses via $http_x_forwarded_for field.

Example for log format named cf_custom4

log_format cf_custom4 '$remote_addr - $remote_user [$time_local] $request '
              '"$status" $body_bytes_sent "$http_referer" '
              '"$http_user_agent" "$http_x_forwarded_for" "$gzip_ratio" "$brotli_ratio"'
              ' "$connection" "$connection_requests" "$request_time" $http_cf_ray '
              '$ssl_protocol $ssl_cipher $http_content_length $http_content_encoding $request_length';

You can alter and create your own custom log formats too.

To enable this, you need to edit Nginx vhost config files include file /usr/local/nginx/conf/cloudflare.conf in /usr/local/nginx/conf/conf.d/tun.domain.com.ssl.conf and/or /usr/local/nginx/conf/conf.d/tun.domain.com.conf. This enables Cloudflare’s documented restoration of real visitor IP addresses.

# uncomment cloudflare.conf include if using cloudflare for
# server and/or vhost site
include /usr/local/nginx/conf/cloudflare.conf;

And also add a second access log line assigning the log format named cf_custom4

access_log /home/nginx/domains/tun.domain.com/log/cf-ssl-access.log cf_custom4;

to existing one so it looks like

access_log /home/nginx/domains/tun.domain.com/log/access.log combined buffer=256k flush=5m;
error_log /home/nginx/domains/tun.domain.com/log/error.log;
access_log /home/nginx/domains/tun.domain.com/log/cf-access.log cf_custom4;

Then restart Nginx

service nginx restart

Or via Centmin Mod command shortcut

ngxrestart

Visit Argo Tunnel site and then check the new access log and see the $http_x_forwarded_for field record the real visitor IP = 122.xxx.xxx.xxx

tail -1 /home/nginx/domains/tun.domain.com/log/cf-access.log

127.0.0.1 - - [07/Feb/2021:22:39:12 +0000] GET /?test HTTP/1.1 "200" 2112 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.104 Safari/537.36 OPR/74.0.3911.75" "122.xxx.xxx.xxx" "3.14" "-" "54043" "3" "0.000" 61f323cc591c32a4-ORD TLSv1.3 TLS_AES_256_GCM_SHA384 - - 1303

Cloudflared Log Rotation

The above cloudflared config.yml configuration file specifies logging at logfile: /var/log/cloudflared.log. To manage the log size, you’d also need to setup log rotation profile at /etc/logrotate.d/cloudflared containing the follow:

/var/log/cloudflared.log {
        daily
        dateext
        missingok
        rotate 8
        maxsize 100M
        compress
        delaycompress
        copytruncate
        notifempty
        missingok
}

Then the /etc/logrotate.d/cloudflared will rotate daily automatically.

Debug logrotate run:

logrotate -d /etc/logrotate.d/cloudflared
reading config file /etc/logrotate.d/cloudflared
Allocating hash table for state file, size 15360 B

Handling 1 logs

rotating pattern: /var/log/cloudflared.log after 1 days (8 rotations)
empty log files are not rotated, log files >= 104857600 are rotated earlier, old logs are removed
considering log /var/log/cloudflared.log
log needs rotating
rotating log /var/log/cloudflared.log, log->rotateCount is 8
dateext suffix '-20211021'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
glob finding logs to compress failed
glob finding old rotated logs failed
copying /var/log/cloudflared.log to /var/log/cloudflared.log-20211021
truncating /var/log/cloudflared.log

Forced a manual logrotate run:

logrotate -f /etc/logrotate.d/cloudflared

Then check your /var/log directory

ls -lahrt /var/log | grep cloudflared
-rw-r--r--   1 root  root  435M Oct 21 22:54 cloudflared.log-20211021
-rw-r--r--   1 root  root     0 Oct 21 22:54 cloudflared.log

Cloudflared System Settings

Cloudflare Tunnel’s cloudflared daemon defaults to http2 protocol persistent connections. However, it can now be manually switched to using quic protocol persistent connections via it’s config.yml config file setup outlined above. cloudflared uses quic-go library it seems and wants to use a UDP receive buffer size of 2MB. On Centmin Mod LEMP stack systems this is not a problem as initial installs of Centmin Mod already increase the buffer size out the box.

// DesiredReceiveBufferSize is the kernel UDP receive buffer size that we'd like to use.
const DesiredReceiveBufferSize = (1 << 20) * 2 // 2 MB

However, on non-Centmin Mod LEMP systems, you may need to raise your default UDP receive buffer size as outlined at https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for more optimal transfer performance when using for your Cloudflare Tunnel when configured to use quic protocol persistent connections. Otherwise, you may encounter this error where 208 kiB is usually the default Linux receive buffer size which via SO_RCVBUF gets doubled by the Linux Kernel to 416 kiB. But it isn’t enough as quic-go wants to ideally use 2MB.

failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.

Cloudflare Authenticated Origin Pull Incompatibility

Cloudflare Argo Tunnel is incompatible with Cloudflare Authenticated Origin Pull as they both do the same thing to prevent visitor access that bypasses Cloudflare proxy and tries HTTP access via the server’s real IP access – just done in different ways. So if you have manually enabled Cloudflare Authenticated Origin Pull certificate configuration in your Nginx vhost config file at /usr/local/nginx/conf/conf.d/yourdomain.com.conf and/or /usr/local/nginx/conf/conf.d/yourdomain.com.ssl.conf, you will need to disable them again by commenting out the relevant lines using a hash # in front of the 2nd and 3rd lines below and then restart Nginx service. By default Cloudflare Authenticated Origin Pull is disabled with a hash in front usually.

 # cloudflare authenticated origin pull cert community.centminmod.com/threads/13847/
 # ssl_client_certificate /usr/local/nginx/conf/ssl/cloudflare/yourdomain.com/origin.crt;
 # ssl_verify_client on;

Cloudflare Restrict IPs

The final step once Cloudflare Tunnels is working is to restrict HTTP (port 80) and HTTPS (port 443) access on your origin server to just Cloudflare client/edge server requests so that no other non-Cloudflare requests can reach your origin server. You do this by restricting at origin server firewall level only requests made by Cloudflare servers. As this guide is for Centmin Mod LEMP stack users who use CSF Firewall, then to do this step just remove port 80 and 443 from CSF Firewall’s /etc/csf/csf.conf (backup file before editing) config file for TCP_IN and TCP6_IN comma separated list and then restart CSF Firewall.

csf -ra

Centmin Mod initial install and above CSF Firewall whitelist configuration would have already taken care of steps to allow Cloudflare only requests at CSF Firewall level. Removing the port 80 and 443 from CSF Firewall’s /etc/csf/csf.conf config file would take care of the remaining step to disallow non-Cloudflare requests from hitting the origin server on those ports.

If you need to reverse this change, just re-add port 80 and 443 to CSF Firewall’s /etc/csf/csf.conf settings for TCP_IN and TCP6_IN comma separated list and then restart CSF Firewall again.

If you’re using a non-Centmin Mod or non-CSF Firewall setup, then you’d have to deal with iptables based restrictions outlined by the Cloudflare support article here.

Cloudflare Tunel http2 vs quic protocol connections

Cloudflare Tunnels now support either HTTP/2 or QUIC protocol based persistent connections from Cloudflare edge server to local origin cloudflared daemon instance which sits as a reverse proxy in front of your local origin server.

I ran some loader.io load tests comparing Cloudflare Tunnel protocol connection methods http2 vs quic for both Cloudflare CDN cached and also for Cloudflare CDN bypass cached configurations. As you can see cache bypassed Cloudflare Tunnel with QUIC protocol based persistent connections did slightly better than with the default HTTP/2 protocol based persistent connections with ~16.6% better average response times.

Below are the configurations tested:

  • CF Tunnel http2 protocol – CF Cache Bypassed – 500 users with HTTP gzip compressed requests – cache bypassed means testing origin server and thus the Cloudflare edge server to local cloudflared daemon instance connection.
  • CF Tunnel quic protocol – CF Cache Bypassed – 500 users with HTTP gzip compressed requests – cache bypassed means testing origin server and thus the Cloudflare edge server to local cloudflared daemon instance connection.
  • CF Tunnel http2 protocol – CF Cached – 1000 users with HTTP gzip compressed requests – cached means testing Cloudflare CDN Cache
  • CF Tunnel quic protocol – CF Cached – 1000 users with HTTP gzip compressed requests – cached means testing Cloudflare CDN Cache

This WordPress blog is hosted in US West Coast, San Jose on a US$5/month – 1 CPU, 1GB memory based KVM VPS server at Upcloud.com running my optimized Centmin Mod LEMP stack on CentOS 7 64bit OS with fully optimized full HTML page caching at Centmin Mod Nginx level and origin server PHP-FPM is compiled with Profile Guide Optimization training for specific WordPress PHP usage loads to further boost PHP performance (PHP 8 vs 7 with/without PGO).

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 1
On-line CPU(s) list: 0
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Gold 6136 CPU @ 3.00GHz
Stepping: 4
CPU MHz: 2992.968
BogoMIPS: 5985.93
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 4096K
L3 cache: 16384K
NUMA node0 CPU(s): 0
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat umip pku ospke md_clear spec_ctrl intel_stibp

Tabulated results:

Loader.io results

CF Tunnel http2 protocol – CF Cache Bypassed – 500 users

loader.io cache bypass tunnel http2

CF Tunnel quic protocol – CF Cache Bypassed – 500 users

loader.io cache bypass tunnel quic

CF Tunnel http2 protocol – CF Cached – 1000 users

loader.io cached tunnel http2

CF Tunnel quic protocol – CF Cached – 1000 users

loader.io cached tunnel quic

The origin WordPress server’s CPU and memory usage statistics for both Tunnel HTTP/2 and QUIC protocol persistent connections. The HTTP/2 is the higher CPU usage peak and the QUIC UDP based protocol connection test is the lower CPU usage peak.

Newrelic CPU & Memory usage