Deep Dive Kioskgaming Infrastructure
From Cloudflare to Nginx and all the backend services

1. Architectural overview

All public traffic of the platform passes through3 floors: Cloudflare (DNS + CDN + TLS termination) → nginx-lb(an nginx container as reverse proxy / load balancer, routing accordinglyserver_name) → app servers / data services / toolson the internal network10.0.0.0/24. All components run Docker, each service has a filedocker-composeonly in the root repo.

Nginx configuration illustrated step by step

The actual configuration segments correspond to each arrow in the diagram above — for complete details see sections 3–5.

set_real_ip_from 173.245.48.0/20;
… (15 Cloudflare CIDR bands) …
set_real_ip_from 131.0.72.0/22;
real_ip_header CF-Connecting-IP;

upstream backend {
    least_conn;
    server 10.0.0.5:3001 max_fails=3 fail_timeout=30s;
    …(8 instance: 10.0.0.5 + 10.0.0.4, :3001–3004)…
}

proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

worker_processes auto;
worker_connections 1024;
gzip on; gzip_comp_level 6;
include conf.d/*.conf;

image: nginx:alpine
restart: always
network_mode: host

22/tcp ← Any IPv4/IPv6 (unlimited)
80/tcp ← 5/15 Cloudflare strips (10 strips missing!)
443/tcp ← 15/15 ranges (nginx does not listen to 443)

The key point to grasp in 60 seconds

• A single gateway:Every public domain points to the containernginx-lbrun onV1-Kiosk (178.156.194.24)— both reverse proxy and load balancer, and issingle point of failure (SPOF): If V1 dies, the whole system dies, even though other servers are still alive.
• Cloudflare Flexible mode:client ↔ Cloudflare is HTTPS, Cloudflare ↔ origin isPlain HTTP :80. Origin doesn't have TLS. The backend distinguishes HTTPS thanks to the headerX-Forwarded-Protomounted by Cloudflare.
• The two levels of source-IP filtering are different:nginx trusts 15 Cloudflare bands, but Hetzner Firewall at port :80 only whitelists 5/15 bands — real requests from the remaining 10 bands can be blocked before reaching nginx. SSH (:22) is open to the entire Internet. See section 13.
• API productionRun 8 instances (2 servers × 4 ports), load balancedleast_conn.
• Data is concentrated on 10.0.0.2 (Server 3): Redis master + Postgres 16 master; replicate sang 10.0.0.3 / Server 4(Redis slave + Postgres slave, see section 7). Redis/OpenSearch does not have application auth, but Hetzner Firewall (the only firewall in the project) does not open their ports to the Internet.
• 2-storey log:Dozzle quick view per-server; Fluent Bit → OpenSearch → Dashboards for centralized viewing. All four Dozzle containers (V1–V4) reportedunhealthydue to wrong image healthcheck — not a problem, just fix/disable healthcheck (risk #11).
• Hostname is not reliable to identify the server:V2 and V3 both return hostnameubuntu-8gb-ash-1when SSH — only IP can reliably differentiate.

2. List of VPS

Source:iplist, comment in nginx/conf.d/*.conf, compose files; server name/plan/IP server 5:Hetzner Console(project "Kiosk VPS", region Ashburn VA / us-east, screenshot 2026-07-03). The role of each server is derived from the internal IP that nginx points to and commented in the conf.

* vCPU/RAM according to Hetzner's CCX (dedicated vCPU) plan catalog; disk according to the value displayed in the console. Server created 4–9 months ago (V1 the longest).iplistServer 5 is missing in the repo — should be added.

3. Cloudflare tier (edge)edge

3.1. SSL mode: Flexible

Entire domain*.kioskservice.clubproxy via Cloudflare with SSL modeFlexible:

Client ──HTTPS :443──▶ Cloudflare ──HTTP :80──▶ nginx-lb (origin)

• TLS ends at Cloudflare. Origin only listens for HTTP :80 — nonelisten 443which inconf.d/.
• Cloudflare mounts itselfX-Forwarded-Proto: httpswhen the client uses HTTPS; nginx forwards this value back to the backendproxy_set_header X-Forwarded-Proto $http_x_forwarded_protoso the app knows the original session is HTTPS (Secure cookies, redirect...).Source:nginx/conf.d/default.conf:46-54
• OmniDB alone has blocksmapfallback: if the header is empty, force ithttpsto avoid 403 after login due to Secure cookies.Source:nginx/conf.d/omnidb.conf:6-10

3.2. Restore the client's real IP

Because every request to the origin comes from Cloudflare's IP, nginx uses modulerealip: each server block declared15 Cloudflare IP ranges via set_real_ip_fromand real_ip_header CF-Connecting-IP— meaning: if request comes from Cloudflare IP range, get header value CF-Connecting-IPdo$remote_addr. Thanks to that, the log and rate-limit behind see the correct user IP. Source:nginx/conf.d/default.conf:28-43(repeat in every conf file)

set_real_ip_from 173.245.48.0/20;   set_real_ip_from 103.21.244.0/22;
set_real_ip_from 103.22.200.0/22;   set_real_ip_from 103.31.4.0/22;
set_real_ip_from 141.101.64.0/18;   set_real_ip_from 108.162.192.0/18;
set_real_ip_from 190.93.240.0/20;   set_real_ip_from 188.114.96.0/20;
set_real_ip_from 197.234.240.0/22;  set_real_ip_from 198.41.128.0/17;
set_real_ip_from 162.158.0.0/15;    set_real_ip_from 104.16.0.0/13;
set_real_ip_from 104.24.0.0/14;     set_real_ip_from 172.64.0.0/13;
set_real_ip_from 131.0.72.0/22;
real_ip_header CF-Connecting-IP;

3.3. The Cloudflare header is forwarded to the backend

3.4. Architectural consequences of Flexible mode

• Unencrypted hop Cloudflare → origin:Data (including tokens/credentials in the request) goes through plain HTTP over the Internet between POP Cloudflare and the server. Upgrading to Full (strict) requires an origin with a certificate (Cloudflare Origin CA is enough).
• Origin nghe :80:nginx does not have an origin-check mechanism; Cloudflare out-of-range IP blocking due toHetzner Firewalltake care of — rule :80 only includes Cloudflare bands, but is missing 10/15 bands (risk #13, item 13).
• Cloudflare's WAF/Bot/CDN protection only works when traffic actually passes through Cloudflare.

3.5. DNS record model (from README in repo)

The README guides DNS mapping according to the following principles:Each domain has an A record (Proxied) pointing directly to the public IP of the target server- For examplelog-server1 → IP public Server 1, log-server2 → IP public Server 2, redisinsight/pgadmin → IP public Server 4. Source:README-DOZZLE.md(section "Map Domain in Cloudflare"),README-REDISINSIGHT.md, README-OMNIDB.md.

4. Nginx gateway (nginx-lb) gateway

4.1. Deploy containers

Image

nginx:alpine

Container

nginx-lb, restart: always

Network

network_mode: host— to call internal IPs directly10.0.0.xnot through NAT/bridge

Mount

./nginx/nginx.conf → /etc/nginx/nginx.conf (ro)
./nginx/conf.d → /etc/nginx/conf.d (ro)
./nginx/logs → /var/log/nginx
./nginx/.htpasswd-redisinsight, ./nginx/.htpasswd-app-logs (ro)

Source:nginx.yml:1-15. Original repo on GitLab namekiosk-deployments; The deploy directory on the server iskiosk_deploy/kiosk_deployments(inferred from the volume name prefix in the backup commands of the README).Host: V1-Kiosk (178.156.194.24 / 10.0.0.5); see section 3.5.

4.2. Core configuration (nginx/nginx.conf)

Source:nginx/nginx.conf:1-37. Note: not enabled yetkeepaliveto upstream (linekeepalive 32commenting indefault.conf:15) — each request creates a new TCP connection to the backend.

4.3. Log & logrotate

• Access/error log ghi ra nginx/logs/On the host, each domain has its own pair of files (dev-api-access.log, api-gamify-pro-access.log…).
• Logrotate: daily, keep 7 copies, compressed (delaycompress), send after rotatenginx -s reopeninto the container. Install itsetup-nginx-logrotate.sh → /etc/logrotate.d/nginx-kiosk. Source:nginx/logrotate.conf

4.4. Configuration change process (required)

# 1. Edit the file in nginx/conf.d/ (local repo is just a place to edit — must be synchronized to the server)
#2. Check syntax BEFORE reloading:
docker exec nginx-lb nginx -t
#3. Reload without downtime:
docker exec nginx-lb nginx -s reload

5. Domain routing & upstream

5.1. Full routing table

5.2. Upstream API production — analysis

upstream backend {
least_conn;                                    # choose the server with the least connection
    server 10.0.0.5:3001 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.5:3002 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.5:3003 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.5:3004 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.4:3001 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.4:3002 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.4:3003 weight=1 max_fails=3 fail_timeout=30s;
    server 10.0.0.4:3004 weight=1 max_fails=3 fail_timeout=30s;
}

• 8 instanceNode.js backend (4 ports × 2 servers), equal weight.
• least_connSuitable for APIs with irregular processing times — new requests flow to the freest instance, instead of rotating blindly like round-robin.
• Passive health check:instance is marked as down later3 errorsconsecutively (max_fails=3) and removed from the internal pool30 seconds (fail_timeout=30s), then retried. No active health check (requires nginx Plus or external module — noted in file).Source:nginx/conf.d/default.conf:1-16, README-NGINX.md
• Only use gamify-prodefault round-robin(not declaredleast_conn). api-gamify-pro.conf:1-6

5.3. Common proxy parameters of all server blocks

6. Sample request flow (API production)

7. Data services data

Each service has a compose file in the root repo, running independently:docker compose -f <file>.yml up -d.

7.1. Redis 7.2 — master / slave / dev

Source:redis-master.yml, redis-slave.yml, redis-dev.yml; Host slave confirms passredis-cli info replication (slave0:ip=10.0.0.3,state=online,lag=0). One-way replication master→slave; repo without Sentinel/Cluster ⇒No automatic failover— slave is for manual read/standby only.

7.2. PostgreSQL 16 — streaming replication

• Master (postgres-master.yml): db kiosk, user postgres(plaintext password in yml — not stated here),network_mode: bridge, volume postgres-master-data, init script setup-master.shmount itdocker-entrypoint-initdb.d.
• Archive WALturn on (archive_mode=on, copy it/var/lib/postgresql/archive/in containers) — supports PITR, but the archive is located on the same machine so it cannot prevent losing the whole server.
• Slavepublish host port5433(not 5432) — read connection from app must point<host-slave>:5433. hot_standby_feedback=onReduces canceled query errors on the slave, in return can keep bloat on the master.

7.3. RabbitMQ 3.13

File

rabbitmq.yml — image rabbitmq:3.13-management, host 10.0.0.3 (Server 4)

Port

5672 (AMQP) · 15672 (Management UI — exposed via nginx not visible, direct access/SSH tunnel)

Account

user kiosk, vhost /(plaintext password in yml)

Limit

memory 1024M; healthcheck rabbitmq-diagnostics ping/10s; volume rabbitmq-data

8. Log & Observability observability

8.1. 2-storey architecture

1. Xem nhanh per-server:Each server runs 1 containerDozzle(:8080, host network, readdocker.sockread-only, only currently running container, tail 1000 lines) — accessed vialog-server1…4.kioskservice.club.
2. Concentrate:Fluent Bit on 10.0.0.6 log collection → OpenSearch → view with OpenSearch Dashboards atapp-logs.kioskservice.club(basic auth in nginx).

8.2. Index naming rules (Lua filter)

fluent-bit/tag_prefix.lua(jawappend_fluentbit_tag) assign 2 fields to each record:docker_log_tag(original tag) andlogstash_prefix. Rules for converting tag → prefix:

• Tag fluentd formdocker.kiosk-gaming-backend.dev→ normalize lowercase,.→-, remove the prefixdocker-, add suffix-logs ⇒ index kiosk-gaming-backend-dev-logs-YYYY.MM.DD.
• Tag from INPUT tail (containsvar.lib.docker.containersor/) → combineddocker-tail-logs-*(avoid each container ID being a separate index).
• Empty/invalid tag →misc-logs. Prefix cuts up to 120 characters.

8.3. Prevent log bloat at the Docker layer

• docker-daemon.json(apply equalsetup-docker-logs.sh): daemon-wide defaultjson-file, max-size 10m, max-file 3, compress— each container has a maximum of ~30MB of logs on disk.
• Important composes also declare the same logging options at the service level.
• Test script:check-logs.sh, check-dozzle.sh, check-redisinsight.sh. README recommends setting cron0 2 * * *runcheck-logs.sh, the monitoring log is written out/var/log/kiosk-logs-check.log.
• Expected capacity after full setup (according toREADME-LOG-MANAGEMENT.md): each container ~30MB (10m×3), nginx log ~70–100MB/7 days — total ~100–150MB/server, automatically deleted. Note: changedaemon.jsonmust restart Dockerandrestart each container to take effect.

8.4. Details of setup & Limitations (summarized from READMEs)

• Prerequisite kernel for OpenSearch:host must setvm.max_map_count=262144 (sysctl, persist via /etc/sysctl.d/99-opensearch.conf) — without this OpenSearch will not work.README-OPENSEARCH-SETUP.md §1
• How a new service sends logs to the pipeline: join network external kiosk-logging+ declare logging driver in compose:

  networks:
    kiosk-logging: { external: true, name: kiosk-logging }
  services:
    ten-service:
      networks: [default, kiosk-logging]
      logging:
        driver: fluentd
        options:
  fluentd-address: tcp://fluent-bit:24224 # same host
  tag: docker.name-service.env # tag → index name

  Service inother machinethen pointtcp://10.0.0.6:24224(requires internal firewall permission; absolutely do not open 24224 to the Internet).README-OPENSEARCH-SETUP.md §5
• View logs on Dashboards: Discover → Create index pattern (vd kiosk-*-logs-*), field time@timestamp.
• Memorable historical incidents:Lua files by nameadd_log_tag.luamistakenly mounted by Dockerfolderon host → Fluent Bit error "Is a directory"; renamedtag_prefix.lua. If we meet again:rm -rf fluent-bit/add_log_tag.luaAlreadygit pull. README-OPENSEARCH-SETUP.md §8
• Production recommendation has not been done yet(fromREADME-OPENSEARCH-PRODUCTION.md): enable security plugin + set admin password via env file without committing; retention by ISM or cron to delete index by day (not available yet — Server 5 disk will gradually fill up); backup index using Snapshot API; The heap keeps 512m–1g when the host is running other services, only increasing by 2–4g if OpenSearch is the main workload.

9. Gamification engine

Directorygamify-engine/contains script + env deploy gamification backend on server 10.0.0.6 (code is in separate GitLab repokiosk_gamification_engine, not in the nginx repo).

10. General port table

11. Infrastructure check playbook (health-check)

This section is combinedHow to verify whether each floor is alive or dead, going from outside (Cloudflare) to inside (data server) — Use when suspecting a problem, before/after reloading nginx, or periodically checking. Rule: always checkfrom the layer closest to the user first (public domain) and then go deeper into the internal server, to quickly localize which layer is failing.

11.1. Quick lookup table by server

11.2. Layer 1 — Edge & gateway (from any machine, no SSH required)

# Does the domain respond via Cloudflare (note the cf-ray header = via Cloudflare)
curl -sI https://api.kioskservice.club | grep -i "cf-ray\|http"

# Is the current nginx syntax valid (running ON V1, before reload)
docker exec nginx-lb nginx -t

# Is nginx-lb running, which containers have restarted recently (sign of crash loop)
docker ps --filter name=nginx-lb --format "table {{.Names}}\t{{.Status}}"

Latest run results on V1 and V2: bothnginx -tOK (V1, gateway); 4/4 API prod instances (:3001–3004) + 1 dev (:4001) allhealthyon each server; Low CPU/RAM, lots of headroom on both (~90-100MB/container / 8GB RAM). Privatedozzle unhealthyon both — see 11.4.

11.3. Layer 2 — Data server (V3-Kiosk, 10.0.0.2)

# Redis master
docker exec -it redis-master redis-cli ping                     # → PONG
docker exec -it redis-master redis-cli info replication # connected_slaves matches slave number pointing to 10.0.0.2
docker exec -it redis-master redis-cli info memory | grep used_memory_human

# Postgres master
docker exec -it postgres-master pg_isready
docker exec -it postgres-master psql -U postgres -d kiosk -c "SELECT * FROM pg_stat_replication;"
# each return line = 1 slave replicating; "streaming" state is fine, lack of stream ⇒ slave lost connection

Actual container name dependscontainer_namedeclared inredis-master.yml / postgres-master.yml- usedocker psto confirm if the above command says "No such container".

Latest run result: RedisPONG, replica online lag=0; Postgres pg_isreadyOK, replica streaming. The server runs exactly 3 containers (postgres-master, redis-master, dozzle). Privatedozzle unhealthy — see 11.4.

11.4. Layer 2 — Tools server (V4-Kiosk, 10.0.0.3)

V4 runs7 container: postgres-slave, redis-slave, redis-dev, redisinsight, omnidb, dozzle, rabbitmq.

docker ps --format "table {{.Names}}\t{{.Status}}"
# expect all 7 containers "Up"; Notice the STATUS column has "(healthy)" / "(unhealthy)" / nothing (container does not declare healthcheck)

• RedisInsight (redisinsight.kioskservice.club, basic auth) — used to visually see the live Redis master/slave, key number, realtime memory, instead of typing commandsredis-clihandmade.
• OmniDB / pgadmin (pgadmin.kioskservice.club) — view Postgres master :5432 and slave :5433 via UI, run test query directly.
• Both are behind nginx, so they also indirectly confirm that the gateway layer + routing to 10.0.0.3 is still open.

11.5. Layer 2 — Gamify + Log hub (10.0.0.6)

# OpenSearch cluster (runs on the 10.0.0.6 server itself, or via SSH tunnel)
curl -s localhost:9200/_cluster/health?pretty          # status: green/yellow/red
curl -s localhost:9200/_cat/indices?v # index list, detect abnormal index swelling

# Fluent Bit — is receiving/pushing logs
curl -s localhost:2020/api/v1/metrics | head -50        # HTTP monitor plugin, :2020

# Gamify backend
docker ps --filter name=app --format "table {{.Names}}\t{{.Status}}" # enough 4 instances :7003-7006 (pro) / 1 instance :4003 (dev)

Latest run results: V5 runs exactly 8 containers — gamifydeployments_app_1..4 (prod) + deployments_app-dev_1(dev, enough 5healthy, match scale compose item 9),opensearch (healthy), opensearch-dashboards, fluent-bit. For details on OpenSearch/Fluent Bit, see section 8.4.

11.6. End-to-end testing (quick checklist when a problem is suspected)

1. curl -sI https://api.kioskservice.club- Havecf-ray+ status 2xx/3xx? None ⇒ stops at DNS/Cloudflare, no need to SSH to any server.
2. SSH V1 → docker exec nginx-lb nginx -t && docker ps— gateway alive, valid syntax?
3. docker logs --tail 100 nginx-lb— at faultconnect() failedto the backend (points to which app server is dead)?
4. SSH into the suspected app server (V1/V2) →docker psenough 4 API instances + docker stats without OOM container/restart loop.
5. If API returns error regarding data → SSH V3, run 11.3 (Redis ping, Postgres pg_isready).
6. Need detailed log for further debugging → Dozzle (log-server1…4) for instant logs, or Dashboards (app-logs) to look up by time/keyword throughout the system.

12. Common operations

13. Risks & recommended

14. Appendix

14.1. Repo structurenginx/

nginx/ # infrastructure repo (infra/ops), does NOT contain app code
                                # original GitLab name: kiosk-deployments (dustin.le)
├── nginx.yml                   # compose container nginx-lb
├── nginx/
│ ├── nginx.conf # core configuration
│ ├── conf.d/ # 8 conf files ↔ 10 domains (section 5)
│   └── logrotate.conf
├── redis-master.yml / redis-slave.yml / redis-dev.yml
├── postgres-master.yml / postgres-slave.yml
├── rabbitmq.yml
├── opensearch.yml              # OpenSearch + Dashboards + Fluent Bit
├── fluent-bit/                 # fluent-bit.conf + tag_prefix.lua
├── dozzle.yml / redisinsight.yml / omnidb.yml
├── gamify-engine/              # script + env deploy (dev/, production/)
├── setup-*.sh, check-*.sh # operation script
├── docker-daemon.json # limit Docker logs daemon-wide
├── iplist                      # IP public 4 server
├── env, env-dev # ⚠ real env contains secret
└── README-*.md # documents each component

14.2. References in the repo

Documents created from repo/Users/nhim/dev/2026/kiosk/nginx+ Direct survey of 5 servers, updated 2026-07-04. Passwords/tokens in configuration files are intentionally omitted. Blueprint diagram:system-diagram.html· Risk handling plan section 13:comparison-and-migration-plan.html· target architecture:target-system.html.