Compare each recommendation inSystem Design Guideline — Finance API Platform(Draft v1, July 3, 2026) with kioskgaming infrastructure running on Hetzner (live survey of 5 servers + Hetzner Console, July 4), and conversion roadmap. Target architecture after conversion: seeTarget System· Current status details:Deep Dive infrastructure · System diagram.
The guideline proposes four main changes. Comparing with real infrastructure gives the following results.
| Key recommendations of the guideline | Current Status | Conclude |
|---|---|---|
| 2nd Nginx + Cloudflare LB (remove SPOF) | Not yet | Priority #1.The reality is more serious than the guidelines describe — see section 04. |
| Idempotency key via Redis | Not yet | The work of the backend code (kioskgaming_backend), parallelizes phase 1. |
| PgBouncer transaction mode | Not yet | Need to audit connection immediately (phase 2), deploy phase 3. |
| Measure before adding compute | Measure once | Snapshot docker statson V2 (July 4): CPU <1%, ~90–100MB/instance — no compute bottlenecks. There is still a lack of peak hour data. |
| PM2 cluster in compute layer | Already have | Worked on DDoS Resilience Phase 1 — the guideline was written as if it didn't exist. |
| Postgres read replica "if needed" | Already have | postgres-slave on V4, streaming without lag (confirmed 07/04) — just missing read routing. |
| (In addition to guidelines)Network firewall matches the application layer | Deviated | Discovered on July 4:rule :80 Hetzner Firewall only whitelists 5/15 Cloudflare bands that nginx trusts — can be fixed immediately on Console, see Phase 0. |
Agree with the guideline: this is itsupplement, not destroy and rebuild. Much of the platform (Cloudflare edge, least_conn, health check upstream, Redis/Postgres replication, PM2 cluster) is on track. The missing work focuses on 3 places: SPOF nginx, idempotency, and connection control to Postgres.
Full details ininfrastructure-deep-dive.html.
| Server | Plan | Internal IP | Role |
|---|---|---|---|
| V1-Kiosk | CCX23 · 4 vCPU/16GB | 10.0.0.5 | nginx-lb :80 (the only gateway in the entire system) + API prod :3001–3004 + API dev :4001 |
| V2-Kiosk | CCX13 · 2 vCPU/8GB | 10.0.0.4 | API prod :3001–3004 + API dev :4001 — actual measurements: ~90–100MB/instance, low CPU |
| V3-Kiosk | CCX13 · 2 vCPU/8GB | 10.0.0.2 | Redis master :6379 + Postgres master :5432 |
| V4-Kiosk | CCX13 · 2 vCPU/8GB | 10.0.0.3 | 7 container: Redis slave + Redis dev + Postgres slave :5433 + RabbitMQ + RedisInsight + OmniDB + Dozzle |
| kiosk-gamify-engine | CCX23 · 4 vCPU/16GB | 10.0.0.6 | Gamify dev/pro (5 instance) + OpenSearch + Dashboards + Fluent Bit |
Request flow: client → HTTPS → Cloudflare (Flexible mode) → HTTP :80 →nginx-lbon V1 → upstreambackendleast_conn 8 instances (V1 + V2). Postgres and Redis replicated to V4, online without lag; failover is currently manual. 6 frontend sites (client/shop/agent/super-agent/admin/cashier) hosted onCloudflare Pages— outside the range of these 5 servers.
① Hetzner Firewall firewall-ssh(only in project, applies to 5/5 servers): rule:80 only whitelist 5/15 Cloudflare bands— real traffic via PoP in the 10 missing bands is DROP before reaching nginx;:22 open the entire Internet; data ports (6379/5432/9200…) default-deny ✓. ② All Dozzle containers (4/4 servers) reportunhealthy— image healthcheck error, not a problem. ③ V2 and V3 have the same hostnameubuntu-8gb-ash-1— Server identification must be by IP.
Compare each point in the guideline with the actual configuration.
| Recommended guidelines | Actual situation (verified source) | Status |
|---|---|---|
| Cloudflare at the edge blocks DDoS L3/4 | All domains*.kioskservice.clubafter Cloudflare; 15 stripsset_real_ip_from + CF-Connecting-IPin every conf (nginx/conf.d/*.conf) |
Achieved |
| 2 standalone Nginx + Cloudflare LB | 1 nginx-lbOnly on V1. Heavier than the guideline describes: V1 carries both the gateway and 4/8 API prod instances — if V1 dies, the gateway is lostandhalf an app at the same time |
SPOF |
least_conn + max_fails=3 + fail_timeout |
There is already enough in upstreambackend (conf.d/default.conf) |
Achieved |
keepaliveto upstream |
Being commented on —# keepalive 32; (default.conf:15) |
Turn off |
Rate limit limit_req/limit_connnginx layer |
Does not exist in any conf files (grep allconf.d/ + nginx.conf). Rate limit is currently only available at the backend layer (DDoS Phase 1) |
Not yet |
| Short timeout: connect 3s / read 8s | Currently 60s / 60s (default.conf:61-63) |
Too long |
PM2 instances= number of vCPUs, no more |
PM2 cluster is running (DDoS Phase 1). V2 is CCX132 vCPUs running 4 instances:3001–3004 — exceeds the number of cores. Snapshot 04/07: CPU <1%, RAM ~100MB/instance ⇒ no congestion, but 4 instances on 2 cores is still redundant (only costs context-switch, no additional throughput) | Measure once |
| (In addition to guidelines)The network layer firewall matches the nginx layer trust list | Deviation (Console, July 4):rule :80 only has 5/15 Cloudflare bands whileset_real_ip_from15-strip news; rule :443 is enough to get 15/15 even though nginx doesn't listen to 443; :22 opens Any IPv4+IPv6 |
Deviated |
| PgBouncer transaction pooling | Not yet.max_connections=100 (postgres-master.yml:31); 8 prod + 2 dev + gamify instances pointing to 1 master — pool 10/instance has hit the ceiling |
Not yet |
| Read replica "consider when necessary" | Already available: postgres-slave streaming replication from 10.0.0.2 (postgres-slave.yml) — guideline not known. Just missing the read route from app |
Already have |
| Redis: cache + idempotency key | Redis master-slave is running, using cache + rate limit backend. Idempotency keynot yet— is the work of the kioskgaming_backend code, not part of the nginx repo | Half |
| Do not use Kubernetes at this scale | Currently not using K8s — consensus remains the same | Agree |
The guideline is written for the general model "Cloudflare → 1 Nginx → app → DB" so it misses some realities of this system.
Redis master and Postgres masterlocated on the same CCX13 server. Slaves for both are available but failover is manual — no Sentinel (Redis) or Patroni/repmgr (Postgres). After deleting SPOF nginx, this is the next dead point of the system: V3 dies and all write paths are lost even though the app and gateway are still alive.
The guideline considers Nginx to be a separate node. In fact, V1 is both a system-wide gateway and runs 4 instances of API prod + API dev — an app layer resource problem on V1 brought down the gateway. Nginx #2 sits on top of V4 even though V4 is piggybacking7 container(Redis slave 1.5GB limit + Postgres slave shared_buffers 1GB + RabbitMQ 1GB + 4 tools) on 8GB RAM — because nginx is very light (a few tens of MB) andV4's failure-domain is better: loss of V4 = loss of gateway #2 + replica (write path still alive); put on V2, lose V2 = lose gateway #2 + half API prod at the same time — repeat the exact error structure of V1. Additional conditions: audit RAM V4 before installing (step 1A).
The guideline only discusses from Cloudflare to nginx, ignoring the firewall layer of the vertical infrastructure providerbeforenginx. In fact, this layer is deviating from the application layer: rule :80 is missing 10/15 Cloudflare bands (real traffic can be DROP depending on PoP - intermittent errors are difficult to access), :22 is open to the entire Internet. These are 2 fixes on Console in a few minutes, without touching the server — put them in Phase 0 below, before anything else.
Recommended guidelinesrate=5r/sfor "finance API calls sparse". Kioskgaming is a gaming platform — the calling client is much thicker (balance polling, game callback, session). Applying 5r/s directly will mistakenly block real users. Need to start loose (≈20r/s burst 40), observe the log and then gradually tighten. Same with 3s/8s timeout — report/game callback endpoints may be slower.
Rate limit + hardening nginx layer coincides with Edge L0–L1 ofDDoS-Resilience-Technical-Plan(pending). Should be combined into one editconf.d/, avoid two plans touching the same file.
Phase 0: fix firewall error detected on 04/07 — just Console operation, few minutes, do it immediately. Phase 1: remove SPOF nginx + Idempotency — highest business risk.
Console → Firewalls → firewall-ssh→ TCP rule :80 → add:173.245.48.0/20, 103.21.244.0/22, 103.22.200.0/22, 103.31.4.0/22, 141.101.64.0/18, 108.162.192.0/18, 190.93.240.0/20, 188.114.96.0/20, 197.234.240.0/22, 198.41.128.0/17— then rule :80 matches exactly 15 ranges ofset_real_ip_from. Apply immediately to all 5 servers, no need to restart. This is the jobis silently blocking real traffic, prioritize everything.
Rule :22 is opening Any IPv4+IPv6. If you have a fixed admin/VPN IP → limited to that range. If not yet → minimum confirmationPasswordAuthentication noin sshd_config all 5 servers + consider fail2ban; Record your decision so you don't forget.
Choose V4 becausefailure-domain: loss of V4 = loss of gateway #2 + replica, write path is alive; put on V2, lose V2 = lose gateway #2 + half API prod — repeat the structural error of V1. V4 is running 7 containers but nginx only takes a few dozen MB so it still fits, with the condition of checking first.
free -h + docker stats --no-streamon V4 — confirm there is ≥500MB of stable free RAM before installing.nginx.yml+ directorynginx/ (conf.d, htpasswd) sang V4 → docker compose -f nginx.yml up -d. Config uses internal IP 10.0.0.x so it runs intact, no changes. Notenetwork_mode: host: nginx listens for :80 on the host — V4 has nothing taking up :80 so there is no conflict.firewall-sshapplies to all 5 servers so :80 on V4already openedfor Cloudflare range — no further action required (correction 15/15 included in step 0A).curl -H "Host: api.kioskservice.club" http://178.156.211.230/from allowed IPs (or from V4 itself:curl -H "Host: api.kioskservice.club" http://localhost/).Two options, choose according to budget:
From here everything changesconf.d/Must sync both servers. Moresync-nginx.shgo to the repo: rsync conf →nginx -teach one → reload each one → stop immediately ifnginx -tfail. fail. The configuration difference between the two gateways is the most difficult operational error to trace later.
Not part of the nginx repo — separate tasks for the backend. Middleware reads headersIdempotency-Key, useSET key value NX EX ttlon Redis master for the deposit / withdraw / transfer endpoints. Note: must save bothresponse resultto return when there is a duplicate key (SETNX illustrated in the new guideline only blocks double processing, does not return old results). TTL 60–300s covers the client's retry cycle.
Measurement & audit — do not change compute before having data, in accordance with principle 05 of the guideline.
docker stats / pm2 monit. Question to answer: V2 (2 vCPUs) runs 4 instances: 3001–3004 — if CPU is not a bottleneck, reduce to 2 instances to free up RAM and reduce context-switching.
On master:SELECT count(*), usename, application_name FROM pg_stat_activity GROUP BY 2,3;. Calculate total = (8 prod instances + 2 dev + gamify) × pool size per instance, compared to ceilingmax_connections=100Subtract buffer 10–20 for backup/migration/monitoring. This number determines whether phase 3 should be done quickly or slowly.
keepalive 32 in upstream backend(remove commentsdefault.conf:15, moreproxy_http_version 1.1 + proxy_set_header Connection "").limit_req_zone≈20r/s burst 40 (do not use guideline 5r/s — see section 04), key according to$binary_remote_addr saureal_ip Cloudflare. Observe the log for 1–2 weeks and then gradually tighten.docker exec nginx-lb nginx -t → nginx -s reloadeach child.PgBouncer + other tasks for later, deployed according to phase 2 data.
Add servicespostgres-master.yml(or separate yml),pool_mode=transaction, max_client_conn=500, default_pool_size=20. App changes connection from :5432 → :6432 — gradually changes each instance, can rollback each one.
Transaction pooling does not support session-level prepared statements. Before migrating: check kioskgaming_backend ORM/driver (TypeORM / node-postgres) — or enablemax_prepared_statements(PgBouncer ≥ 1.21), or turn off prepared statements on the driver side. Skipping this step will cause random errors that are difficult to trace in production.
Report, transaction history, dashboard read from slave. Only do it when data 2B shows that the master has a large read load — the replica is already running so the only cost is to fix the connection string on the app side.
Minimum: document Redis/Postgres manual failover process (promote slave, change IP in backend env, order of operations). Advanced: Redis Sentinel; Postgres considers repmgr/Patroni when operations team is ready.
Limit the scope clearly so that the plan does not bloat.
max_connections Postgres— mask symptoms; PgBouncer is the correct fix.free -h, docker stats) — premise of 1A.sync-nginx.shgo to the nginx repo (step 1C).number_of_replicas: 0+ set retention/ISM for OpenSearch; Change hostname V2/V3/V4 to eliminate duplicates.All conf changes are compiled in the local repo first; Apply on the server always in orderdocker exec nginx-lb nginx -t → docker exec nginx-lb nginx -s reload(do not restart). Changes that affect data services (Postgres/Redis) only give commands for the operator to run themselves, with impact warnings.