Internal Engineering Document

Target System
Kioskgaming architecture after conversion

Infrastructure status when all plans are completedCompare & Conversion plan— used as a common goal for the whole team: each item below either already exists, or is associated with a specific step (0A→3C) in the plan.

Date of establishment:04/07/2026 Departure:Current status of production 5 server Principle:addition — do not destroy and rebuild Status:Target version v1

Target architecture diagram

Differences from the present are markedNEW. Anything not marked as remaining from the running system.

User / Client apps

6 frontends on Cloudflare Pages (client · shop · agent · super-agent · admin · cashier) + mobile/kiosk — HTTPS :443

Cloudflare — DNS · CDN · WAF · Load Balancing NEW · 1B

Health-check proactively 2 origins, automatically eliminating dead origins in ~15–60s.
SSL Flexible remains unchanged in the target version v1 — leaving Full (strict) unchanged, see section 06.

HTTP :80 → 2 origin

Hetzner Firewall firewall-ssh— patchedNEW · 0A/0B

:80 whitelist enough 15/15Cloudflare band (matchset_real_ip_from) · :22 admin/VPN IP limit · port data is still default-deny

nginx-lb #1 — V1 (10.0.0.5)

keep · least_conn · realip Cloudflare

NEW · 1A

nginx-lb #2 — V4 (10.0.0.3)

same conf set, sync withsync-nginx.sh (1C)

keepalive 32 (2C) · rate limit ≈20r/s (2C) · timeout 15s/30s (2C)

App tier — V1 + V2

10.0.0.5 · 10.0.0.4

8 instance API prod (least_conn)
2 instance API dev
Idempotency middleware NEW · 1D
Number of instances/core adjusted according to 2A measurements

NEW · 3A

Data tier — V3 (master) + V4 (replica)

10.0.0.2 · 10.0.0.3

PgBouncer :6432 (transaction mode)
Postgres 16 master :5432
Redis master :6379 (+ requirepass)
Slave PG :5433 / Redis :6379 on V4
Read-only → slave (3B, metric)
Runbook failover (3C)

Observability — V5 + 4 Dozzle

10.0.0.6

Fluent Bit → OpenSearch → Dashboards
replicas:0 + retention ISM NEW
Dozzle per-server, healthcheck fixed
RabbitMQ :5672 (V4, unchanged)

Target architecture — 2 gateways, application layer matched firewall, idempotency, PgBouncer. The remaining foundation remains the same.

How to read this diagram

Each labelNEW · xpoints to exactly one step in the conversion plan (0A, 0B, 1A…3C). When a step is completed, the corresponding category on this diagram changes from "target" to "as is" — when all labels are complete, the two documents merge into one.

Before / After — each index

"Before" is the current status on July 4, 2026. "After" is the status when the plan is completed.

Index	Before (current status)	After (destination)	Step
Gateway nginx	1 instance — SPOFon V1	2 instance V1 + V4, Cloudflare LB health-check	1A–1C
Firewall :80 vs nginx trust list	Deviation 5/15 vs 15/15— real traffic may be DROP	Match 15/15	0A
SSH :22	Open the entire Internet	Limit admin/VPN IP(or documented alternative hardening)	0B
Double-processing transactions on retry	There is no protection	Idempotency-Keyon every money endpoint	1D
Connection to Postgres	No control— 11 potential clients vs max 100	PgBouncer transaction mode, multiplex about ~20 real conn	2B, 3A
Redis auth	No password— rely only on firewalls	requirepass/masterauth— 2 independent layers	3C
Failover data (V3 dies)	Manual, not yet documented	Runbook promote slave(automation later)	3C
Nginx configuration: keepalive / rate limit / timeout	Off / none / 60s	Turn on 32 / ≈20r/s / 15s-30s— merged with DDoS Edge L0–L1	2C
OpenSearch	Permanently yellow, no retention— index from April 2026 intact	replicas:0, ISM deletes by date— green, disk is stable	toilet
Monitor container status	Dozzle 4/4 unhealthy fake— loss of value alarm	Healthcheck is correct— unhealthy = real problem	toilet
Identify the server when SSH	V2/V3 have the same hostname	Unique hostname V1…V5	toilet

Server layout after conversion

Do not rent additional servers. The only change in layout: V4 gets nginx-lb #2 and PgBouncer placed next to Postgres master on V3.

Server	Target role	Change from now
V1 · CCX23 · 10.0.0.5	Gateway #1 + App	No change in ingredients; less "unique" — allowed to maintain/deploy errors and still have the system live
V2 · CCX13 · 10.0.0.4	App	The number of instances can be reduced by 4→2 according to the 2A (2 vCPU) measurement results; hostname changed to V2-Kiosk
V3 · CCX13 · 10.0.0.2	Data master	+ PgBouncer :6432 master edge; Redis adds requirepass; hostname V3-Kiosk
V4 · CCX13 · 10.0.0.3	Gateway #2 + Replica + Tools	+ nginx-lb #2 (:80 — after RAM audit, step 1A); hostname V4-Kiosk
V5 · CCX23 · 10.0.0.6	Gamify + Log hub	OpenSearch: replicas:0 + ISM retention; nothing else changed

Resource constraints to monitor

V3 and V4 are two servers that "fit" 8GB RAM: V3 = Redis 1.5GB + Postgres buffers/cache ~4GB; V4 = 7 existing containers + new nginx. nginx only takes up a few dozen MB so it's okay, butAny service added to these two servers after the target version must recalculate the total RAM first— that's why step 1A has a RAM audit precondition.

Behavior when losing each server

"Who dies, what is lost" table — before and after the conversion. This is a direct measure of plan value.

Problem	Consequences BEFORE	Consequences FOLLOW
V1 died	Whole system collapse— lost only gateway, all domains died even though V2–V5 was still alive	The system is still running— Cloudflare LB switches to gateway #2 (V4) in ~15–60s; Temporarily lost 4/8 API instances, leaving 4 on V2 to carry
V2 died	Halve API— least_conn converges to V1	As before— unchanged (V2 only runs the app)
V3 died	Lost all write paths— promote slave manually, no pre-written process, unpredictable recovery time	Still lost writebut recovery according to 3C runbook is available step by step (promote PG slave :5433, change Redis replicaof, point back to backend env) — recovery time is measurable
V4 died	Lost replica + tools— does not affect users	Lost gateway #2 + replica— Cloudflare LB redirects to gateway #1, users are not affected; Temporarily lost the ability to failover data until V4 recovers
V5 died	Lost gamify + focus log— is the same in both states; gamify is the specific SPOF of the gamification feature (accepted in target version v1, see section 06)

Bottom line

This conversion is eraseableworst scenario(V1 dies = whole system collapses) and turns the second worst scenario (V3 dies) from "unpredictable" to "processed". The death of the remaining two servers only causes local deterioration - the system still serves users.

Acceptance criteria

Conversion is considered complete when the list below is ticked - each item has a specific command/test scenario, not accepted by feel.

Test the gateway: stop nginx-lbon V1 (docker stop nginx-lb, off-peak hours) →curl https://api.kioskservice.clubstill pay 2xx within 60s (pass through V4). Turn it back on, repeat in reverse.
Firewall matches:count source rule :80 on Console = exactly 15 ranges, match each range withset_real_ip_from in default.conf.
Idempotency:Shoot 2 duplicate deposit requestsIdempotency-Key(dev environment) → exactly 1 transaction is recorded, the second response returns the first result.
PgBouncer: SHOW POOLS;on :6432 there is traffic;pg_stat_activityOn the master, count ≤ 30 real connections when the app is running at full load.
Sync 2 gateway:runsync-nginx.shwith a try change → conf on V1 and V4 are byte-for-byte identical (diff), bothnginx -t pass.
Runbook failover:Rehearse promoting the PG slave on the dev/staging environment at least once, recording the actual time.
OpenSearch: _cluster/health= green; Index older than N days (according to policy) automatically disappears.
Clean monitoring: docker psAll 5 servers no longer have containersunhealthyno — every unhealthy from now on is a real problem.

Outside the target version v1

Make it clear so no one mistakenly thinks they've been forgotten — these are upgrade steps intentionally left for later, with activation conditions.

Category	Reason to leave	Activates when
Cloudflare Full (strict) — TLS to origin	Need to install Origin CA cert + nginx listen 443 on both gateways; Flexible is still running fine and the firewall has been turned off	After the 2 gateways operate stably; or compliance requirements
Auto-failover data (Sentinel / Patroni)	Need ≥3 nodes for quorum + operating skills; Manual runbook is the right stepping stone	The runbook drill showed that the manual RTO was unacceptable
HA for V5 (gamify + log)	Local SPOF of a feature, not core money-path	When gamification becomes a key revenue feature
Kubernetes	Both the guideline and the comparison are wrong — operating costs are not commensurate with the scale	The number of services/teams has increased many times
OpenSearch enables security plugins	Port has default-deny from the Internet; lower priority than retention	When there is a request for internal audit / more people accessing Dashboards