High Availability on Linux & Proxmox VE: Enterprise Cluster Architecture

Downtime Is Not an Accident — It Is an Architectural Decision

Before diving into technical details, let us cite figures that belong on the desk of every IT director and business owner. According to Gartner research, the average cost of one hour of IT infrastructure downtime in the SME sector ranges from $10,000 to $75,000 — depending on industry, scale of operations, and time of day. In industrial enterprises, scrap metal dealers, coal depots, and trading companies, the absolute figure may be lower, but the percentage impact on revenue is dramatically higher. In other words: one night of server failure can erase a week's worth of profit.

Over more than 23 years of AbejaIT's presence in the IT market, we observe the same pattern repeatedly. Companies invest in hardware, purchase licenses, deploy ERP systems — and then run them on a single server with no replication, no disaster recovery plan, and no monitoring. Until the first failure, everything works fine. After the first failure, the business owner asks why nobody told them this was not acceptable practice.

This article is the answer to that question — before the failure occurs.

What Is High Availability and Why 99.9% SLA Is Not Enough

High Availability (HA) is an IT systems architecture designed to eliminate or drastically reduce service unavailability. The key metric describing HA is the availability percentage expressed as annual uptime:

• 99.0% availability = 87.6 hours of downtime per year — catastrophic for any production business.
• 99.9% (three nines) = 8.7 hours of downtime per year — still too much for mission-critical systems.
• 99.99% (four nines) = 52 minutes of downtime per year — acceptable for SMEs with critical infrastructure.
• 99.999% (five nines) = 5.2 minutes of downtime per year — the standard for financial and industrial environments.

Achieving four and five nines without a proper cluster architecture is physically impossible. A single server, no matter how expensive, will eventually fail — a disk, a power supply, a network card, an OS update, or human error. Redundancy is not a luxury — it is an engineering requirement.

Proxmox VE as the Foundation of Enterprise HA Clusters

Proxmox Virtual Environment is an enterprise-class virtualization platform built on Linux (Debian), combining full virtualization (KVM) with containerization (LXC). In AbejaIT production environments, Proxmox VE serves as the backbone of cluster architectures for several key reasons:

• Open source — no vendor lock-in, full security auditability.
• Built-in cluster manager (Proxmox Cluster) — native multi-node support, quorum management, and replication.
• ZFS integration — advanced file system with native replication, compression, snapshots, and data integrity checksumming.
• Proxmox HA Manager — automatic node failure detection and live migration of virtual machines to healthy cluster nodes within 30–120 seconds.
• Ceph Storage — optional distributed storage system eliminating any single point of failure at the storage layer.

Minimum Production Architecture: A 3-Node Cluster

The absolute minimum for an HA environment is three physical nodes. Why three and not two? It is a quorum issue. With two nodes, the loss of connectivity between them (split-brain) prevents automatic failover decisions — neither node knows whether it has lost network connectivity or whether the other node has actually failed. The third node (or a lightweight quorum device — Proxmox Corosync QDevice) resolves this problem definitively.

A reference Proxmox VE cluster architecture for an SME looks as follows:

• Node-1, Node-2, Node-3 — three physical servers (e.g., 2× Intel Xeon, 256 GB RAM, 4× NVMe 2 TB in RAID-10 or ZFS RAIDZ2).
• Cluster network (Corosync) — dedicated 10 GbE interface exclusively for cluster traffic (heartbeat, node communication).
• ZFS replication network — second dedicated 10 GbE interface for data replication traffic (optionally merged with the cluster network in budget environments).
• Production network — 25 GbE or 10 GbE bonding for virtual machine traffic to the outside world.
• Shared storage or ZFS Replication — Ceph or ZFS replication between nodes (asynchronous, every 15 minutes or more frequently).

ZFS Replication — The Heart of the HA Architecture

ZFS (Zettabyte File System) is one of the most mature file systems available for Linux. In the context of a Proxmox cluster, its key advantages include:

• Atomic snapshots — a consistent point-in-time recovery snapshot without stopping the virtual machine.
• Incremental replication (ZFS send/receive) — transmitting only changed data blocks between nodes, minimizing network load.
• Data integrity verification — ZFS automatically detects and repairs silent data corruption — a phenomenon that in traditional file systems can go unnoticed for years.
• LZ4/ZSTD compression — disk space reduction of 20–60% in environments with textual data and databases.

Engineering note: ZFS replication in Proxmox VE operates asynchronously — this means the target node may lag a few minutes behind the source node. In environments requiring RPO (Recovery Point Objective) close to zero, consider synchronous Ceph or hardware storage solutions with synchronous replication.

Automatic Failover — How the Cluster Responds to Failure in Practice

Proxmox HA Manager monitors cluster node health via the Corosync protocol (heartbeat). When a node stops responding for a defined period (default ~30 seconds), the cluster initiates a fencing procedure — isolating the failed node — and migrates protected virtual machines to the remaining healthy nodes.

The automatic failover process step by step:

1. Node-1 stops responding to cluster heartbeat.
2. Node-2 and Node-3 reach quorum and confirm the failure.
3. Proxmox HA Manager issues a fencing command for Node-1 (IPMI/BMC power-off).
4. Virtual machines from Node-1 are started on Node-2 or Node-3 from the last available ZFS snapshot.
5. The entire process takes between 60 and 180 seconds depending on VM size and cluster configuration.

For comparison: manual administrator intervention during a classic server failure takes a minimum of 30–90 minutes (travel time, diagnosis, decision on reinstallation or restoration from backup).

3-2-1 Backup as a Mandatory Component of HA Strategy

HA does not replace backup — it complements it. A cluster protects against hardware failures and momentary unavailability. It does not protect against accidental data deletion, ransomware attacks, or application errors that corrupt the database simultaneously on all nodes.

The 3-2-1 backup strategy we implement in every AbejaIT production installation:

• 3 copies of data — production data plus 2 independent backup copies.
• 2 different media types — e.g., local NAS with ZFS disks plus cloud (AWS S3, Backblaze B2, dedicated off-site backup server).
• 1 off-site copy — outside the primary location, protected against fire, flooding, or physical theft.

In Proxmox environments, we use Proxmox Backup Server (PBS) — a dedicated solution with deduplication, AES-256 encryption, and backup integrity verification at every backup cycle — to implement the 3-2-1 policy.

24/7 Monitoring — Because a Cluster Without Observation Is a Cluster Without a Guarantee

An HA architecture without active monitoring is like a fire suppression system without smoke detectors. We can learn about a failure after the fact — or we can respond before it becomes a problem.

In AbejaIT-managed infrastructures, we apply multilayer monitoring based on:

• Zabbix / Prometheus + Grafana — node performance metrics (CPU, RAM, I/O, network), alerts on threshold breaches.
• Proxmox API monitoring — virtual machine status, ZFS replication health, Corosync cluster status.
• SMART disk monitoring — predictive detection of hard drive failures before physical damage occurs.
• Multi-channel alerting — SMS, email, webhook to ticketing systems (Jira, GitLab Issues, PagerDuty).
• Quarterly failover tests — simulated node failures in a test environment identical to production.

Details of our approach to 24/7 server monitoring are described in the dedicated services section — we encourage you to review our SLA offering.

Hybrid Cloud as an Extension of the On-Premise Cluster

An on-premise Proxmox cluster is an excellent solution for sensitive data, systems requiring low latency, and environments subject to legal regulations on data storage. However, modern enterprise HA architecture should also incorporate public cloud as:

• Off-site backup target — AWS S3 or Google Cloud Storage for encrypted backup copies.
• Disaster Recovery Site — ability to launch critical services in the cloud within minutes when the local data center is unavailable (flood, fire, prolonged power outage).
• Burst capacity — temporary scaling of computing resources during peak periods (inventory audits, fiscal year-end, sales campaigns).

HA in Practice: What to Protect First

Not all services require the same level of HA. Resource classification should precede architectural design:

• Tier 0 — Critical (RTO < 2 min, RPO ~ 0): Production databases, ERP systems, payment systems, weighing and billing systems.
• Tier 1 — Important (RTO < 30 min, RPO < 15 min): Mail servers, CRM, customer portals, reporting systems.
• Tier 2 — Standard (RTO < 4 h, RPO < 1 h): Development environments, internal systems, documentation.

Systems such as mojafirma.io — our cloud ERP system with KSeF integration — are designed with Tier 1 and Tier 0 in mind: customer data, electronic invoices, and financial settlements must remain continuously available, especially in the context of the approaching mandatory KSeF rollout for all Polish businesses.

Implementation Checklist: HA Readiness Audit

Before commissioning an HA architecture project, it is worth conducting a quick audit of the current infrastructure state. Here are the questions your IT team should be able to answer:

• ☑ Do production servers have redundant power supplies (UPS + generator)?
• ☑ Is there more than one compute node for critical services?
• ☑ Is backup tested regularly (at least quarterly)?
• ☑ Has the system restoration time from backup ever been measured?
• ☑ Does a written Disaster Recovery (DR) Plan exist?
• ☑ Does monitoring detect failures before they escalate?
• ☑ Does the file system in use protect against silent data corruption?
• ☑ Does the SLA agreement with the infrastructure provider define realistic response times?

If more than three of the above questions receive a "no" or "I don't know" answer — we invite you to schedule a complimentary IT infrastructure audit. In a single engineering session, we identify critical gaps and propose a remediation roadmap tailored to your company's budget.

Summary: HA Is a Business Strategy, Not Just a Technology

High availability of IT systems is no longer the exclusive domain of large corporations and financial institutions. Today, every business — a scrap metal dealer, a coal depot, an allotment garden association, a manufacturing company, an accounting office — operates on data and processes whose unavailability for even a few hours generates real, measurable losses. A cluster architecture based on Proxmox VE, ZFS replication, 24/7 monitoring, and a 3-2-1 backup strategy is available to SMEs at a reasonable budget.

At AbejaIT, we design and deploy such architectures across more than two decades of engineering practice. Our 23+ years of production engineering experience allow us to select solutions appropriate to business scale — without overpaying for unnecessary complexity and without cutting corners where cutting corners is not acceptable.

Contact our engineering team to discuss an HA architecture for your infrastructure. Learn more about our infrastructure services on the AbejaIT server administration and IT infrastructure page.

Source reference: Ars Technica