Search Results for

    Show / Hide Table of Contents

    Akka.Remote Security

    Important Context: When You Need TLS

    Akka.Remote is designed for internal cluster communication and should NOT be exposed to the public internet. Most Akka.NET deployments run within:

    • Private networks (VPNs, VPCs)
    • Internal data centers
    • Kubernetes clusters with network policies
    • Behind firewalls with strict ingress rules

    When TLS Is Optional

    For many deployments, TLS is not strictly necessary:

    • Internal networks only - If your cluster runs entirely within a trusted network boundary
    • Development/staging environments - Where data sensitivity is low
    • Kubernetes with network policies - Where the container network provides isolation

    When TLS Is Recommended

    You should enable TLS when:

    • Crossing network boundaries - Communication between data centers or cloud regions
    • Public internet transit - Any traffic over public networks (even with VPN)
    • Compliance requirements - PCI-DSS, HIPAA, or other regulatory needs
    • Defense-in-depth - Additional security layer even on private networks
    • Multi-tenant environments - Shared infrastructure with other applications

    Security Layers

    Akka.Remote security operates on three complementary layers:

    1. Network Isolation - Using VPNs or private networks to restrict which machines can reach your actor systems
    2. Transport Encryption - Using TLS to encrypt all communication between nodes
    3. Authentication - Using mutual TLS to verify the identity of all connecting nodes

    You should use all three layers in production for defense-in-depth security.

    TLS (Transport Layer Security) Overview

    TLS encryption was introduced in Akka.NET v1.2 with the DotNetty transport. It provides:

    What TLS Protects Against:

    • Eavesdropping (all messages are encrypted)
    • Man-in-the-middle attacks (certificates verify server identity)
    • Network packet injection (cryptographic integrity checks)

    What TLS Does NOT Protect Against:

    • Misconfigured certificates (see startup validation below)
    • Compromised private keys (rotate certificates regularly)
    • Application-level authorization (implement this separately)

    Certificate Validation: Independent Control

    New in Akka.NET v1.5.52+: Certificate validation is now split into two independent settings for greater flexibility.

    Two Types of Validation

    1. Chain Validation (suppress-validation) - Validates certificate against trusted CAs
    2. Hostname Validation (validate-certificate-hostname) - Validates certificate CN/SAN matches target hostname

    These settings are independent and can be configured separately based on your deployment scenario.

    Chain Validation

    The suppress-validation setting controls whether the certificate chain is validated against trusted root CAs.

    Default Certificate Stores Used:

    When suppress-validation = false, .NET's SslStream validates certificates against the operating system's trusted root certificate stores:

    • Windows: Uses the Windows Certificate Store - specifically the Trusted Root Certification Authorities store
    • Linux: Uses the system's CA bundle (typically /etc/ssl/certs/ca-certificates.crt or /etc/pki/tls/certs/ca-bundle.crt)
    • macOS: Uses the Keychain Access Trusted Certificates

    The validation process follows RFC 5280 (X.509 PKI Certificate and CRL Profile) and RFC 6125 (Service Identity Verification).

    Enabled (Recommended)

    When suppress-validation = false (the default when SSL is enabled):

    What it validates:

    • Certificate chain against system trusted root CAs
    • Certificate expiration dates
    • Certificate hasn't been revoked (if CRL/OCSP configured)

    Does NOT validate:

    • Hostname matching (see Hostname Validation section below)

    When to use: Always in production and any networked environment.

    Disabled (Use With Caution)

    When suppress-validation = true:

    What it skips:

    • Certificate chain validation (accepts self-signed certificates)
    • Expiration date checks
    • CA trust checks

    When it's acceptable:

    • Local development on localhost only
    • Automated testing with self-signed test certificates
    • Initial TLS setup/debugging before obtaining proper certificates

    When it's NOT acceptable:

    • Any production environment
    • Any network-accessible environment (dev, staging, QA)
    • Any environment processing sensitive data
    • Any multi-tenant environment

    Validation Strategies: HOCON vs Programmatic (v1.5.52+)

    Two independent validation decisions determine your TLS security posture:

    1. Chain Validation - Verify certificate against trusted CAs (suppress-validation)
    2. Hostname Validation - Verify certificate CN/SAN matches target (validate-certificate-hostname)
    3. Mutual Authentication - Require both sides authenticate (require-mutual-authentication)

    Decision Matrix: Which Combination to Use

    Use Case suppress-validation validate-hostname mutual-auth Config Approach
    P2P Cluster (Default) false false true HOCON ✓ or Programmatic
    Client-Server with Shared Cert false true true HOCON ✓ or Programmatic
    Development/Testing true false false HOCON only
    Certificate Pinning false false true Programmatic required
    Custom Subject/Issuer Validation false false true Programmatic required

    HOCON Configuration Approach

    When validate-certificate-hostname = false (the default):

    • Skips hostname validation
    • Only validates certificate chain (if suppress-validation = false)
    • Best for: Mutual TLS with per-node certificates, IP-based connections, Kubernetes dynamic discovery

    When validate-certificate-hostname = true:

    • Certificate CN (Common Name) or SAN (Subject Alternative Name) must match the target hostname
    • Traditional TLS hostname validation as used in HTTPS
    • Best for: Client-server architectures with shared certificates and stable DNS names

    HOCON Example - P2P Cluster (Common Default):

    akka.remote.dot-netty.tcp {
      enable-ssl = true
      ssl {
        suppress-validation = false                    # Validate CA chain
        require-mutual-authentication = true           # Both sides authenticate
        validate-certificate-hostname = false          # Default: Allow per-node certs
        certificate {
          use-thumbprint-over-file = true
          thumbprint = "2531c78c51e5041d02564697a88af8bc7a7ce3e3"
        }
      }
    }
    

    HOCON Example - Client-Server with Hostname Validation:

    akka.remote.dot-netty.tcp {
      enable-ssl = true
      ssl {
        suppress-validation = false                    # Validate CA chain
        require-mutual-authentication = true           # Both sides authenticate
        validate-certificate-hostname = true           # Hostname must match
        certificate {
          use-thumbprint-over-file = true
          thumbprint = "2531c78c51e5041d02564697a88af8bc7a7ce3e3"
        }
      }
    }
    

    Programmatic Configuration Approach

    Use DotNettySslSetup with CertificateValidation helpers when you need:

    • Certificate pinning - Accept only specific certificates
    • Subject/Issuer validation - Custom certificate attribute checks
    • Custom business logic - Domain-specific validation rules
    • Dynamic validation - Load rules from runtime sources

    See Programmatic Certificate Validation below for detailed examples.

    Self-Signed Certificates: The Right Way

    If you must use self-signed certificates (development/testing):

    Option 1: Trust the Self-Signed CA (Better)

    # Generate self-signed CA
    $ca = New-SelfSignedCertificate -Subject "CN=Dev-CA" -CertStoreLocation Cert:\CurrentUser\My -KeyUsage CertSign
    
    # Export and import to Trusted Root
    Export-Certificate -Cert $ca -FilePath dev-ca.cer
    Import-Certificate -FilePath dev-ca.cer -CertStoreLocation Cert:\LocalMachine\Root
    
    # Generate server cert signed by CA
    New-SelfSignedCertificate -Subject "CN=localhost" -Signer $ca -CertStoreLocation Cert:\LocalMachine\My
    

    Configuration:

    akka.remote.dot-netty.tcp.ssl {
      suppress-validation = false  # ✓ Still validates, but trusts your CA
      certificate {
        use-thumbprint-over-file = true
        thumbprint = "server-cert-thumbprint"
      }
    }
    

    Pros:

    • Maintains validation checks
    • Catches expiration/configuration errors
    • More realistic test environment

    Option 2: Suppress Validation (Quick but Dangerous)

    akka.remote.dot-netty.tcp.ssl {
      suppress-validation = true  # ⚠️ Development ONLY
      certificate {
        path = "self-signed.pfx"
        password = "password"
      }
    }
    

    Pros:

    • Quick setup
    • No certificate installation needed

    Cons:

    • Doesn't catch real configuration errors
    • False sense of security
    • Easy to accidentally deploy to production

    WARNING: Never commit suppress-validation = true to version control for production configs. Use environment-specific configuration files.

    Certificate Configuration

    Option 1: Certificate File (Recommended for Development)

    akka.remote.dot-netty.tcp {
      enable-ssl = true
      ssl {
        suppress-validation = false  # IMPORTANT: Never use true in production!
        certificate {
          path = "path/to/certificate.pfx"
          password = "certificate-password"
          # Optional: Specify key storage flags
          flags = [ "exportable" ]
        }
      }
    }
    

    When to use: Development, testing, containerized environments where you can mount certificate files.

    Pros:

    • Easy to deploy with containers
    • Simple to version control (store path, not certificate)
    • Works well with configuration management tools

    Cons:

    • Certificate files can be copied if filesystem is compromised
    • Requires file system access for certificate deployment

    Option 2: Windows Certificate Store (Recommended for Production)

    akka.remote.dot-netty.tcp {
      enable-ssl = true
      ssl {
        suppress-validation = false
        certificate {
          use-thumbprint-over-file = true
          thumbprint = "2531c78c51e5041d02564697a88af8bc7a7ce3e3"
          store-name = "My"
          store-location = "local-machine"  # or "current-user"
        }
      }
    }
    

    When to use: Windows production environments, enterprise deployments with centralized certificate management.

    Pros:

    • Leverages Windows ACL for private key protection
    • Integrates with enterprise PKI infrastructure
    • Supports hardware security modules (HSM)
    • Private keys can be marked as non-exportable

    Cons:

    • Windows-specific (not portable to Linux)
    • Requires administrative access for certificate installation
    • More complex initial setup

    Finding Your Thumbprint:

    1. Open certlm.msc (Local Machine) or certmgr.msc (Current User)
    2. Navigate to Personal > Certificates
    3. Double-click your certificate
    4. Go to Details tab
    5. Scroll to Thumbprint field
    6. Copy the value (remove spaces)

    Programmatic Certificate Validation (v1.5.55+)

    New in Akka.NET v1.5.55: Certificate validation can now be configured programmatically using DotNettySslSetup with custom validators. This provides fine-grained control over validation logic while maintaining full backward compatibility with HOCON configuration.

    When to Use Programmatic Configuration

    Use programmatic setup when you need:

    • Custom validation logic - Implement domain-specific validation rules
    • Certificate pinning - Accept only specific certificates by thumbprint
    • Subject/Issuer validation - Verify certificate attributes
    • Dynamic configuration - Load validation rules from runtime sources
    • Composable validators - Combine multiple validation strategies

    CertificateValidation Helper Factory

    The CertificateValidation static class provides 7 helper methods for common validation patterns:

    Basic Chain Validation

    /// <summary>
    /// Example of programmatic mutual TLS setup using DotNettySslSetup with custom validation.
    /// This allows full programmatic control over certificate validation logic.
    /// </summary>
    public static void ProgrammaticMutualTlsSetup()
    {
        // Load or obtain your certificate
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Create custom validator combining multiple validation strategies
        var customValidator = CertificateValidation.Combine(
            // Validate the certificate chain
            CertificateValidation.ValidateChain(),
            // Also pin against known thumbprints for additional security
            CertificateValidation.PinnedCertificate(certificate.Thumbprint)
        );
    
        // Setup SSL with custom validator taking precedence over HOCON config
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: customValidator
        );
    }
    

    Certificate Pinning by Thumbprint

    Accept only certificates with specific thumbprints. Prevents man-in-the-middle attacks if CA is compromised:

    /// <summary>
    /// Example of certificate pinning - only accept certificates with specific thumbprints.
    /// Useful for preventing man-in-the-middle attacks with compromised CAs.
    /// </summary>
    public static void CertificatePinningSetup()
    {
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Allow only specific certificates by thumbprint
        var validator = CertificateValidation.PinnedCertificate(
            "2531c78c51e5041d02564697a88af8bc7a7ce3e3",  // Production cert
            "abc123def456789ghi012jkl345mno678pqr901stu"  // Backup cert
        );
    
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: validator
        );
    }
    

    Custom Validation Logic with ChainPlusThen

    Perform standard chain validation, then apply custom business logic:

    /// <summary>
    /// Example of custom certificate validation logic combined with standard validation.
    /// Allows complete control over what certificates are accepted.
    /// </summary>
    public static void CustomValidationLogicSetup()
    {
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Start with standard chain validation, then add custom logic
        var validator = CertificateValidation.ChainPlusThen(
            // Custom validation - check certificate subject matches expected peer
            (cert, chain, peer) =>
            {
                // Accept only certificates from authorized-peer
                if (cert?.Subject != null && cert.Subject.Contains("CN=authorized-peer"))
                {
                    return true;  // Accept this certificate
                }
                return false;  // Reject all others
            }
        );
    
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: validator
        );
    }
    

    Hostname Validation

    Enable traditional TLS hostname validation (certificate CN/SAN must match target hostname). Use for client-server architectures with shared certificates:

    /// <summary>
    /// Example of enabling traditional hostname validation for client-server architectures.
    /// Use when all nodes share the same certificate with matching CN/SAN.
    /// </summary>
    public static void HostnameValidationSetup()
    {
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Enable both chain validation and hostname validation
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            validateCertificateHostname: true  // Enable traditional TLS hostname validation
        );
    }
    

    Subject DN Validation

    Accept only certificates with specific subject names:

    /// <summary>
    /// Example of subject DN validation - only accept certificates with specific subject names.
    /// Useful for verifying peer identity based on certificate subject.
    /// Supports wildcards: "CN=Akka-Node-*" matches "CN=Akka-Node-001"
    /// </summary>
    public static void SubjectValidationSetup()
    {
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Accept certificates matching the subject pattern
        // Wildcards are supported: CN=Akka-Node-* matches CN=Akka-Node-001
        var validator = CertificateValidation.ValidateSubject(
            "CN=Akka-Node-*"  // Pattern to match
        );
    
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: validator
        );
    }
    

    CertificateValidation Helper Methods

    Method Purpose
    ValidateChain() CA chain validation with full error details
    ValidateHostname() Traditional TLS hostname validation (CN/SAN matching)
    PinnedCertificate() Certificate pinning by thumbprint whitelist
    ValidateSubject() Subject DN pattern matching (e.g., CN, O, OU)
    ValidateIssuer() Issuer DN pattern matching
    Combine() Compose multiple validators (AND logic)
    ChainPlusThen() Chain validation + custom business logic

    Custom Validator Precedence

    When both custom validators and HOCON config are present, custom validators take precedence:

    // This validator will be used regardless of HOCON suppress-validation setting
    var customValidator = CertificateValidation.ValidateChain(log);
    var sslSetup = new DotNettySslSetup(
        certificate: cert,
        suppressValidation: false,  // Ignored when customValidator provided
        customValidator: customValidator
    );
    

    This ensures programmatic validation logic always takes priority for explicit security requirements.

    Startup Certificate Validation (v1.5.52+)

    New in Akka.NET v1.5.52: The transport now validates certificate configuration at startup, preventing runtime failures.

    What It Validates

    The startup validation verifies:

    • Certificate exists in the specified location
    • Certificate has a private key associated
    • Application has permissions to access the private key
    • Private key is accessible for both RSA and ECDSA algorithms

    This fail-fast validation prevents runtime TLS handshake failures by detecting certificate configuration problems during system initialization.

    Common Private Key Permission Issues

    Symptom: "SSL certificate private key exists but cannot be accessed"

    Cause: Application user lacks permissions to the private key file in Windows certificate store.

    Solution: Grant private key access to your application user:

    # Find the certificate
    $cert = Get-ChildItem Cert:\LocalMachine\My | Where-Object {$_.Thumbprint -eq "YOUR_THUMBPRINT"}
    
    # Get private key file location
    $keyPath = $cert.PrivateKey.CspKeyContainerInfo.UniqueKeyContainerName
    $keyFullPath = "C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys\$keyPath"
    
    # Grant read permissions
    $acl = Get-Acl $keyFullPath
    $permission = "DOMAIN\AppUser","Read","Allow"
    $accessRule = New-Object System.Security.AccessControl.FileSystemAccessRule $permission
    $acl.AddAccessRule($accessRule)
    Set-Acl $keyFullPath $acl
    

    Understanding Mutual TLS (mTLS) vs Standard TLS (v1.5.52+)

    Akka.NET supports both standard TLS and mutual TLS (mTLS), configured via the require-mutual-authentication setting in the Validation Strategies section above.

    Visual Comparison

    Standard TLS (Server Authentication Only):

    sequenceDiagram
        participant Client
        participant Server
    
        Client->>Server: Connect (no certificate)
        Server->>Client: Send server certificate
        Client->>Client: Validate server certificate
        Client->>Server: Accept connection
        Note over Client,Server: Encrypted communication established
    

    Mutual TLS (Client + Server Authentication):

    sequenceDiagram
        participant Client
        participant Server
    
        Client->>Server: Connect with client certificate
        Server->>Client: Send server certificate
        Client->>Client: Validate server certificate
        Server->>Server: Validate client certificate
        Client->>Server: Accept connection
        Server->>Client: Accept connection
        Note over Client,Server: Mutually authenticated encryption established
    

    When to Enable Mutual TLS

    Enable mutual TLS (require-mutual-authentication = true) when:

    • All nodes are under your control (typical Akka.NET cluster) ✓ Recommended
    • You need defense-in-depth security
    • Compliance requires bidirectional authentication (PCI-DSS, HIPAA, etc.)
    • You want to prevent misconfigured nodes from joining

    Disable mutual TLS (require-mutual-authentication = false) when:

    • Clients cannot provide certificates (rare in Akka.NET)
    • You're using client-server architecture where clients are untrusted
    • Backward compatibility with older clients required

    Default is TRUE for security-by-default posture (since v1.5.52).

    Security Benefits of Mutual TLS

    1. Prevents Asymmetric Connectivity Issues

      • Without mTLS: A node with broken certificate can connect OUT to cluster (client TLS succeeds)
      • With mTLS: Node cannot connect without working certificate (enforced both ways)
    2. Defense-in-Depth

      • Startup validation prevents broken servers
      • Mutual TLS prevents broken clients
      • Both together provide complete protection
    3. Identity Verification

      • Every node must prove it owns the certificate
      • Prevents certificate theft attacks (attacker needs private key)

    For configuration examples in both HOCON and programmatic styles, see Validation Strategies and Programmatic Certificate Validation sections above.

    Configuration Examples and Security Analysis

    This section provides concrete examples of different security configurations and their tradeoffs.

    HOCON Configuration Security Levels

    Development/Testing Only (INSECURE):

    // WARNING: Development only - never use suppress-validation = true in production!
    public static Config DevelopmentTlsConfiguration = ConfigurationFactory.ParseString(@"
        akka.remote.dot-netty.tcp {
            enable-ssl = true
            ssl {
                suppress-validation = true  # INSECURE: Accepts any certificate
                require-mutual-authentication = false
                certificate {
                    path = ""self-signed-dev-cert.pfx""
                    password = ""password""
                }
            }
        }
    ");
    
    • ⚠️ suppress-validation = true accepts ANY certificate (self-signed, expired, invalid chains)
    • Vulnerable to man-in-the-middle attacks
    • No client authentication
    • Use only: Local development, never in networked environments

    Standard TLS (Medium-High Security):

    public static Config StandardTlsConfiguration = ConfigurationFactory.ParseString(@"
        akka.remote.dot-netty.tcp {
            enable-ssl = true
            ssl {
                suppress-validation = false
                require-mutual-authentication = false  # Server authentication only
                certificate {
                    path = ""path/to/certificate.pfx""
                    password = ""certificate-password""
                }
            }
        }
    ");
    
    • Server proves identity to clients
    • All traffic encrypted
    • Startup validation prevents misconfigurations
    • Use when: Mutual TLS is not feasible

    Mutual TLS with Windows Certificate Store (Maximum Security - RECOMMENDED):

    public static Config WindowsCertificateStoreConfiguration = ConfigurationFactory.ParseString(@"
        akka.remote.dot-netty.tcp {
            enable-ssl = true
            ssl {
                suppress-validation = false
                require-mutual-authentication = true
                certificate {
                    use-thumbprint-over-file = true
                    thumbprint = ""2531c78c51e5041d02564697a88af8bc7a7ce3e3""
                    store-name = ""My""
                    store-location = ""local-machine""  # or ""current-user""
                }
            }
        }
    ");
    
    • ✓ Both client and server prove identity
    • ✓ All traffic encrypted
    • ✓ Prevents misconfigured nodes from connecting
    • ✓ Private keys protected by Windows ACL
    • Use when: Production Akka.NET clusters (default recommended configuration)

    Mutual TLS for P2P Clusters with Per-Node Certificates:

    Refer to the Validation Strategies section for HOCON example showing P2P cluster setup.

    Client-Server with Hostname Validation:

    Refer to the Validation Strategies section for HOCON example with hostname validation enabled.

    Programmatic Configuration Security Levels

    For certificate pinning, subject/issuer validation, or custom logic, use programmatic setup:

    /// <summary>
    /// Example of programmatic mutual TLS setup using DotNettySslSetup with custom validation.
    /// This allows full programmatic control over certificate validation logic.
    /// </summary>
    public static void ProgrammaticMutualTlsSetup()
    {
        // Load or obtain your certificate
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Create custom validator combining multiple validation strategies
        var customValidator = CertificateValidation.Combine(
            // Validate the certificate chain
            CertificateValidation.ValidateChain(),
            // Also pin against known thumbprints for additional security
            CertificateValidation.PinnedCertificate(certificate.Thumbprint)
        );
    
        // Setup SSL with custom validator taking precedence over HOCON config
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: customValidator
        );
    }
    
    /// <summary>
    /// Example of certificate pinning - only accept certificates with specific thumbprints.
    /// Useful for preventing man-in-the-middle attacks with compromised CAs.
    /// </summary>
    public static void CertificatePinningSetup()
    {
        var certificate = new X509Certificate2("path/to/certificate.pfx", "password");
    
        // Allow only specific certificates by thumbprint
        var validator = CertificateValidation.PinnedCertificate(
            "2531c78c51e5041d02564697a88af8bc7a7ce3e3",  // Production cert
            "abc123def456789ghi012jkl345mno678pqr901stu"  // Backup cert
        );
    
        var sslSetup = new DotNettySslSetup(
            certificate: certificate,
            suppressValidation: false,
            requireMutualAuthentication: true,
            customValidator: validator
        );
    }
    

    See Programmatic Certificate Validation section for more examples.

    Untrusted Mode

    In addition to TLS, Akka.Remote supports "untrusted mode" which prevents clients from sending system-level messages:

    akka.remote {
      untrusted-mode = true
    
      # Whitelist specific actors that can receive remote messages
      trusted-selection-paths = [
        "/user/api-handler",
        "/user/public-endpoint"
      ]
    }
    

    When to enable:

    • You're exposing Akka.Remote to untrusted clients
    • You want to prevent remote actor creation/supervision
    • Defense against malicious remote commands

    Note: This does NOT replace TLS encryption. Use both together.

    Virtual Private Networks (VPNs)

    The best practice for network security is to make the network itself secure. Run Akka.Remote on private networks that require VPN access.

    Why VPNs matter:

    • Restricts who can even attempt to connect
    • Provides network-level access control
    • Adds authentication layer before TLS
    • Protects against network scanning/discovery

    VPN Options

    Self-Hosted:

    • WireGuard - Modern, fast, simple to configure
    • OpenVPN - Mature, widely supported

    Cloud Provider VPNs:

    • AWS Virtual Private Cloud (VPC)
    • Azure Virtual Networks (VNet)
    • Google Cloud VPC

    Managed Solutions:

    • Tailscale - Zero-config VPN mesh networking
    • ZeroTier - Software-defined networking

    Troubleshooting

    Error: "SSL Certificate Private Key Exists but Cannot Be Accessed"

    Cause: Application lacks permissions to private key file.

    Fix: Run PowerShell script above to grant permissions.

    Error: "The Remote Certificate Is Invalid According to the Validation Procedure"

    Cause: Certificate validation failed (expired, wrong CA, hostname mismatch).

    Fix:

    • Verify certificate is not expired: Get-ChildItem Cert:\LocalMachine\My
    • Check certificate CN/SAN matches hostname
    • For testing only: Set suppress-validation = true to identify if it's a validation issue

    Error: "TLS Handshake Failed" with No Client Certificate

    Cause: Server requires mutual TLS but client didn't provide certificate.

    Fix:

    • Ensure all nodes have require-mutual-authentication set consistently
    • Verify client certificate is configured correctly
    • Check client application has private key access

    Error: "RemoteCertificateNameMismatch" - Hostname Validation Failure

    Full error message:

    TLS certificate validation failed (full validation):
      - Certificate name mismatch
        - RemoteCertificateNameMismatch: The hostname being connected to does not match
          the hostname(s) on the server certificate.
    
    Certificate Details:
      Subject: CN=node1.example.com
      Issuer: CN=My-CA
      Valid: 2025-01-01 to 2026-01-01
    
    Connection target: 192.168.1.100:4053
    

    Cause: Certificate CN/SAN doesn't match the target hostname/IP address.

    Common scenarios:

    1. Connecting via IP but certificate has DNS name

      • Connecting to: 192.168.1.100
      • Certificate CN: node1.example.com
    2. Per-node certificates in P2P cluster

      • Node A cert CN: node-a.cluster.local
      • Node B cert CN: node-b.cluster.local
      • Each node's certificate doesn't match the other node's hostname

    Fix:

    Option 1 (Recommended for P2P clusters): Disable hostname validation

    akka.remote.dot-netty.tcp.ssl {
      validate-certificate-hostname = false  # Allow per-node certs
    }
    

    Option 2: Use certificates with matching CN/SAN

    # Ensure certificate CN matches connection target
    # For IP connections, add IP SAN to certificate:
    New-SelfSignedCertificate -Subject "CN=node1" `
      -DnsName "node1", "node1.example.com" `
      -TextExtension @("2.5.29.17={text}IPAddress=192.168.1.100")
    

    Option 3: Connect via DNS names that match certificate CN

    akka.remote.dot-netty.tcp {
      hostname = "node1.example.com"  # Must match cert CN
    }
    

    Error: "UntrustedRoot" - Certificate Chain Validation Failure

    Full error message:

    TLS/SSL certificate validation failed:
      - Certificate chain validation errors
        - UntrustedRoot: A certificate chain processed, but terminated in a root
          certificate which is not trusted by the trust provider.
    
    Certificate Details:
      Subject: CN=localhost
      Issuer: CN=localhost (self-signed)
    

    Cause: Certificate is self-signed or signed by untrusted CA.

    Fix:

    Option 1 (Development only): Suppress chain validation

    akka.remote.dot-netty.tcp.ssl {
      suppress-validation = true  # WARNING: Development only!
    }
    

    Option 2 (Recommended): Trust the CA certificate

    # Windows: Import CA to Trusted Root store
    Import-Certificate -FilePath ca.cer -CertStoreLocation Cert:\LocalMachine\Root
    
    # Linux: Add to system CA bundle
    sudo cp ca.crt /usr/local/share/ca-certificates/
    sudo update-ca-certificates
    

    Understanding TLS Error Messages (v1.5.52+)

    Since v1.5.52, TLS handshake failures provide detailed diagnostic information including:

    • Error category (chain validation, hostname mismatch, etc.)
    • Specific SSL policy error with explanation
    • Certificate details (subject, issuer, validity period)
    • Connection context (local/remote addresses)
    • Actionable recommendations

    Example comprehensive error:

    TLS handshake failed on channel [127.0.0.1:4053->127.0.0.1:54321](Id=...)
    
    Detailed TLS Error:
      - Certificate chain validation errors
        - UntrustedRoot: A certificate chain processed, but terminated in a root
          certificate which is not trusted by the trust provider.
      - Certificate name mismatch
        - RemoteCertificateNameMismatch: The hostname being connected to does not
          match the hostname(s) on the server certificate.
    
    Certificate Information:
      Subject: CN=node-test
      Issuer: CN=node-test (self-signed)
      Serial Number: 1A2B3C4D5E6F
      Valid From: 2025-01-01 00:00:00 UTC
      Valid To: 2026-01-01 00:00:00 UTC
      Thumbprint: 2531c78c51e5041d02564697a88af8bc7a7ce3e3
    
    Recommendations:
      - For development: Set 'suppress-validation = true' (testing only!)
      - For production: Install certificate in trusted root store
      - For hostname issues: Set 'validate-certificate-hostname = false' if using
        per-node certificates or IP-based connections
    

    Additional Resources

    • Windows Firewall Configuration Best Practices
    • TLS 1.2 Specification (RFC 5246)
    • OWASP Transport Layer Security Cheat Sheet

    Related:

    • Akka.Remote Configuration
    • DotNetty Transport
    In this article
    • githubEdit this page
    Back to top
    Contribute
    • Project Chat
    • Discussion Forum
    • Source Code
    Support
    • Akka.NET Support Plans
    • Akka.NET Observability Tools
    • Akka.NET Training & Consulting
    Maintained By
    • Petabridge - The Akka.NET Company
    • Learn Akka.NET