System Catalog › Gateways › INSSLR - Redundant HTTP Input Gateway with SSL Support

INSSLR - Redundant HTTP Input Gateway with SSL Support

Latest version: 2.0.2-1

Functional Overview

The INSSLR appliance is a layer-7 gateway for secure HTTP requests. It converts the requests to unencoded HTTP requests. This can be used whenever it is necessary to support secure HTTP on the client's side, but the back-end processing infrastructure does not or cannot support SSL, including:

• using a fast and light-weight HTTP server that does not have SSL support
• using multiple back-end servers, for performance or for redundancy, connected through a load-balancing switch
• using multiple back-end servers for unrelated functions, connected through a URL switch
• offloading the SSL encryption/decryption to a separate server to improve throughput

If configured, INSSLR also supports client certificate authentication. In the case of SSL mutual authentication, both the client and server exchange their certificates and corresponding public keys. The client may contact the certificate authority (CA Technologies) which issued the server's certificate and confirm the certificate is authentic before proceeding. The server may do likewise.

In its default configuration INSSLR acts as a layer-7 gateway for secure HTTP requests, also providing firewalled entry point for network traffic into an CA 3Tera AppLogic application, which can be configured with an Internet-accessible static IP address.

When configured, two INSSLR appliances can be used in failover mode to provide a redundant service. In this case, INSSLR IP address (configured via ip_addr) is running on only one of the nodes and is automatically transferred to the other INSSLR appliance in case of failure. At any given moment, only one of the INSSLR appliance is active. When running in failover mode INSSLR can be configured to run in several modes:

• Asymmetric - one of the appliances is primary and whenever it is online it takes over the IP address and becomes active. The backup appliance becomes active only if the primary node fails (or is stopped). Whenever the primary appliance becomes available again, it takes over the IP address and becomes active. The primary appliance is the one with the lowest IP address (string comparison) as set in the fover_local_ip property.
• Symmetric - both appliances are primary and whichever appliance starts first takes over the IP address and becomes active. The other appliance becomes active only if the first appliance fails (or is stopped). Whenever the first appliance becomes available again, it does not automatically take over the IP address.

When running in redundant mode, INSSLR provides an interface on its ctl terminal for:

• forcing INSSLR to become primary
• forcing INSSLR to become backup
• check the state of INSSLR (primary/backup)

INSSLR constantly monitors the health state of the backend appliance connected to its http terminal. The health state checks conducted by INSSLR may include a simple TCP connect check or a more complex HTTP request (specified on INSSLR's boundary). In the case of a failure of the connected appliance, INSSLR reports an error to the grid dashboard or, if in redundant mode and configured to do so, it does a failover to the backup INSSLR appliance.

To support applications that need to appear at a single IP address for more than one service, INSSLR can be configured to direct non-HTTP traffic transparently to a separate output terminal. For such connections, the appliance acts as a layer-3 firewall/NAT router.

Boundary

Resources

The default memory for the CA 3Tera AppLogic 2.7 release was changed to 128MB. The minimum memory remains 64MB.

Notes

• When using INSSLR in failover mode ( fover_mode is not none), the minimum memory is 100M.

The amount of memory given to INSSLR does not affect its throughput. Use more memory only to support more concurrent requests where the back-end servers may hold the requests for excessively long periods of time.

Terminals

Properties

Health check properties

Advanced Properties (used in failover scenarios)

Volumes

Error Messages

In case of appliance startup failure, the following errors may be logged to the system log:

Performance

Application Failover

INSSLR detects the failure of the other INSSLR in about 10 seconds. It takes an additional 10 seconds to failover to the other application. The total time from when the application failure first occurs to when the other application takes over the traffic is about 20 seconds.

Request Rate

INSSLR routes no less than 3000 transactions (request/response pairs) per second, subject to document size and network bandwidth available.

Data Throughput

INSSLR routes no less than 7 MBytes/second, subject to document size and network bandwidth available

Concurrent Connections

With its default memory INSSLR supports no less than 500 concurrently pending requests. (A pending request being an open TCP connection from the client, on which there is one or more un-completed HTTP request in progress). Maximum amount of concurrent connections depends of available free memory. Each 16 MB additional memory increases the number of concurrent http transactions by 500. For example, if you run in redundant mode and you want to be able to serve 2000 concurrent connections, you need 100M + 3*= 148M (minimum memory when running in redundant mode is 100M).

Notifications

When running in redundant mode (fover_mode is not none), INSSLR triggers notifications whenever it becomes active/passive. This is done on startup of the active node or whenever a failover occurs (each node send a notification that it became active/passive).

INSSLR send two notifications:

• A notification to the grid dashboard: INSSLR appliance APP_NAME:COMP_NAME <acquired|gave up> resource ip_addr
• An http request via the nfy terminal. It is up to the receiving end to take some action based on the notification. The requested URL is:

  http://nfy/fover_nfy_prefixfover_mode=fover_mode&state=<start|stop>&ip_addr=ip_addr&fover_local_ip=fover_local_ip&fover_remote_ip=fover_remote_ip&fover_netmask=fover_netmask

  You can use the fover_nfy_prefix to change the location of the remote script that is called or/and add additional parameters. Examples for fover_nfy_prefix values: /path/to/script.php?, /path/to/script.php?username=user&password=pass&.

Important!

• When using fover_nfy_prefix different than the default, help ensure it ends with ? if fover_nfy_prefix is just the path the script or & if fover_nfy_prefix includes additional parameters.
• If a node becomes unavailable it may not be able to send a notification that it became passive and only the node that became active sends notification.

The timeout for the http request is 5 seconds so that the failover is not slowed.

Health Check

INSSLR does a basic check on its http terminal. Errors are reported to grid dashboard (at maximum rate 1 per 10 minutes). If INSSLR is running in redundant mode and fover_on_healthcheck is set to non-zero value, INSSLR tries to do a failover to its backup appliance after fover_on_healthcheck number of subsequent health check failures. The failover may not succeed if the backup INSSLR appliance is not running or not configured properly.
Important! Setting fover_on_healthcheck to 1 makes INSSLR failover on each failed healthcheck, which may not be always desired. Using a higher value helps avoid false positives (like when stopping the application).

Server Certificates

To use SSL you need both the signed certificate and the private key it was encrypted with. The key and the certificate should be in PEM format and put in a single file specified by the cert_file property.

Generate Server Certificates

CA 3Tera AppLogic lets you generate server certificates.

To generate server certificates

1. Generate a private key using the following command:

   openssl genrsa -out privkey.pem 2048 
   

   To generate a pass protected key, use the following (To use the key with INSSLR you need a passwordless key, if you create a pass protected key you need to remove the password before using it in INSSLR)

   openssl genrsa -des3 -out privkey.pem 2048 
   

2. Next you need a certificate. You have two options here - create a certificate request and have it signed by a trusted CA (for which they will charge you), or create a self-signed certificate for test purposes (in this case browsers requesting your site will issue warnings that the certificate is not signed by a trusted CA).

   To generate a certificate request, execute the following:

   openssl req -new -key privkey.pem -out server.csr 
   

   After you send the .csr file to your trusted certificate authority, it will give you back a signed certificate ( .crt file) which you can use.

3. To generate a self signed certificate, execute the following:

openssl req -new -x509 -key privkey.pem -out server.crt -days 1095

Using Server Certificates

You can now put the certificate and the key in a file and use it in INSSLR:

cat privkey.pem  server.crt > server.pem 

If your key is password-protected, you can remove the password by executing the following:

openssl rsa -in key_with_pass.pem -out privkey.pem

Using Existing apache+mod_ssl Server Certificates

If you have an existing certificate that you use in Apache, you can use it in INSSLR too. Help ensure the key is not password-protected (see above) and put the private key and the certificate in one file in that order.

Example:

cat privkey.pem  server.csr > server.pem 

If you are using a chained certificate, you should also include the intermediate certificate provided by the issuer. Put the private key, the certificate and the intermediate certificate in one file in that order.

Example:

cat privkey.pem server.csr sf_issuing.crt > server.pem 

Important! The server signing key is your website's proof of identity. It is also vulnerable, because it is not password-encrypted (so that the appliance can read it without your assistance). Take the necessary measures to protect the key file, when installing it on the data volume. Do not use the same data volume for other purposes and don't make it visible to a web server, for example, to host Web-accessible data (HTML pages, scripts and such).

Client Certificates

The following example outlines how to create your own CA Technologies and use it to sign your own certificates using OpenSSL. This was tested using OpenSSL 0.9.8b, the same version as is installed in INSSLR.

Create a Certificate Authority

If you already have your own certificate authority used to create self signed server certificate, you can skip this step and use that certificate authority. When creating the instruments of the trusted authority for your application, it is important that the environment be secure.

To create a certificate authority

1. On a secure host system, create a working directory and a private directory within the working directory using the following commands:

   mkdir CA 
   mkdir CA/private 
   

2. Create a password-protected RSA key for the certificate authority with a 2048 bit key length:

   openssl genrsa -des3 -out CA/private/CA_key.pem 2048 
   

• Using the private key, create the public-key certificate for the certificate authority:

  openssl req -new -key CA/private/CA_key.pem -x509 -days 365 -out CA/CA_cert.pem 
  

Important! The private key for the certificate authority is the basis of trust for the application, so do not misplace it or provide it to other parties.

Create Client Certificates Signed by the Certificate Authority

CA 3Tera AppLogic lets you create client certificates that are signed by the certificate authority.

To create client certificates signed by the certificate authority

1. Generate an RSA private key (to create a password-protected key, use the -des3 switch):

   openssl genrsa -out clientA_privkey.pem 2048 
   

2. Generate a certificate signing request (CSR) from the private key:

   openssl req -new -key clientA_privkey.pem -out clientA_request.csr 
   

3. Sign the certificate contained in the CSR using the certificate generated for the CA:

   openssl x509 -req -days 365 -in clientA_request.csr -CA CA/CA_cert.pem -CAkey CA/private/CA_key.pem -CAcreateserial -out clientA.cer 
   

Be aware of the following:

• The -CAcreateserial switch enables the assignment of unique serial numbers to issued certificates. Because this is the first certificate issued by CA Technologies, the file CA/CA_cert.srl is created to track serial number assignments. In creating subsequent client certificates, the -CAserial switch is used in place of the -CAcreateserial switch. For example:

  openssl genrsa -out clientB_privkey.pem 2048 
  openssl req -new -key clientB_privkey.pem -out clientB_request.csr 
  openssl x509 -req -days 365 -in clientB_request.csr -CA CA/CA_cert.pem -CAkey CA/private/CA_key.pem -CAserial CA/CA_cert.srl -out clientB.cer 
  

• To create a single PEM formatted file which includes both the client certificate and key:

  cat clientA_privkey.pem clientA.cer > clientA.pem 
  

• To create the equivalent file in pkcs12 format (usable by most browsers):

openssl pkcs12 -export -in clientA.cer -inkey clientA_privkey.pem -out clientA.p12

Revoke Client Certificates Signed by the CA

To revoke a client certificate issued by the CA:

• openssl ca -config openssl.conf -revoke clientB.cer -crl_reason keyCompromise

Where an example "openssl.conf" is shown below and the "crl_reason" parameter is one of: unspecified, keyCompromise, CACompromise, affiliationChanged, superseded, cessationOfOperation, certificateHold or removeFromCRL. The matching of reason is case insensitive.

To re-generate the certificate revocation list (CRL):

• openssl ca -config openssl.conf -gencrl -out CA/crl.pem

In this example "crl.pem" is the file that should be provided to INSSLR as the "cert_revocation_list" property.

An sample config file used for the above examples is:

      ####################################################################
      [ ca ]
      default_ca      = CA_default            # The default ca section

      ####################################################################
      [ CA_default ]

      dir             = ./CA                  # Where everything is kept
      certs           = $dir/certs            # Where the issued certs are kept
      crl_dir         = $dir/crl              # Where the issued crl are kept
      database        = $dir/index.txt        # database index file.
      #unique_subject = no                    # Set to 'no' to allow creation of
                                        # several ctificates with same subject.
      new_certs_dir   = $dir/newcerts         # default place for new certs.

      certificate     = $dir/cacert.pem       # The CA certificate
      serial          = $dir/serial           # The current serial number
      crlnumber       = $dir/crlnumber        # the current crl number
                                        # must be commented out to leave a V1 CRL
      crl             = $dir/crl.pem          # The current CRL
      private_key     = $dir/private/cakey.pem# The private key
      RANDFILE        = $dir/private/.rand    # private random number file

      default_days    = 365                   # how long to certify for
      default_crl_days= 30                    # how long before next CRL
      default_md      = sha1                  # which md to use.
      preserve        = no                    # keep passed DN ordering

      policy          = policy_anything

      [ policy_anything ]
      countryName             = optional
      stateOrProvinceName     = optional
      localityName            = optional
      organizationName        = optional
      organizationalUnitName  = optional
      commonName              = supplied
      emailAddress            = optional

Create the Certificate Authority List Files Used by INSSLR

CA 3Tera AppLogic lets you create certificate authority list files that are used by INSSLR.

To create the file identified by the ca_list_client property

1. Access the following file:

   cat CA/private/CA_key.pem CA/CA_cert.pem > ca_list_client.pem 
   

2. Place this file either on the key volume or the volume which is nfs mounted via the fs terminal.
3. If desired, INSSLR's server certificate (for example, server.pem) can also be created in the same manner as the client certificates and likewise signed by the created certificate authority. Do not use a password for this certificate.
4. Once server.pem and ca_list_client.pem are in place, client certificate authentication can be tested as follows:
   • To test from a browser, import a pkcs12 formatted client certificate into Firefox using Tools, Options, View Certificates, Your Certificates, Import. Point the browser at the IP address of INSSLR.

• To test the SSL handshake from a remote host:

openssl s_client -host IP-address -port 443 -showcerts -ssl3 -cert clientA.cer -key clientA_privkey.pem -state

Client Certificate Headers

If a client connects to INSSLR using HTTPS, and if it presents a client certificate, INSSLR adds the following headers to the request it issues to the backend server:

• X-SSL-Subject details about the certificate owner.
• X-SSL-Issuer details about the certificate issuer (Certificate Authority).
• X-SSL-serial certificate serial number (decimal).
• X-SSL-cipher the cipher currently in use.
• X-SSL-certificate the full client certificate (PEM-format multi-line).

It is the application's responsibility to make use of these headers or not. INSSLR simply passes this information without verifying it in any way (except for signature and encryption correctness).

Typical Usage

Web Applications

To provide http(s) access to your application, connect the http terminal directly to the WEB appliance.

If you need a scalable web application hook the http terminal to a HALB appliance.

Configuration for Web Applications

The configuration examples list only properties set to non-default values and lack the network configuration (ip_addr, netmask, gateway).

Web Applications with Additional Services

If you have more services than just http in your application, you can use INSSLR to pass http traffic to its http terminal and use the aux terminal for other services.

Web Applications with More than One Additional Services

If you have multiple tcp/udp services and http, you can use INSSLR to pass http traffic to its http terminal and use the aux to feed the rest of the traffic to PS8, which feeds the desired traffic to the back end servers.

Redundant Web Applications

If you need to provide a redundant web application, you can copy the application and use INSSLR to provide failover capabilities for the external IP address.

Backup Web application

If you only want a backup application that will notify users for the downtime, you can use INSSLR to build a backup application that requires a minimum resources.

Appliances in use:

• user - input gateway for user requests
• web - web server displaying maintenance message

INSSLR on primary application:

INSSLR on backup application:

Redundant Web application (single application)

If you want to have your application run in redundant mode without creating a new application, you can simply double the appliances in it and run them in redundant mode. Because this involves using (at least) two web servers and two database appliances, in normal mode they are all used (providing scalability), but each of them is able to serve the application alone in case the other appliance fails. If you need additional scalability you can add more web and database appliances. In this example half of the appliances (in1, sw1, web1, db1) are running in one failover group and the rest (in2, sw2, web2, db2) are in another failover group so that the application can survive a server crash. This application provides redundancy at a very low cost as all of the appliances (except for one INSSLR and one HALB appliance which require very low resources) are in active state and no resources are spent for a backup application that runs only when the primary fails.

in1

in2

db1

db2

Redundant Web application (two identical applications)

You can run two identical applications on the same grid (but on separate servers so that failure of a server can only affect one of the application), or on different grids, provided that the IP address you use is accessible from both grids. The example below shows an application that uses a database appliance which is also running in redundant mode so in case of failure of one application, the other application can take over without data loss.

Appliances in use:

• in1 - redundant input gateway for user requests
• admin - input gateway for log files access
• sw - redirect port 8080 from admin to ui on db
• repl_in - input for the remote application to connect to the db appliance in order to replicate the database
• web_lb - web load-balancer for user requests
• web1, web2 - web servers with active content (for example, CGI scripts)
• db - MYSQLR64 configured to be both a master and a slave at the same time
• content - storage for database error log files, web content and web logs
• logs - storage for database error log files
• repl_out - output gateway for the db appliance to connect to the remote application in order to replicate the database
• mon - MON appliance

Client request arrive on the in1 gateway. The gateway forwards the requests to the web_lb load balancer, which directs the request to one of the web servers web1 and web2. The web servers access the db database. The db appliance connects to the remote application (which is an identical copy, the only difference being the server_id of db and the network setup) to replicate the database. The remote application connects to the db appliance via the repl_in gateway which is configured to allow connection only from the repl_out gateway of the remote application. The db appliances in the two applications are running in master-master setup so they always have identical data.

Example property configuration (properties that are not listed should be left to their default values):

Web access to db is available using admin gateway on port 8080.

in1

db

Important! Only one of the applications is active at any time, the other one is running simply for failover and is used when the active application fails.

Notes

INSSLR supports HTTP 1.0 and HTTP 1.1, but if the client identifies itself as MSIE, HTTP 1.1 support is turned off for that connection (to avoid bugs in some versions of MSIE).

If the client is not MSIE, INSSLR supports HTTP 1.1 for the client (including the multiple requests per TCP session ability), even if the back-end server uses supports only HTTP 1.0.

INSSLR supports a single external IP and therefore only a single SSL certificate may be used.

Open source and 3rd party software used inside the appliance

INSSLR uses the following 3rd party open source packages in addition to the 3rd party open source packages used by its base class LUX5.