[[PageOutline(2-4)]]
== Failover Testbed == 

One of the great improvements of Diameter is the '''failover''' mechanism. 

This mechanism works as follow: after a request has been sent or relayed by a node, if the connection it was sent over is teared down before the corresponding answer is received, the request must be sent again through another connection (when possible). The request will reach the final destination through a different route.

Coupled with the watchdog mechanism of Diameter, this failover mechanism ensures that all requests receive an answer (which can be an error). The main drawbacks are that the same request may be received several times by the server (duplicates), and that there is no limit of time for receiving the answer to a message, which can create congestion in the Diameter network.

The '''failback''' term means that after a connection has been restored, new messages are again sent to this peer.

Failover and failback are '''natively supported''' by the {{{freeDiameter}}} daemon.

This testbed involves 4 nodes in two different realms: a client ({{{nas}}} in realm {{{a}}}), two relays ({{{proxy}}} in realm {{{a}}}, and {{{proxy}}} in realm {{{b}}}), and a server ({{{serv}}} in realm {{{b}}}). 

The client and server are using the [wiki:test_app.fdx test_app] application to exchange messages. The routing configuration allows the messages to be conveyed through one of the relays, but not directly between the client and server. 

In the graph bellow, the blue route is the primary route, and green is secondary in case the primary route is down.

{{{
#!graphviz
graph G { 
    graph [bgcolor="#E8E8E8C8"];
    node [shape=box];
    nas [ 
         label="nas.a.rt.testbed.aaa"
         URL="#nas.a.rt.testbed.aaa"
    ]
    proxya [ 
         label="proxy.a.rt.testbed.aaa"
         URL="#proxy.a.rt.testbed.aaa"
    ]
    proxyb [ 
         label="proxy.b.rt.testbed.aaa"
         URL="#proxy.b.rt.testbed.aaa"
    ]
    serv [
         label="serv.b.rt.testbed.aaa"
         URL="#serv.b.rt.testbed.aaa"
    ]
    nas -- proxya -- serv [color=green]; 
    nas -- proxyb -- serv [color=blue]; 
    proxya -- proxyb [color=grey];
}
}}}

=== Screencast ===

The demonstration is split across two videos, because of its length.
It is recommended to watch the video in HD and fullscreen. 

==== First video: preparation of the testbed. ====
[[Embed(youtube=sTlunjatL_w&hd=1,w=660,h=525)]]

Here is what happens in this first video:
 * (0:00 ~ 0:19) Short explanation of the testbed structure.
 * (0:20 ~ 1:19) Start the two relays, check they are connected properly.
 * (1:20 ~ 1:49) Add the server, it connects to the two relays.
 * (1:50 ~ 2:44) Add the client, also connects to the two relays.
 * (2:45 ~ 2:59) Confirm the [wiki:dbg_monitor.fdx] output.
 * (3:00 ~ 3:27) Trig the [wiki:test_app.fdx] application.
 * (3:28 ~ 4:35) Confirm the basic behavior of Test application (see [wiki:TBSimple Simple Testbed] for detail).

==== Second video: Failover experiments. ====
[[Embed(youtube=9YyZpB8JHyA&hd=1,w=660,h=525)]]

This is the script of the second video:
 * (0:08 ~ 1:53) First test: stopping freeDiameter on a relay (notified failover).
   * (0:08 ~ 0:21) Explanation and test (stop freeDiameter on {{{proxy.b}}}).
   * (0:22 ~ 0:44) Confirm the result is as expected.
   * (0:45 ~ 1:10) Start {{{proxy.b}}} again and confirm the failback.
   * (1:10 ~ 1:53) Stop {{{proxy.a}}}, no effect, start it again.
 * (1:55 ~ 5:45) Second test: simulate a network failure (unnotified failover).
   * (1:55 ~ 2:45) Explanation
   * (2:46 ~ 2:54) Cut the (virtual) network links for {{{proxy.b}}}
   * (2:55 ~ 4:20) Observe and comment the reaction of the testbed.
   * (4:21 ~ 4:55) Reconnect the network, observe the REOPEN state and then failback.
   * (4:55 ~ 5:45) Confirm the failback by stopping {{{proxy.a}}} then restart it.
 * (5:45 ~ 6:56) Stop the server, then the proxy, and check the error code received is consistent.


=== Configuration files ===

Please note that these files may have been updated since the time the video was created. However, the purpose of the testbed remains the same.

==== Scripts ====
There is no script demonstrated in this video. Anyway, you might find the scripts from the [wiki:TBSimple Simple Testbed] useful also here.

==== nas.a.rt.testbed.aaa ====

These are the configuration files for freeDiameter on the client:
 [source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/CMakeFlags CMakeFlags]::
   The flags passed to {{{cmake}}} (on the command-line) when preparing the source. The noticeable ones are:
    * {{{-DBUILD_TEST_APP:BOOL=ON}}} : compile the [wiki:test_app.fdx] extension.
    * {{{-DBUILD_DBG_MONITOR:BOOL=ON}}} : compile the [wiki:dbg_monitor.fdx] extension.
    * {{{-DDEFAULT_CONF_PATH:PATH=/root/conf/freeDiameter}}} : Default configuration file path (avoids typing it on the command line).
    * Note that the [wiki:rt_default.fdx] extension is also built (by default).

 [source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/freeDiameter.conf freeDiameter.conf]::
   The main freeDiameter configuration file. It simply indicates to load the extensions and to connect to the relays (not the server directly).

 [source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/test_app.conf test_app.conf]::
   Configuration file for the [wiki:test_app.fdx] extension, indicating the routing AVPs for the Test message (Destination-Host and Destination-Realm).

 [source:VirtualTestbed/conf/nas.a.rt.testbed.aaa/freeDiameter/rtd.conf rtd.conf]::
   Configuration file for the [wiki:rt_default.fdx] extension. We indicate to the daemon that all messages can be sent to the local relay ({{{proxy.a.rt.testbed.aaa}}}) by default, if no better route is available.

==== proxy.a.rt.testbed.aaa ====

The following details of the configuration files should be highlighted:

 [source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/CMakeFlags CMakeFlags]::
    * {{{-DBUILD_DBG_RT:BOOL=ON}}} : compile the [wiki:dbg_rt.fdx] extension.
    * Note that the [wiki:acl_wl.fdx] extension is also built (by default).
    * Note also that the [wiki:test_app.fdx] extension is '''not''' built on the relay: the node does not need to understand fully the message in order to relay it.

 [source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/freeDiameter.conf freeDiameter.conf]::
    * The {{{NoRelay}}} directive is commented out here.
    * The only peer to which this relay actively tries to connect is the other peer. It does not initiate connections to the client or server.
    * It loads the [wiki:acl_wl.fdx] extension, that enables incoming connections from other peers.

 [source:VirtualTestbed/conf/proxy.a.rt.testbed.aaa/freeDiameter/acl_wl.conf acl_wl.conf]::
   The configuration file passed to the [wiki:acl_wl.fdx] extension. The meaning here is that any node that claims a Diameter Identity that matches the expression "*.rt.testbed.aaa" is allowed to connect to this peer.

==== proxy.b.rt.testbed.aaa ====

The configuration files are very similar to the other relay, the same explanation apply.

The file [source:VirtualTestbed/conf/proxy.b.rt.testbed.aaa/freeDiameter/CMakeFlags-altlibs CMakeFlags-altlibs] is an informational example on how to link {{{freeDiameter}}} against a non-default GNUTLS library.

==== serv.b.rt.testbed.aaa ====

The configuration files for the server can be found [source:VirtualTestbed/conf/serv.b.rt.testbed.aaa/freeDiameter/ in this folder]. There is nothing special to add here.

----