Special Features¶
Synchronizing Supvisors instances¶
The INITIALIZATION
state of Supvisors is used as a synchronization phase so that all Supvisors instances
are mutually aware of each other.
The following options defined in the rpcinterface extension point of the Supervisor configuration file are particularly used for synchronizing multiple instances of Supervisor:
supvisors_list
;
core_identifiers
;
internal_port
;
synchro_timeout
;
auto_fence
.
Once started, all Supvisors instances publish the events received, especially the TICK
events that are
triggered every 5 seconds, on their Publish socket bound to the internal_port
.
On the other side, all Supvisors instances start a thread that subscribes to the internal events
through an internal Subscribe socket connected to the internal_port
of all Supvisors instances
of the supvisors_list
.
At the beginning, all Supvisors instances are declared in an UNKNOWN
state.
When the first TICK
event is received from a remote Supvisors instance, a hand-shake is performed
between the 2 Supvisors instances. The local Supvisors instance:
sets the remote Supvisors instance state to
CHECKING
;performs a couple of XML-RPC to the remote Supvisors instance:
supvisors.get_master_identifier()
andsupvisors.get_supvisors_state()
in order to know if the remote instance is already in an established state ;
supvisors.get_instance_info(local_identifier)
in order to know how the local Supvisors instance is perceived by the remote Supvisors instance.
At this stage, 2 possibilities:
the local Supvisors instance is seen as
ISOLATED
by the remote instance:
the remote Supvisors instance is then reciprocally set to
ISOLATED
;the URL of the remote Supvisors instance is disconnected from the Subscribe socket ;
the local Supvisors instance is NOT seen as
ISOLATED
by the remote instance:
a
supervisor.getAllProcessInfo()
XML-RPC is requested to the remote instance ;the processes information is loaded into the internal data structure ;
the remote Supvisors instance is finally set to
RUNNING
.
When all Supvisors instances are identified as RUNNING
or ISOLATED
, the synchronization is completed.
Supvisors then is able to work with the set (or subset) of Supvisors instances declared in supvisors_list
.
However, it may happen that some Supvisors instances do not publish (very late starting, no starting at all,
system down, network down, etc). Each Supvisors instance waits for synchro_timeout
seconds to give a chance
to all other instances to publish. When this delay is exceeded, all the Supvisors instances that are not
identified as RUNNING
or ISOLATED
are set to:
SILENT
if Auto-Fencing is not activated ;
ISOLATED
if Auto-Fencing is activated.
Another possibility is when it is predictable that some Supvisors instances may be started later.
For example, the pool of nodes may include servers that will always be started from the very beginning and consoles
that may be started only on demand.
In this case, it would be a pity to always wait for synchro_timeout
seconds. That’s why the core_identifiers
attribute has been introduced so that the synchronization phase is considered completed
when a subset of the Supvisors instances declared in supvisors_list
are RUNNING
.
Whatever the number of available Supvisors instances, Supvisors elects a Master among the active Supvisors
instances and enters the DEPLOYMENT
state to start automatically the applications.
By default, the Supvisors Master instance is the Supvisors instance having the smallest deduced name among all
the active Supvisors instances, unless the attribute core_identifiers
is used. In the latter case, candidates
are taken from this list in priority.
Important
About late Supvisors instances
Back to this case, here is what happens when a Supvisors instance is started while the others are already in
OPERATION
.
During the hand-shake, the local Supvisors instance gets the Master identified by the remote Supvisors.
That confirms that the local Supvisors instance is a late starter and thus the local Supvisors instance adopts
this Master too and skips the synchronization phase.
Auto-Fencing¶
Auto-fencing is applied when the auto_fence
option of the rpcinterface extension point is set.
It takes place when one of the Supvisors instances is seen as inactive (crash, system power down, network failure)
from the other Supvisors instances.
In this case, the running Supvisors instances disconnect the corresponding URL from their subscription socket.
The Supvisors instance is marked as ISOLATED
and, in accordance with the program rules defined,
Supvisors may restart somewhere else the processes that were eventually running in that Supvisors instance.
If the incriminated Supvisors instance is restarted, the isolation doesn’t prevent the new Supvisors instance to receive events from the other instances that have isolated it. Indeed, it has not been considered so far to filter the subscribers from the Publish side.
That’s why the hand-shake is performed in Synchronizing Supvisors instances. Each newly arrived Supvisors instance asks to the others if it has been previously isolated before taking into account the incoming events.
In the case of a network failure, the same mechanism is of course applied on the other side. Here comes the premises of a split-brain syndrome, as it leads to have 2 separate and identical sets of applications.
If the network failure is fixed, both sets of Supvisors are still running but do not communicate between them.
Attention
Supvisors does NOT isolate the nodes at the Operating System level, so that when the incriminated nodes become active again, it is still possible to perform network requests between all nodes, despite the Supvisors instances do not communicate anymore.
Similarly, it is outside the scope of Supvisors to isolate the communication at application level. It is the user’s responsibility to isolate his applications.
Extra Arguments¶
Supervisor users have requested the possibility to add extra arguments to the command line of a program without having to update and reload the program configuration in Supervisor.
Indeed, the applicative context is evolving at runtime and it may be quite useful to give some information to the new process (options, path, URL of a server, URL of a display, etc), especially when dealing with distributed applications.
Supvisors introduces new XML-RPCs that are capable of taking into account extra arguments that are passed to the command line before the process is started:
supvisors.start_args
: start a process in the local Supvisors instance ;
supvisors.start_process
: start a process using a starting strategy.
Note
The extra arguments of the program are shared by all Supvisors instances. Once used, they are published through a Supvisors internal event and are stored directly into the Supervisor internal configuration of the programs.
In other words, considering 2 Supvisors instances A and B, a process that is started in Supvisors instance A with extra arguments and configured to restart on node crash (refer to Running Failure strategy). if the Supvisors instance A crashes (or simply becomes unreachable), the process will be restarted in the Supvisors instance B with the same extra arguments.
Attention
A limitation however: the extra arguments are reset each time a new Supvisors instance connects to the other ones, either because it has started later or because it has been disconnected for a while due to a network issue.
Starting strategy¶
Supvisors provides a means to start a process without telling explicitly where it has to be started, and in accordance with the rules defined for this program.
Choosing a Supvisors instance¶
The following rules are applicable whatever the chosen strategy:
the process must not be already in a running state in a broad sense, i.e.
RUNNING
,STARTING
orBACKOFF
;the process must be known to the Supervisor of the Supvisors instance ;
the Supvisors instance must be
RUNNING
;the Supvisors instance must be allowed in the
identifiers
rule of the process ;the load of the node where multiple Supvisors instances may be running must not exceed 100% when adding the
expected_loading
of the program to be started.
The load of a Supvisors instance is defined as the sum of the expected_loading
of each process running in this
Supvisors instance.
The load of a node is defined as the sum of the loads of the Supvisors instances that are running on this node.
When applying the CONFIG
strategy, Supvisors chooses the first Supvisors instance available in the
supvisors_list
.
When applying the LESS_LOADED
strategy, Supvisors chooses the Supvisors instance in the supvisors_list
having the lowest load.
The aim is to distribute the process load among the available Supvisors instances.
When applying the MOST_LOADED
strategy, Supvisors chooses the Supvisors instance in the supvisors_list
having the greatest load.
The aim is to maximize the loading of a Supvisors instance before starting to load another Supvisors instance.
This strategy is more interesting when the resources are limited.
When applying the LESS_LOADED_NODE
strategy, Supvisors chooses the Supvisors instance in the supvisors_list
having the lowest load on the node having the lowest load.
When applying the MOST_LOADED_NODE
strategy, Supvisors chooses the Supvisors instance in the supvisors_list
having the greatest load on the node having the greatest load.
When applying the LOCAL
strategy, Supvisors chooses the local Supvisors instance.
A typical use case is to start an HCI application on a given console, while other applications / services may be
distributed over other nodes.
Attention
A consequence of choosing the LOCAL
strategy as the default starting_strategy
in the rpcinterface extension point is that all programs will be started on the Supvisors Master instance.
Note
When a single Supvisors instance is running on each node, LESS_LOADED_NODE
and MOST_LOADED_NODE
are
strictly equivalent to LESS_LOADED
and MOST_LOADED
.
Starting a process¶
The internal Starter of Supvisors applies the following logic to start a process:
supvisors.start_args(namespec)
XML-RPC to the chosen Supvisors instanceThis single job is considered completed when:
a
RUNNING
event is received and thewait_exit
rule is not set for this process ;an
EXITED
event is received with an expected exit code and thewait_exit
rule is set for this process ;an error is encountered (
FATAL
event,EXITED
event with an unexpected exit code) ;no
STARTING
event has been received 2 ticks after the XML-RPC ;no
RUNNING
event has been received X+2 ticks after the XML-RPC, X corresponding to the number of ticks needed to cover thestartsecs
seconds of the program definition in the Supvisors instance where the process has been requested to start.
This principle is used for starting a single process using a supvisors.start_process
XML-RPC.
Attention
About using the wait_exit rule
If the process is expected to exit and does not exit, it will block the Starter until Supvisors is restarted.
Starting an application¶
The application start sequence is re-evaluated every time a new Supvisors instance becomes active in Supvisors. Indeed, as explained above, the internal data structure is updated with the programs configured in the new Supervisor instance and this may have an impact on the application start sequence.
The start sequence corresponds to a dictionary where:
the keys correspond to the list of
start_sequence
values defined in the program rules of the application ;the value associated to a key contains the list of programs having this key as
start_sequence
.
Important
Only the Managed applications can have a start sequence, i.e. only those that are declared in the Supvisors Supvisors’ Rules File.
The programs having a start_sequence
lower or equal to 0 are not considered in the start sequence, as they are
not meant to be automatically started.
The internal Starter of Supvisors applies the following principle to start an application:
start_sequence
This principle is used for starting a single application using a supvisors.start_application
XML-RPC.
Starting all applications¶
When entering the DEPLOYMENT
state, all Supvisors instances evaluate the global start sequence using
the start_sequence
rule configured for the applications and processes.
The global start sequence corresponds to a dictionary where:
the keys correspond to the list of
start_sequence
values defined in the application rules ;the value associated to a key is the list of application start sequences whose applications have this key as
start_sequence
.
The Supvisors Master instance starts the applications using the global start sequence. The following pseudo-code explains the logic used:
start_sequence
Note
The applications having a start_sequence
lower or equal to 0 are not considered, as they are not meant to be
automatically started.
Important
When leaving the DEPLOYMENT
state, it may happen that some applications are not started properly
due to missing relevant Supvisors instances.
When a Supvisors instance is started later and is authorized in the Supvisors ensemble, Supvisors transitions
back to the DEPLOYMENT
state and tries to repair such applications. The applications are not restarted.
Only the stopped processes are considered.
May the new Supvisors instance arrive during a DEPLOYMENT
or CONCILIATION
phase, the transition to the
DEPLOYMENT
state is deferred until the current deployment or conciliation jobs are completed.
It has been chosen NOT to transition back to the INITIALIZATION
state to avoid a new synchronization phase.
Starting Failure strategy¶
When an application is starting, it may happen that any of its programs cannot be started due to various reasons:
the program command line is wrong ;
third parties are missing ;
none of the Supvisors instances defined in the
identifiers
of the program rules are started ;the applicable Supvisors instances are already too much loaded ;
etc.
Supvisors uses the starting_failure_strategy
option of the rules file to determine the behavior to apply
when a required
process cannot be started. Programs having the required
set to False are not considered as
their absence is minor by definition.
Possible values are:
ABORT
: Abort the application starting ;
STOP
: Stop the application ;
CONTINUE
: Skip the failure and continue the application starting.
Running Failure strategy¶
The autorestart
option of Supervisor may be used to restart automatically a process that has crashed
or has exited unexpectedly (or not). However, when the node itself crashes or becomes unreachable,
the other Supervisor instances cannot do anything about that.
Supvisors uses the running_failure_strategy
option of the rules file to warm restart a process that was
running on a node that has crashed, in accordance with the default starting_strategy
set in the
rpcinterface extension point and with the supvisors_list
program rules set in the Supvisors’ Rules File.
This option can be also used to stop or restart the whole application after a process crash. Indeed, it may happen that some applications cannot survive if one of their processes is just restarted.
Possible values are:
CONTINUE
: Skip the failure and the application keeps running ;
RESTART_PROCESS
: Restart the lost process on another Supvisors instance ;
STOP_APPLICATION
: Stop the application ;
RESTART_APPLICATION
: Restart the application ;
SHUTDOWN
: Shutdown Supvisors (i.e. all Supvisors instances) ;
RESTART
: Restart Supvisors (i.e. all Supvisors instances).
Important
The RESTART_PROCESS
is NOT intended to replace the Supervisor autorestart
for the local Supvisors
instance.
Provided a program definition where autorestart
is set to false
in the Supervisor configuration
and where the running_failure_strategy
option is set to RESTART_PROCESS
in the Supvisors rules file,
if the process crashes, Supvisors will NOT restart the process.
Note
Given that this option is set on the program rules, program strategies within an application may be incompatible
in the event of multiple failures. That’s why priorities have been set on this strategy.
STOP_APPLICATION
supersedes RESTART_APPLICATION
, which itself supersedes RESTART_PROCESS
and finally
CONTINUE
. So if a program with the RESTART_APPLICATION
option fails at the same time that a program
of the same application with the STOP_APPLICATION
option, only the STOP_APPLICATION
will be applied.
When the RESTART_PROCESS
strategy is evaluated, if the application is fully stopped - supposedly because of the
failure -, Supvisors will promote the RESTART_PROCESS
into RESTART_APPLICATION
. The idea is to benefit
from a full start sequence at application level rather than uncorrelated program restarts in the event of multiple
failures within the same application.
Stopping strategy¶
Supvisors provides a means to stop a process without telling explicitly where it is running.
Stopping a process¶
The internal Stopper of Supvisors applies the following logic to stop a process:
supervisor.stopProcess(namespec)
XML-RPC to the Supervisor instances where the process is runningThis single job is considered completed when:
a
STOPPED
event is received for this process ;an error is encountered (
FATAL
event,EXITED
event whatever the exit code) ;no
STOPPING
event has been received 2 ticks after the XML-RPC ;no
STOPPED
event has been received X+2 ticks after the XML-RPC, X corresponding to the number of ticks needed to cover thestopwaitsecs
seconds of the program definition in the Supvisors instance where the process has been requested to stop.
This principle is used for stopping a single process using a supvisors.stop_process
XML-RPC.
Stopping an application¶
The application stop sequence is defined at the same moment than the application start sequence. It corresponds to a dictionary where:
the keys correspond to the list of
stop_sequence
values defined in the program rules of the application ;the value associated to a key is the list of programs having this key as
stop_sequence
.
Note
The Unmanaged applications do have a stop sequence. All their programs have the default stop_sequence
set to 0
.
Hint
All the programs sharing the same stop_sequence
are stopped simultaneously, which solves some of the requests
described in the following Supervisor unresolved issue:
The internal Stopper of Supvisors applies the following algorithm to stop an application:
stop_sequence
This principle is used for stopping a single application using a supvisors.stop_application
XML-RPC.
Stopping all applications¶
The applications are stopped when Supvisors is requested to restart or shut down.
When entering the DEPLOYMENT
state, each Supvisors instance evaluates also the global stop sequence
using the stop_sequence
rule configured for the applications and processes.
The global stop sequence corresponds to a dictionary where:
the keys correspond to the list of
stop_sequence
values defined in the application rules ;the value associated to a key is the list of application stop sequences whose applications have this key as
stop_sequence
.
Upon reception of the supvisors.restart
or supvisors.shutdown
, the Supvisors instance uses
the global stop sequence to stop all the running applications in the defined order.
The following pseudo-code explains the logic used:
stop_sequence
Conciliation¶
Supvisors is designed so that there should be only one instance of the same process running on a set of nodes, although all of them may have the capability to start it.
Nevertheless, it is still likely to happen in a few cases:
using a request to Supervisor itself (through Web UI, supervisorctl, XML-RPC) ;
upon a network failure.
Attention
In the event of a network failure - let’s say a network cable is unplugged -, if the auto_fence
option is not
set, a Supvisors instance running on the isolated node will be set to SILENT
instead of ISOLATED
and its
URL will not disconnected from the subscriber socket.
Depending on the rules set, this situation may lead Supvisors to warm restart the processes that were running in the lost Supvisors instance onto other Supvisors instances.
When the network failure is fixed, Supvisors will likely have to deal with a bunch of duplicated applications and processes.
When such a conflict is detected, Supvisors enters in the CONCILIATION
state.
Depending on the conciliation_strategy
option set in the rpcinterface extension point, it applies a strategy to be rid
of all duplicates:
SENICIDE
When applying the
SENICIDE
strategy, Supvisors keeps the youngest process, i.e. the process that has been started the most recently, and stops all the others.
INFANTICIDE
When applying the
INFANTICIDE
strategy, Supvisors keeps the oldest process and stops all the others.
USER
That’s the easy one. When applying the
USER
strategy, Supvisors just waits for a third party to solve the conflicts using Web UI, supervisorctl, XML-RPC, process signals, or any other solution.
STOP
When applying the
STOP
strategy, Supvisors stops all conflicting processes, which may lead the corresponding applications to a degraded state.
RESTART
When applying the
RESTART
strategy, Supvisors stops all conflicting processes and restarts a new one.
RUNNING_FAILURE
When applying the
RUNNING_FAILURE
strategy, Supvisors stops all conflicting processes and deals with the conflict as it would deal with a running failure, depending on the strategy defined for the process. So, after the conflicting processes are all stopped, Supvisors may restart the process, stop the application, restart the application or do nothing at all.
Supvisors leaves the CONCILIATION
state when all conflicts are conciliated.