Administration: Difference between revisions

From acumen Wiki
Jump to navigation Jump to search
 
(47 intermediate revisions by 2 users not shown)
Line 16: Line 16:
|77.68.28.115
|77.68.28.115
|No
|No
|Agregate all inbound message feeds and serve via RabbitMQ Broker to application servers.
|Aggregate all inbound message feeds and serve via RabbitMQ Broker to application servers.
|-
|-
|acumen.tms-uk-rail.co.uk
|acumen.tms-uk-rail.co.uk
Line 30: Line 30:
|}
|}


All servers are hired VPS - Virtual Private Server, with the sevice provider being [https://www.ionos.co.uk/ IONOS by 1&1] under several contracts. All servers are located within the UK at a data centre in London.
All servers are hired VPS - Virtual Private Server, with the service provider being [https://www.ionos.co.uk/ IONOS by 1&1] under several contracts. All servers are located within the UK at a data centre in London.


= Gateway Server =
= Gateway Server =
Line 43: Line 43:
TD.net feed uses MQIPT protocols for message publication and consumption, whilst NRE Darwin subscription utilises a protocol known as STOMP; The gateway server manages the connections to these services and publishes all messages to a message broker, for consumption by the application server(s).
TD.net feed uses MQIPT protocols for message publication and consumption, whilst NRE Darwin subscription utilises a protocol known as STOMP; The gateway server manages the connections to these services and publishes all messages to a message broker, for consumption by the application server(s).


[https://www.rabbitmq.com/ RabbitMQ] was chosen as the message broker component within the gateway - all message orientated services on the application servers connect to this broker.
[https://www.rabbitmq.com/ RabbitMQ] was chosen as the message broker component within the gateway - all message orientated services on the application servers connect to this broker.
 
::- The interface is [http://77.68.28.115:15672/ here]. <b>Note:</b> It will not open with zscaler.
::- More information can be found here [[Gateway server#RabbitMQ Interface]].


There are several tools and interfaces provided to administrators for the purposes of monitoring and diagnosing issues with the [[gateway server]].
There are several tools and interfaces provided to administrators for the purposes of monitoring and diagnosing issues with the [[gateway server]].
Line 53: Line 56:
Services running on the application server can conceptionaly be grouped thus:
Services running on the application server can conceptionaly be grouped thus:


* [[Middleware]] - those services which are needed by the application to function (MySQL, Redis, Apache, nginx, RabbitMq);
* [[Middleware]] - those services which are needed by the application to function;
* UI services - services which are responsible for creating and managing front end (UI rendering and user interaction) functionality;
* [[UI services]] - services which are responsible for creating and managing front end (UI rendering and user interaction) functionality;
* Forecasting - services which maintain the train service forecast;
* [[CIF Processing]] - the service which takes the POINTA CIF from PPTE and maintains the TSDB (Train Service Database);
* Conflict - services which parse the forecast for defined conflict types.
* [[Forecasting]] - services which maintain the train service forecast, including conflicts.
 
= User Administration =
 
The [[User Management]] section provides guidance and instructions for Administrators and Superusers in managing the users of acumen.
 
= Diagnosing Issues =
 
<i> Under development </i>
 
The [[gateway server]] should be the first port of call if something goes wrong.
:- Check Glances to see how the system is working.
:- Check the Application logs to see if there are any errors.
:- Use the RabbitMQ Interface to see if the problem is with the external services or within acumen.
 
{| class="wikitable"
!| Prompt
! Initial actions
 
|-
|Missing movement reports
|Check Gateway
|-
|No train movement reports
|Use the <i>docker ps</i> command to see whether any containers have fallen over. See [[Middleware]].
|-
|Many services showing overdue (white boxes on the platform docker)
|Check [http://77.68.28.115:15672/ RabbitMQ] TRUST feeds
|-
|502 Bad Gateway error message
|Follow instructions on [[UI services]]
|-
|Wrong schedule (hasn't had STP overlay)
|Follow instructions on [[CIF Processing]]
|-
|User reports application is lagging
|Check Glances and consider draining the forecast
|}
 
= Reporting Issues =
 
Should a fault be diagnosed with an external service, the following table provides information on who the Administrator should contact:
 
{| class="wikitable"
!| Component / Service
! Organisation
! primary contact for first response
|-
|TD.net
|Network Rail 4C's helpdesk
|Calls can be logged by dialling 085 51600 (internal) or 01270 721600 (external).
|-
|PPTE (CIF)
|Network Rail 4C's helpdesk
|Calls can be logged by dialling 085 51600 (internal) or 01270 721600 (external).
|-
|Darwin Feed
|National Rail Enquiries
|dsg_darwind3.support@caci.co.uk
|-
|}
 
= All Administration Web Pages =
 
=== Administration wiki pages ===
 
:* [[Admin Interfaces]] - instructions for new PuTTY-replacement interfaces
 
:* [[CIF Processing]]


= Glances / top =
:* [[Database CLI]] (Command Line Interface) - command help with [[CIF Processing]] issues


These tools are provided to visually display the performance and load of the operating system and to view running services. Administrators should be cognisant of:
:* [[Forecasting]] - including last resort actions


* High CPU load and any associated warnings;
:* [[Gateway server]] - including Glances, Top, application logs, RabbitMQ, acumen service management scripts (are services UP or DOWN), MQIPT
* High memory usage and any associated warnings - including swap file usage;
* High file system usage;
* Running processes, or processes not running that should be (note: a script is provided for this purpose).


=== Glances ===
:* [[Middleware]] - including Redis


Click [https://nicolargo.github.io/glances/ here to learn more about Glances].
:* [[PuTTY Help]]


There are 2 ways to access Glances on the Gateway Server
:* [[UI services]]


* Method 1: Access via the [http://77.68.28.115:61208/ Web Interface], or
=== Other helpful wiki pages ===
* Method 2: Start an ssh session and at the prompt: <code><user>@prod_tms_server:~$ glances</code>.


=== top ===
:* [[Data Feeds]]


Click [https://en.wikipedia.org/wiki/Top_(software) here to learn more about top].
:* [[Support]]


To access top, start an ssh session and at the prompt: <code><user>@prod_tms_server:~$ top</code>.
:* [[User Management]]

Latest revision as of 12:25, 28 September 2020

This section of the acumen wiki is written for the express purpose of providing information and guidance to acumen administrators.

Simplified Architecture Overview

Acumen architecture.png

The application is deployed to 3 servers, as follows:

Host Name IP Address Containerised Deployment? Purpose
gateway.tms-uk-rail.co.uk 77.68.28.115 No Aggregate all inbound message feeds and serve via RabbitMQ Broker to application servers.
acumen.tms-uk-rail.co.uk 77.68.118.126 Yes Serve the acumen application (main server)
acumen-training.tms-uk-rail.co.uk 77.68.3.32 Yes Serve the acumen application (training server)

All servers are hired VPS - Virtual Private Server, with the service provider being IONOS by 1&1 under several contracts. All servers are located within the UK at a data centre in London.

Gateway Server

The functionality of this component is to aggregate various data feeds and make any resulting messages available to be consumed by one or more application servers.

There are 2 distinct feeds that the gateway server connects to:

  • Network Rail TD.net feed;
  • National Rail Enquiries Darwin Feed.

TD.net feed uses MQIPT protocols for message publication and consumption, whilst NRE Darwin subscription utilises a protocol known as STOMP; The gateway server manages the connections to these services and publishes all messages to a message broker, for consumption by the application server(s).

RabbitMQ was chosen as the message broker component within the gateway - all message orientated services on the application servers connect to this broker.

- The interface is here. Note: It will not open with zscaler.
- More information can be found here Gateway server#RabbitMQ Interface.

There are several tools and interfaces provided to administrators for the purposes of monitoring and diagnosing issues with the gateway server.

Application Server

Each application server runs a single instance of the acumen application which consists of numerous containerised services which make up the application - acumen utilises a microsystem architecture where each service is responsible for defined or limited tasks and as a whole, make up the application.

Services running on the application server can conceptionaly be grouped thus:

  • Middleware - those services which are needed by the application to function;
  • UI services - services which are responsible for creating and managing front end (UI rendering and user interaction) functionality;
  • CIF Processing - the service which takes the POINTA CIF from PPTE and maintains the TSDB (Train Service Database);
  • Forecasting - services which maintain the train service forecast, including conflicts.

User Administration

The User Management section provides guidance and instructions for Administrators and Superusers in managing the users of acumen.

Diagnosing Issues

Under development

The gateway server should be the first port of call if something goes wrong.

- Check Glances to see how the system is working.
- Check the Application logs to see if there are any errors.
- Use the RabbitMQ Interface to see if the problem is with the external services or within acumen.
Prompt Initial actions
Missing movement reports Check Gateway
No train movement reports Use the docker ps command to see whether any containers have fallen over. See Middleware.
Many services showing overdue (white boxes on the platform docker) Check RabbitMQ TRUST feeds
502 Bad Gateway error message Follow instructions on UI services
Wrong schedule (hasn't had STP overlay) Follow instructions on CIF Processing
User reports application is lagging Check Glances and consider draining the forecast

Reporting Issues

Should a fault be diagnosed with an external service, the following table provides information on who the Administrator should contact:

Component / Service Organisation primary contact for first response
TD.net Network Rail 4C's helpdesk Calls can be logged by dialling 085 51600 (internal) or 01270 721600 (external).
PPTE (CIF) Network Rail 4C's helpdesk Calls can be logged by dialling 085 51600 (internal) or 01270 721600 (external).
Darwin Feed National Rail Enquiries dsg_darwind3.support@caci.co.uk

All Administration Web Pages

Administration wiki pages

  • Gateway server - including Glances, Top, application logs, RabbitMQ, acumen service management scripts (are services UP or DOWN), MQIPT

Other helpful wiki pages

Retrieved from "http://wiki.acumen-tms.co.uk/index.php?title=Administration&oldid=875"