index

1. Introduction
1. Purpose

The Aguascalientes project requires disaster recovery operation; therefore, this document intended to enable and guide the operation teams to running technical operations in case of a disaster or an emergency event in the main site (C5). The document will describe step by step the process of moving the CityShob products from the main site (C5) to the Disaster Recovery (DR) site.

The document includes the following:

•        DR Architecture from the aspects of components (infrastructure, Application, Hardware and Software).

•        System Functionality available in DR site.

•        DR scenarios – step by step operation guide.

There are 4 Time frames that are handled by the Run Book:

•        Normal operation – C5 site is active and data is synced to DR

•        Disaster Scenario – Disaster in C5 that requires Activating the DR Site, C5 is down

•        Normal Operation at DR Site – DR site is active and data is synced to C5

•        Fail Back – failing back to C5 Site. After this step we are back to “normal operation”

2. Scope

The Run Book is containing all the needed information, procedure (work order including prerequisite operation) to make operation teams be able to handle disaster event and to meet the customer requirements in terms of Move-to-DR and Move-Back-to-C5 for operational continuity. 

For each component in the system, there is a set of activities that need to be taken in order to perform the failover/failback between the C5 and DR sites.

Most of the activities are automatic while some of them will be manual, each of the scenarios will have the information of the steps that need to be done, and how to activate/execute each of the steps.

3. System Functionality

The following table describes the list of functionalities exist in the Main site and in the DR site.

Remark: Not all the functionalities in the C5 site are available in the DR site. This is part of the project definition.  

2. DR Scenarios

1. Normal Operation

During normal operation, the system C5 is fully operative and the entire CityShob products data is replicated to the DR site constantly to the big-data servers and storage in the DRP site.

During this time, the DR site is offline and not providing service.

C-Insight clients in C5 and C2 are connected to C5 DC and all functionalities are available.

2. Move To DRP

This scenario to be activated in case of a disaster or an emergency event in the main site (C5) and activate this scenario should be decided and communicated by the C5 manager or C5 management.  

Activating this scenario will require the operation teams to execute the run book scenario step-by-step in order to make the DR site active.

The “Move-to-DR” process may take up to 8 hours and during this time the service will be down.

Upon completing the process, the DR site becomes the active site and all C-Insight clients in C5, C2 and DR are connected to DR DC and only DR functionalities will be available (see the table in Section 1.3).

The C5 site is offline and cannot be activated as long the DRP site is running as the Active Site.

3. Move Back to Main (C5)

Once C5 is restored from the disaster or an emergency event and the C5 manager decides it time to move the service back to the main site, the operation teams require to execute the run book scenario step-by-step in order to make the C5 site active again.

The “Move-Back-To-Main” process may take up to 8 hours and during this time the service will be down.

The process will include synchronizing all new data updated in DR while it was active back to C5 so no data will be lost.

Once the C5 site back to be the active site, all C-Insight clients in C5, C2 and DR are connected to C5 DC with full system functionalities (see the table in Section 1.3).

The DRP site is offline and data is synchronized from the C5 site.

3. Data Replication Verification

During normal operation the data is constantly replicated from main to DR site.
Following steps show how to verify data replication is working properly.

1. SQL Server Replication 

1. Login to SQL Server in DRP using the SQL Management studio
2. Verify Vulcan and VulcanAuditDB are in Synchronizing status 

1. Login to server “DRP-INS-SRV1” and open CIT application

2. Go to “Administrator” Tab 🡪 “Open Main To DR”

3. Enter user, password and SQL Server instance of the DRP site

User name: drprunner
Password: DRP!ex1234

4. Click “Sync Status” and verify C5 is active.

2. Elastic Service Replication 

The Elastic Search data replication service is located in C5 SQL server (C5-SQL-SRV01)

1. Login to SQL Server in C5 
2. 
   

3. Open windows services and verify “ElasticsearchMaintenanceService” is running

4. Service logs is in 🡪 C:\C-INSIGHT\ElasticsearchMaintenance\logs
5. Open last log and verify there are no errors.

4. Move To DRP” Procedure

Following describe the technical step by step to move the system to the DRP site.

1. High Level Plane:

2. Stage 1: IT Preparation

Make sure IT will perform the following:

1. VMS – the Video Management system should be fully operating from DR site and all the live stream from the outdoor is redirected by the network to the DRP site.
2. Verify DDN is available in DRP
3. DNS Routing for C-REACT is redirected by Firewall to DR site
4. DRP DMZ VLAN is open to internet for C-SHARE, C-REACT and Public Transportation services.

Remark: if the phase is not completed and verify, CityShob products will not works.

3. Stage 2: Stop C5

1. Login to server “DRP-Maint-SRV01” and open PDQ application.
2. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE TO DRP 🡪 STOP C5 APP SRV 
   on all C-Insight application server in C5 site.
   ** If C5 site is not available / down then skip this step.

6. Login to C5-SQL-SRV01 Server in C5
7. 
   

8. Open windows services and stop “ElasticsearchMaintenanceService” service

4. Stage 3: Move To DR

Following steps should be done from the DR site:

1. Login to server “DRP-INS-SRV1” and open CIT application

2. Go to “Administrator” Tab 🡪 “Open Main To DR”

3. Enter user, password and SQL Server instance of the DRP site

User name: drprunner
Password: DRP!ex1234

4. Click “Move To DR” and wait until all steps completed with “Pass” result.

5. Stage 4: Start DR Site

1. Login to server “DRP-Maint-SRV01” and open PDQ application.
2. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE TO DRP 🡪 START DRP APP SRV 
   ** Executed this job on all DRP-INS-SRV* servers 

3. Login to server “DRP-INS-SRV1” and open CIT application
4. In CIT go to “Project” Tab 🡪 Open from Server

Enter user, password and SQL Server details in DRP

5. Go to Management  🡪 Select the “Fit” button to display all BU’s 

6. Verify all BU’s are green:

6. Stage 5: Setting C-Insight Clients to DR

1. Login to server “DRP-Maint-SRV01” and open PDQ application.
2. 
   

3. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE TO DRP 🡪 SET CLIENT TO DRP DC 
   on all C-Insight clients (C5, C2, C4).

7. Stage 6: Testing

• Run system Sanity test

5. “Move Back to Main (C5)” Procedure

Following describe the technical step by step moving system back to main site.

1. High Level Plane:

2. Stage 1: IT Preparation

Make sure IT will perform the following:

1. VMS – the Video Management system should be fully operating and all the live stream from the outdoor is redirected by the network to the main site.
2. Verify DDN is available in C5
3. DNS Routing for C-REACT is redirected by Firewall to C5 site
4. C5 DMZ VLAN is open to internet for C-SHARE, C-REACT and Public Transportation services.

3. Stage 2: Stop DR 

1. Login to server “DRP-Maint-SRV01” and open PDQ application.
2. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE BACK TO C5 🡪 STOP DRP APP SRV 
   on all C-Insight application server in DRP site.

** Executed this job on all DRP-INS-SRV* servers 

4. Stage 3: Data Replication 

1. Login to server DRP-SQL-SRV01
2. 
   

3. Open windows services – and start service: “ElasticsearchMaintenanceService”

4. Wait until all data replicate back from DR site to main.
5. Check the logs for replication completed 🡪 C:\C-INSIGHT\ElasticsearchMaintenance\logs
6. Stop service: “ElasticsearchMaintenanceService”.
7. Login to server “DRP-INS-SRV1” and open CIT application

8. Go to “Administrator” Tab 🡪 “Open DR To C5”

9. Enter user, password and SQL Server instance of the DRP site

User name: drprunner
Password: DRP!ex1234

10. Click “Move To C5 - Sync” and wait until all steps completed with “Pass” result.

5. Stage 4: Move To C5 

1. Login to server “DRP-INS-SRV1” and open CIT application
2. Go to “Administrator” Tab 🡪 “Open DR To C5”

3. Enter user, password and SQL Server instance of the DRP site

User name: drprunner
Password: DRP!ex1234

4. Click “Move To C5” and wait until all steps completed with “Pass” result.

6. Stage 5: Start C5 Site

1. Login to server “C5-Maint-SRV01” and open PDQ application.
2. 
   

3. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE BACK To C5  🡪 START C5 APP SRV 
   ** Executed this job on all C5-INS-SRV* servers 

4. Login to server “C5-INS-SRV1” and open CIT application
5. In CIT go to “Project” Tab 🡪 Open from Server

Enter user, password and SQL Server details in C5:

6. Go to Management  🡪 Select the “Fit” button to display all BU’s 

7. Verify all BU’s are up (green):

11. Login to server C5-SQL-SRV01
12. 
    

13. Open windows services – and start service: “ElasticsearchMaintenanceService”

14. Wait until all data replicate back from DR site to main.
15. Check the logs for replication completed 🡪 C:\C-INSIGHT\ElasticsearchMaintenance\logs

7. Stage 6: Setting C-Insight Clients to C5

1. Login to server “C5-Maint-SRV01” and open PDQ application.
2. Execute Job 🡪 DRP RUN BOOK 🡪 MOVE BACK TO C5 🡪 SET CLIENT TO C5 DC 
   on all C-Insight clients (C5, C2, C4).

8. Stage 7: Testing

• Run system Sanity test

6. Server List
1. Main Server List (C5)

2. DRP Server List (C4)

Created with the Personal Edition of HelpNDoc: Easily create CHM Help documents