Shadow Journaling

Shadow journaling, or shadowing, enables a secondary computer to maintain a “shadow” version of selected databases as they are updated on a primary machine. By continually transferring journal information from the primary machine to the secondary machine, shadow journaling enables recovery to a database which is within only a few transactions of the source database. This process is sometimes referred to as replication.

By itself, shadow journaling is not sufficient to ensure successful failover, but it is a very simple and inexpensive approach to maintaining a disaster recovery system. Often, this approach is also used to update a report server, where ad hoc reporting tasks can operate on current data without affecting production.

This chapter discusses the following topics:

Shadow Journaling Overview

Shadow journaling monitors database activity on a primary system, the source, and causes the same activity to occur on a secondary system, the destination. It does this through a shadow client service running on the destination that continually requests journal file details from a shadow service running on the source. The shadow service responds by sending the details of the actual Set or Kill journal record entry to the destination shadow over a TCP connection.

Shadow journaling features two different ways of transmitting information. These ways are mutually exclusive, and each has its own merits:

Fast Transmission Mode—allows more efficient performance because it transfers a compacted journal file by blocks. Fast transmission requires the data to be written to the journal file, which may introduce a delay of a few seconds, and it requires more disk activity than compatible transmission mode. In releases prior to 5.0, this mode was block mode.
Compatible Transmission Mode—involves the packing and unpacking of journal records rather than blocks, which is somewhat less efficient. The benefit is compatibility and new journal records are received immediately, rather than cached. In releases prior to 5.0, this mode was record mode.

Important:
Use compatible transmission mode when the source and destination systems have different endian representations.

See Setting Shadowing Properties for information on setting the transmission mode.

Fast Transmission Mode

Fast transmission (journal transmission) means sending the journal file block by block. The shadow establishes a TCP connection to the server and receives the journal file. As the journal file downloads, another shadow process applies the journal entries to the local copy of the database.

Upon connecting to the source server, the destination shadow sends the server the name of the journal file and the starting block number. When the server reaches the ending block number, even if it has changed since the download began, the transmission ends. Subsequently, the shadow checks the source to see if it has the latest records. If it does, the shadow waits, checking on new records periodically. If it does not have the latest records, the shadow downloads them and updates the database as described.

There are two types of fast transmission:

Fast mode, apply changes—Applies all transactions to the local (shadow) database allowing you to maintain a shadow copy of both journal and database.
Fast mode, do not apply changes—Maintains an identical journal, but does not apply the journal changes to the database. The journal can then be used to update either database; however, the shadow database cannot immediately be used as an automatic failover database. This method essentially creates a remote journal file maintained via TCP stream across a network, eliminating the need to mount a separate disk to maintain a remote journal file.

Compatible Transmission Mode

Compatible transmission means sending the transaction record information in a journal as a string and processing the string. The shadow establishes a TCP connection to the source database server and receives a series of strings. Each string contains a global modification. Caché applies the modification described in the string to the specified database and logs the modification in its own journal file.

The source server sends the destination shadow all the records in the journal file, even if the entry was created while the shadow was receiving transmissions from the server. When the shadow has all the records, it waits for an interval before requesting more journal information. If there is more information, the server sends it; if there have been no additional transactions, the shadow returns to a waiting state.

There are two different forms of compatible transmission:

Compatible mode, apply changes—Applies all transactions to the local shadow database allowing you to maintain a shadow copy of both the journal and the database.
Compatible mode, scan changes—Instead of applying each transaction to the local database, the shadow scans each transaction using a predefined command string. Information about a transaction is passed to the command for further action.

Enabling Shadow Journaling

This section explains how to configure and set up shadow journaling in Caché. It outlines the tasks necessary for the following:

Note:

A shadow service cannot run on a system with a single-server license.

Configuring the Source Database Server

To enable shadow journaling on a source database server, first ensure that the source system can make a TCP connection to the destination system. Next, use the Caché Configuration Manager from the Caché instance running on the source system to enable journaling and security:

To enable global journaling, click the Advanced tab from the Caché Configuration Manager.
Double-click Journal to expand the journal properties.
Ensure that the Journal Mode property is set to “All Globals”; if it is set to “Selected Globals”, verify that journaling is enabled for the globals to be shadowed. See Journaling Selected Globals in the “Journaling” chapter of this guide.
For greater security, restrict the systems that can shadow the database on the source system. Double-click Shadow to expand the shadowing properties.
Double-click the Source folder to expand its properties.
Double-click Enable Security to change its value to “Yes”. This restricts shadowing capabilities to only those servers with the IP addresses you provide.
Click Permissible Shadow Server Addresses. Any previously entered server addresses are displayed in the TCP/IP Address list. If a list does not appear, no servers are defined.
Click Add to add a TCP/IP Address property. Click Change to enter the IP address of any destination shadow allowed to connect to this database server, and click OK. Repeat this step until you have entered all permissible addresses.

Configuring the Destination Shadow

To configure shadow journaling on a destination shadow server, first ensure that the destination system can make a TCP connection to the source system. Next, use the Caché Configuration Manager from the Caché instance running on the destination system to configure the destination shadow properties:

Open the Caché Configuration Manager and click the Advanced tab.
Double-click Shadow to expand the shadowing properties.
Click Destination. If there are no destination properties, this system has not yet been configured as a shadow. Click Add to configure this system as a shadow of a source database server.
Enter an identifying name for this shadow in the Add New Shadow Destination dialog box and click OK. The system uses this name to distinguish between shadow configurations that may be running on the same system.
To edit each destination shadow property, either click the property name and click Change, or double-click each to display its Change Property dialog box:
- Start—Select the check box to automatically start the shadow at the time Caché starts.
  
  The first time you configure the shadow, clear the check box until you finish setting shadowing properties. See Setting Shadowing Properties for details on entering this information.
- Database Server—Enter the TCP/IP address or host name (DNS) of the source database server you are shadowing.
- Database Server Port #—Enter the default port number of the Caché configuration of the source database server you are shadowing.
You may also change the following Advanced Options:
- # of Errors Retained—Enter the number of shadowing errors from 0 to 200 Caché should retain.
- # of Seconds Between Probes—Enter the number of seconds from 0 to 86,400 between shadowing probes (when the system is caught up).
- Preserve Shadow Server Definitions During Startup—When checked, this preserves the properties of the shadows defined in previous Caché sessions, but not in the active configuration.
Click OK. A restart of Caché is required to activate the changes.

Important:

The journal reader and database update processes on the shadow destination communicate via shared memory allocated from the generic memory heap. Set the generic memory heap as large as possible for the optimal performance of shadow dejournaling. The minimum requirement is five pages for shadowing. The shadow destination fails to start if there is insufficient generic memory heap allocated.

Change the setting of Generic Memory Heap under the Memory branch on the Advanced tab of the Caché Configuration Manager.

Setting Shadowing Properties

Once you configure both the source system and the destination shadow system for shadowing, use the Caché Control Panel on the destination system to set additional shadowing properties.

Verify that Caché is running and that you have restarted Caché to apply any shadowing configuration changes.
Open the Caché Control Panel and expand the Shadowing task folder.
Click This System as Shadow to display a list of shadow servers in the details pane. Right-click the appropriate shadow name and click Properties on the shortcut menu.
On the Journal Setting tab, enter the name of the Database server journal file; use the full name, including the path, of the journal file on the source system. This is a required field.
In the Method of journal transmission list, click the appropriate mode. Subsequent property choices on this tab change depending on the selected transmission mode. See Select Transmission Mode for details on entering this information.
On the Directory Mapping tab, click New to associate the database on the source system with the directory on the destination system using the Add Directory Mapping dialog box.

In the Database box, enter the physical pathname of the source database file—the CACHE.DAT file. Enter the pathname of its corresponding destination shadow database file in the Shadow box.

For information on the Cluster shadowing check box, see the Cluster Shadowing section of the “Cluster Journaling” chapter of this guide.

Select Transmission Mode

Setting some shadowing properties depends on the mode you select from the Method of journal transmission list:

Fast mode, apply changes—

Click the browse button to select the Local directory for copied journal files.

Select the Journal Applied Transactions check box and enable journaling on the shadow to maintain a separate journal file showing the activity on the shadow database. See Using Shadow Journaling for more information.
Fast mode, don't apply changes—

Click the browse button to select the Local directory for copied journal files.
Compatible mode, apply changes—

Select the Journal Applied Transactions check box and enable journaling on the shadow to maintain a separate journal file showing the activity on the shadow database. See Using Shadow Journaling for more information.
Compatible mode, scan changes—

Enter filtering commands to apply to the received journal records. See Filtering Commands for more information.

Filtering Commands

There are four parameters derived from each journaled record. These parameters are used in the command string (or in the external filtering routine/function) to isolate information and act on the information received:

a—Address of the current record
t—Type of transaction
g—Global involved, if any
v—New value to which the global is set

The following values apply to the transaction type field:

Transaction Type Values

Value	Description
4	Begin transaction
5	Commit transaction
6	Set
7	Kill (node)
9	ZKill
10	NSet
11	NKill
12	NZKill
13	Journal Marker
14	Bit Set *
15	ECP Marker
–1	Journal End

* The values of a bit set record have the $List format; the first element is the bit position and the second is the bit value.

Here is an example of a command to be applied to received records:

 Use AuditFile Write !,"("_t_"): ",g,"=",v Use $Principal

The command string may also invoke an external Caché ObjectScript routine or function dedicated to the task of filtering and processing information received from the journal file. For example:

 Do ^ShadowFilterMaster

Journaling on Shadow

The destination shadow can also maintain a journal of applied updates, which provides an additional level of redundancy. If journaling is enabled on the destination shadow, be careful not to place these journals in the same directory as the journal files coming over from the source database server.

Using Shadow Journaling

The following sections describe how to use shadow journaling on your Caché system:

Starting Shadow Journaling

To start shadowing, follow these steps:

From the Caché Control Panel, expand the Shadowing task folder.
From the destination shadow server, click the This System as Shadow folder to display a list of shadow servers.

Otherwise, from the source database server, click the This System as Source folder to display a list of destination servers to which this is shadowing.
Right-click the appropriate shadow server and click Start on the shortcut menu.

Stopping Shadow Journaling

To stop shadowing, follow these steps:

From the Caché Control Panel, expand the Shadowing task folder.
From the destination shadow server, click the This System as Shadow folder to display a list of shadow servers.

Otherwise, from the source database server, click the This System as Source folder to display a list of destination servers to which this is shadowing.
Right-click the appropriate shadow server and click Stop on the shortcut menu.

Purging Shadow Journal Files

Shadow journal files—the journal files copied from the source database server to a shadow in fast transmission mode—are not purged automatically by default. You can change this by running:

$$ConfigPurge^SHDWCLI(cli)

where cli is the shadow name.

Once the purging option is properly configured, either let the shadow purge the shadow journal files automatically or run the following to purge the files:

$$PURGE^SHDWCBLK(cli)

Viewing Error Messages

The shadow server generates internal error messages to report error conditions, but does not display them in a message box. When performance degrades on the shadow server, or you suspect there is a problem with shadowing, check the error message logs for help in diagnosing the problem.

You can view the shadowing error logs on either the destination shadow server or from the source database server. To check the shadowing error logs:

From the Caché Control Panel, expand the Shadowing task folder.
From the destination shadow server, click the This System as Shadow folder to display a list of shadow servers.

Otherwise, from the source database server, click the This System as Source folder to display a list of destinations to which this is shadowing.
Right-click the appropriate shadow server and click Errors on the shortcut menu. A dialog box appears with dated errors messages for the selected shadow server.
Click Clear to clear the error messages from the log, or click Refresh to display the most recent error information.

Using the Shadow Destination for Disaster Recovery

The makers of operating systems and computer hardware provide several appealing failover strategies for high availability. Caché is compatible with these strategies (see the “System Failover Strategies” chapter in this guide for more information). In addition, Caché provides capabilities to support Disaster Recovery strategies. Caché shadowing provides low-cost logical data replication over heterogeneous network configurations. A Caché shadow server can apply journals from several dissimilar platforms on a small-scale server over any TCP network. Since only logical updates are conveyed to the destination shadow, the risk of proliferating any structural problem is eliminated.

You should consider the following limitations before deciding if Caché shadowing best suits your Disaster Recovery strategy.

The shadow server applies production journals asynchronously so as not to affect performance on the production server. This results in possible latency in data applied to the shadow server, although it is generally seconds behind at most. Consequently, if you want to use the shadow server databases, they might be slightly out of date. This latency could increase if the shadow server connection with the production server is lost for any sustained period. Caché provides mechanisms to monitor the state and progress of the shadow server to help you determine the risk of using the shadow server databases during disaster recovery.
When gracefully shutting down the source database server, Caché does not wait for connected destination shadows to catch up before stopping the shadow processes. Therefore, for planned outages, InterSystems recommends you develop a script to run before shutdown that quiesces the system and synchronizes the shadow destinations with the source.
When the production server issues a TRollback, the Set and Kill operations from the rollback are journaled and propagated to the shadow. However, if the production server has crashed, the shadow server may not be left in a transactional-consistent state, since transactions are not tracked independently on the shadow server.

If this consideration is a serious concern, please contact the InterSystems Worldwide Response Center (WRC) for help in assessing the associated risks.

Enabling the shadow server to replace the production server is not automatic. The following procedure highlights how you might recover to the shadow server. If your database system functions as an application server, install identical applications on your shadow system to speed recovery.

To use your shadow system as a master database:

Follow the procedure for stopping shadow journaling on the shadow server. See Stopping Shadow Journaling for more information.
Stop Caché and change the IP address and fully qualified domain name (FQDN) of the shadow system so that it exactly matches the original database system.
Restart Caché.

Caution:

It is not safe to fail over to the shadow system if using transaction processing, objects, or SQL because transactional integrity is not guaranteed on the shadow when the primary has crashed.