.. sectionauthor:: Dr. Mickey Martini <mickey@swabianinstruments.com>
.. sectionauthor:: Dr. Edoardo Mornacchi <edoardo.mornacchi@swabianinstruments.com>

.. _remotesynch-tutorial:

***************************************
Remote Synchronization of Time Taggers
***************************************

In high-precision applications such as distributed time monitoring, telecommunications, quantum communication,
and quantum key distribution, synchronizing time-tagging devices across remote locations is crucial.
When Time Taggers operate independently, their clocks run free, and even small differences in their frequencies
accumulate over time, creating significant timing discrepancies across multi-device setups.

To overcome this challenge, a synchronization-agnostic and versatile approach is employed.
This method supports any synchronization technology that provides a frequency reference signal (e.g., 10 MHz)
and a 1PPS (Pulse Per Second) timing signal, as shown in the figure below.
For example, the `White Rabbit <https://white-rabbit.web.cern.ch/>`_ is a widely adopted technology
that enables picosecond-level synchronization precision with sub-nanosecond accuracy across remote setups, as detailed in this
`application note <https://www.swabianinstruments.com/static/app_notes/Remote_Synchronization_Swabian_Instruments.pdf>`_.
The synchronization process relies on the two timing signals combined with software-based solutions,
including functionalities such as the ReferenceClock, to unify the time bases of the distributed Time Taggers.
Users must ensure that their hardware infrastructure supports the required synchronization signals to enable this functionality.
Moreover, when using the ReferenceClock with |tt20-name|, an inherent timing error of approximately 200 ps needs to be considered.
More details can be found in the :cpp:func:`~TimeTaggerSource::setReferenceClock` documentation
and the related in-depth guide (:doc:`/InDepthGuides/ReferenceClock`).

This tutorial demonstrates how to achieve remote synchronized operation of Time Taggers.
It provides a step-by-step guide to synchronizing multiple Time Taggers across distributed locations,
unifying their time bases, and merging time tag streams from remote network nodes for real-time processing
using the :cpp:class:`TimeTaggerNetwork` functionality, which supports simultaneous connections to multiple servers from version 2.18 onward.


.. image:: imgs/Remote_Sync.*
    :align: center

Establishing a common time base across distributed locations
=============================================================

A critical step in synchronizing Time Taggers across remote locations is establishing a unified time base.
Within our approach, this is achieved by associating the 1PPS signal fed into the Time Tagger,
which is generated at each location with an external clock locked to a Grand Master (GM),
with the Coordinated Universal Time (UTC) provided by the host PC.
To ensure accurate synchronization, the PC clocks at all locations must remain aligned to UTC
with a time shift of less than 0.5 seconds.
Synchronizing the PC clock is therefore essential because it allows the system to associate the 1PPS signal
at each location with the correct UTC second, ensuring that all timing data is unified under a common temporal framework.

The PC clock alignment to UTC can be achieved using reliable
synchronization protocols such as Network Time Protocol (NTP) or Precision Time Protocol (PTP).
Standard operating system synchronization tools (e.g., Windows Time Service, Linux's *ntpd* or *chrony*)
configured to synchronize with either publicly available or internal time servers, are sufficient
to maintain synchronization shift below 0.5 seconds. For enhanced robustness, ease of use,
or advanced configuration options, dedicated synchronization software solutions such as
the `Meinberg NTP <https://www.meinbergglobal.com/english/sw/ntp.htm>`_ software package are recommended.

Starting a Time Tagger Server at each location
===============================================

Once the PC clocks at all locations are aligned to UTC, the next step is to configure each Time Tagger
to operate within a unified time base.
By default, timestamps generated by a Time Tagger are relative to when the device was initialized in software.
This means that if multiple Time Taggers start at different times, identical physical events
occurring simultaneously at different locations receive different timestamps.

To synchronize the Time Taggers, enabling measurements across time tag streams generated at remote locations,
the ReferenceClock must be activated.
This ensures that all Time Taggers apply a UTC-based offset to their time base,
allowing simultaneous events at remote locations to receive identical timestamps,
regardless of when each Time Tagger was started.
The ReferenceClock achieves this by locking the internal clock to an external distributed frequency reference (e.g., 10 MHz)
and using the 1PPS signal to establish absolute time alignment.
To guarantee that the 1PPS edge is associated with the correct UTC second, the *synchronization_offset* argument in the
:cpp:func:`~TimeTaggerSource::setReferenceClock` function can be specified to compensate any constant phase offset
between the computer’s system time and the PPS signal.
Initially, the *synchronization_offset* parameter can be set to 0, allowing the Time Tagger to determine
the required offset and prompt the user with a warning if the system time is badly aligned to the 1PPS signal.
In such a case, the lock is released and the :cpp:func:`~TimeTaggerSource::setReferenceClock` function must then be
called again by the user with the reported offset (in ps) to establish and maintain a stable lock.

To set up each Time Tagger, a connection to the device is first established in software.
Hardware settings such as trigger levels and dead time should be configured according to the specific measurement requirements.
Then, the ReferenceClock is enabled, aligning the internal time base with the external synchronization signals.
Finally, a server is started on the host PC, making the Time Tagger accessible over the network for real-time data collection.

.. code-block:: python

    from Swabian import TimeTagger

    # Connect to the Time Tagger via USB
    tagger = TimeTagger.createTimeTagger()

    # Declare the channel names and corresponding physical connections
    frequency_channel = 1
    PPS_channel = 2

    # Define the hardware settings here, such as trigger level or dead time.

    # Enable the ReferenceClock
    tagger.setReferenceClock(clock_channel=frequency_channel,
                             clock_frequency=10e6,
                             time_constant=1e-4,
                             synchronization_channel=PPS_channel,
                             synchronization_offset=0,
                             wait_until_locked=true)

    # Start the Server.
    # TimeTagger.AccessMode sets the access rights for clients.
    # Port defines the network port to be used
    tagger.startServer(access_mode=TimeTagger.AccessMode.Control,port=41101)

.. warning::
    Activating the ReferenceClock shifts and rescales the internal time base of the Time Tagger.
    As a result, all ongoing measurements on every channel will be affected.
    Data recorded after activating the ReferenceClock will no longer be comparable
    with data acquired beforehand from the same Time Tagger instance.


Connecting to multiple Time Tagger Servers over the network
===========================================================

Once the Time Tagger servers are running at different locations,
a client PC can connect to them to retrieve and process synchronized measurement data.
The connection process relies on network communication,
where the client searches for available Time Tagger servers and establishes a link to them.

To automatically discover servers on the local network, the function :cpp:func:`scanTimeTaggerServers` is used.
This function sends multicast UDP messages to detect active Time Tagger servers and retrieve their IP addresses.
However, its effectiveness depends on the network configuration.
Since multicast messages typically remain within the same local subnet,
their ability to reach servers across different subnets depends on whether the network routers forward multicast packets.
If the function does not detect any servers, it is likely that multicast routing is either not enabled or unreliable in the network.
In cases where :cpp:func:`scanTimeTaggerServers` fails to find a server,
the user needs to know the IP addresses and network ports of the servers to connect to,
when calling :cpp:func:`createTimeTaggerNetwork()` on the client PC.

.. code-block:: python

    # Use the scanTimeTaggerServers() function to search for Time Tagger servers in the local network
    servers = TimeTagger.scanTimeTaggerServers()
    print("{} servers found.".format(len(servers)))
    print(servers)

    # Create a TimeTaggerNetwork instance and connect to the desired servers
    server_addresses = ["192.168.1.100:41101", "192.168.1.101:41101"]  # Replace with the IPs of the servers to connect to
    ttn = TimeTagger.createTimeTaggerNetwork(server_addresses)

.. warning::
    When a :cpp:class:`TimeTaggerNetwork` object is created and connected to multiple servers,
    the time tag streams from different locations are merged into a unified data stream for measurements.
    This merging process relies on a sorting algorithm and requires that all incoming time tag streams have similar timestamps,
    meaning the Time Taggers must be properly synchronized. If the time bases are not aligned, stream merging fails.

Accessing individual servers
============================

When using a :cpp:class:`TimeTaggerNetwork` object to connect to multiple Time Tagger servers,
it is possible to access and control each individual server separately.
This is particularly useful when extracting detailed device-specific information
that is not channel-dependent, such as checking overflow states using methods like :cpp:func:`~TimeTaggerSource::getOverflows`.

The :cpp:func:`~TimeTaggerNetwork::getServers` method of the :ref:`TimeTaggerNetwork <time-tagger-network>` class returns a list of :cpp:class:`TimeTaggerServer` objects,
each representing one of the connected servers.
These server objects act as proxies, providing access to all relevant control functions available in TimeTaggerBase and TimeTaggerHardware,
as long as the servers were created with :cpp:enumerator:`AccessMode::Control` privileges.
Unlike the :cpp:class:`TimeTaggerNetwork` object, a :cpp:class:`TimeTaggerServer` object cannot be used to perform measurements directly.
Namely, it is not possible to pass the :cpp:class:`TimeTaggerServer` object on to any measurement creator.

For example, to retrieve overflow information from a specific server:

.. code-block:: python

    # Retrieve the list of connected Time Tagger servers
    servers = ttn.getServers()

    # Access a specific server
    server_A = servers[0]

    # Retrieve overflow information from the server
    overflows = server_A.getOverflows()

It is also possible to adjust hardware settings using either the :cpp:class:`TimeTaggerServer` object or the globally mapped channels
via the :cpp:class:`TimeTaggerNetwork` object. Both approaches yield the same result:

.. code-block:: python

    # Adjust dead time via the server object
    server_A.setDeadtime(1, 1000)

    # Equivalent operation using TimeTaggerNetwork global channel mapping
    ttn.setDeadtime(1001, 1000)

With the current Software, using :cpp:class:`TimeTaggerServer` objects for configuration is primarily a convenience,
as they serve as proxies for direct interaction with specific servers.
However, :cpp:class:`TimeTaggerServer` provides four specific functions that are not available in :ref:`TimeTaggerNetwork <time-tagger-network>`:
:cpp:func:`~TimeTaggerServer::getAddress`, :cpp:func:`~TimeTaggerServer::getAccessMode`,
:cpp:func:`~TimeTaggerServer::getClientChannel`, :cpp:func:`~TimeTaggerSource::getReferenceClockState`.

Verification of the synchronization technology using a single Time Tagger
=========================================================================

To evaluate the precision of the synchronization technology, a single Time Tagger
can be used before considering a multi-device setup. The proposed verification consists of enabling the ReferenceClock
on one external reference, typically the master if there is a hierarchy, and using a second external reference
as an input to the :ref:`api.FrequencyStability` measurement class.
This allows direct analysis of the synchronization precision using built-in measurement tools.

The measurement provides stability metrics such as Allan Deviation (ADEV), Modified Allan Deviation (MDEV), and Time Deviation (TDEV),
which quantify timing fluctuations over different timescales.
These results characterize the synchronization performance and provide a reference before working with multiple Time Taggers.

The first external reference is connected to an input channel and used to enable the ReferenceClock.
Since all analyses are performed on the same hardware, there is no need for a PPS signal for time synchronization.
The second and additional external references are fed into other input channels and analyzed using the :ref:`api.FrequencyStability` measurement.
The measurement runs over a range of averaging times to extract stability metrics.


.. code-block:: python

    import numpy as np
    import time

    # Define synchronization channels
    ch_master = 1
    ch_slave = 2

    # Enable the ReferenceClock using the first external reference
    tt.setReferenceClock(clock_channel=ch_master, clock_frequency=10e6)

    # Define measurement steps (logarithmically spaced averaging times)
    steps = np.unique(np.logspace(0, 7, 100, dtype=np.int64))

    # Initialize Frequency Stability measurement on the second reference
    fs = TT.FrequencyStability(tt, ch_slave, steps, average=1)

    # Allow the measurement to collect data
    time.sleep(100)  # Adjust as needed

    # Retrieve frequency stability results
    obj = fs.getDataObject()
    tau = obj.getTau()
    ADEV = obj.getADEV()
    TDEV = obj.getTDEV()
    MDEV = obj.getMDEV()

Measuring synchronization precision across multiple Time Taggers
=================================================================

Before starting experiments where Time Taggers are deployed in remote locations,
it is useful to verify the synchronization precision while the units are still physically close to each other.
This ensures that the synchronization setup is working correctly before the Time Taggers are separated.

The verification follows the method described in Swabian Instruments’ `application note <https://www.swabianinstruments.com/static/app_notes/Remote_Synchronization_Swabian_Instruments.pdf>`_ on remote synchronization.
A common test signal is split using a power splitter or any other method that guarantees that identical copies of the signal
are fed into both Time Taggers. The signal is then connected to an input channel of each device.
The :ref:`api.Correlation` measurement class is used to analyze the arrival times of events recorded by both Time Taggers,
providing a direct measure of synchronization precision.

The measurement is performed using the :ref:`TimeTaggerNetwork <time-tagger-network>` in a dual-server setup, as discussed in the previous sections.

.. code-block:: python

    import matplotlib.pyplot as plt

    # Define the correlation measurement between input channels on both Time Taggers
    # We assume the test signal is fed into the third input of each Time Tagger
    corr = TimeTagger.Correlation(ttn, 1003, 2003, binwidth=1, n_bins=1000)

    # Start the measurement and run it for a given time
    corr.startFor(10e12)
    corr.waitUntilFinished()

    # Retrieve the correlation data
    index = corr.getIndex()
    counts = corr.getData()

    # Plot the correlation result
    plt.plot(index, counts)
    plt.xlabel("Time Difference (ps)")
    plt.ylabel("Counts")
    plt.title("Correlation of Synchronized Time Taggers")
    plt.grid()
    plt.show()
