
=======================
10 Gbit/s Ethernet Link
=======================

The |ttx-name| provides multiple interfaces for transferring time-tag data.
For standard operation, the recommended configuration is to transfer time tags
to the host PC via |USB| 3.0, which provides a specified data transfer rate up to |ttx-tagrate-usb|.
For applications requiring higher sustained throughput or lower latency,
the |ttx-name| also provides SFP+ and QSFP+ interfaces.
These interfaces are also discussed in the :doc:`/InDepthGuides/FpgaLink` in-depth guide,
where they are employed to stream time tags to external FPGA hardware through an Ethernet-based protocol.

This guide describes how to use the SFP+ interface as a 10 Gbit/s Ethernet (10 GbE) link
for streaming time tags directly from the |ttx-name| to an acquisition PC.
The data stream is received and processed by the Time Tagger software.

The 10 GbE interface supplements rather than replaces the |USB| connection: |USB| remains required
for device configuration and control, while the time-tag data stream is routed over the SFP+ interface.
At any given time, all time-tag data is transferred over a single data interface;
it is not possible to route data from some channels over |USB| and from others over the SFP+ interface simultaneously.

.. note::

    The 10 GbE link is intended for applications with sustained time-tag rates
    that exceed what can be transferred consistently over the |USB| interface.
    For lower data rates, the |USB| interface is simpler and therefore recommended.


.. note::
    10 GbE data transfer is available to all users at up to 100 MTags/s.
    With the *High Rate* hardware license, the full transfer rate of up to |ttx-tagrate-sfp+| over SFP+ is enabled.
    To upgrade your license, contact `sales@swabianinstruments.com`. The license can then be updated
    using the standard procedure described in :doc:`/gettingStarted/HardwareLicenseUpgrade`.

The following sections describe the setup requirements and provide step-by-step
guidance for configuring and validating the connection.

Setup overview and requirements
-------------------------------

The acquisition PC must be equipped with a dedicated 10 GbE network adapter with an SFP+ port,
either an internal |PCIe| card or an external adapter, and connected to the |ttx-name| via
two independent interfaces: |USB| 3.0 for device configuration and control,
and the SFP+ interface for the time-tag data stream.
The recommended physical setup is a direct point-to-point connection between the SFP+ port
of the |ttx-name| and the 10 GbE adapter.
The acquisition PC must run Time Tagger software version 2.22 or later.

The time-tag data stream uses |UDP|. Because the |ttx-name| generates Ethernet frames directly without relying on |ARP|,
the IPv4 address and |MAC| address of the receiving network interface must be supplied explicitly when enabling
the stream via the Time Tagger API. How to identify these values is described in the next section.

The achievable time-tag transfer rate depends strongly on the |MTU|,
which defines the maximum packet size used for the |UDP| data stream.
Standard Ethernet operation uses an |MTU| of 1500 bytes.
Larger |MTU| values require support for larger-than-standard Ethernet frames,
commonly referred to as jumbo frames.
The Time Tagger API uses an |MTU| of |ttx-sfp-mtu-default| by default.
For high-rate operation, the receiving network interface must therefore be configured
to accept Ethernet frames large enough to carry |UDP| packets with this |MTU|,
as described in the following section.

If the network path does not support an |MTU| of |ttx-sfp-mtu-default|, the Time Tagger |MTU| parameter must
be reduced to the maximum value accepted by all components in the path.
At the standard Ethernet |MTU| of 1500 bytes, the achievable transfer rate is significantly reduced
because the same data must be carried by many more, smaller |UDP| packets;
in this case, we recommend using the |USB| interface instead.

The following table lists hardware configurations validated for 10 GbE streaming,
together with the achieved sustained transfer rates at |MTU| |ttx-sfp-mtu-default|.
Comparable rates may be achievable with |MTU| values above approximately 5000 bytes.

.. list-table::
    :header-rows: 1
    :widths: 30 30 20 20

    * - 10 GbE SFP+ interface
      - CPU
      - OS
      - Transfer rate (MTags/s)
    * - Intel X520 (|PCIe|)
      - AMD Ryzen 9 9950X
      - Windows 11 / Ubuntu 24.04
      - 300 / 260
    * - Intel X710 (|PCIe|)
      - AMD Ryzen 9 9950X
      - Ubuntu 24.04
      - 264
    * - QNAP QNA-T310G1S (Thunderbolt 3)
      - AMD Ryzen 9 9950X
      - Windows 11
      - 300
    * - QNAP QNA-T310G1S (Thunderbolt 3)
      - AMD Ryzen 7 PRO 8840HS
      - Windows 11
      - 300
    * - QNAP QNA-T310G1S (Thunderbolt 3)
      - Intel Core i7-1360P
      - Windows 11
      - 300


All measurements were performed with Time Tagger software version 2.22 using a modified version
of the ``transfer_rate.py`` example found in the ``6-Various-Examples`` folder of the Time Tagger installation,
with 8 channels enabled.

Preparing the receiving network interface
-----------------------------------------

Before enabling 10 GbE streaming, the receiving network interface must be configured
to accept packets with an |MTU| of at least |ttx-sfp-mtu-default|, which matches the default |MTU|
used by the Time Tagger API.
The IPv4 address and |MAC| address of the receiving interface must also be identified.
The exact steps depend on the operating system and on the network-adapter driver.

Windows
^^^^^^^

Identifying the 10 GbE interface
********************************

Open a PowerShell window and run:

.. code:: powershell

    Get-NetAdapter

The output lists all available network adapters with their name, description, and link state.
Connect the cable between the |ttx-name| SFP+ port and the 10 GbE adapter;
the corresponding entry will change from ``Disconnected`` to ``Up``.
Note the adapter name; it is used in the following steps.

Configuring jumbo frames / |MTU|
********************************

1. Open **Device Manager** and expand **Network Adapters**.
2. Right-click the 10 GbE adapter and select **Properties**.
3. Go to the **Advanced** tab and locate the **Jumbo Packet** (or **Jumbo Frame**) property.
4. Set the value to **9014 Bytes** or the highest available value of at least |ttx-sfp-mtu-default|.
5. Click **OK**.

Finding the IPv4 address and |MAC| address
******************************************

Open a Command Prompt and run:

.. code:: doscon

    ipconfig /all

Locate the entry for the adapter name identified above. The **IPv4 Address** and **Physical Address**
(|MAC| address) fields provide the values required by the Time Tagger API.

Alternatively, open :menuselection:`Settings --> Network & Internet`, select the adapter,
and click **Properties**. Both the **IPv4 address** and the **Physical address (|MAC|)**
are listed there.

Linux
^^^^^

Identifying the 10 GbE interface
********************************

Run the following command to list all network interfaces:

.. code:: bash

    ip link show

The output lists all interfaces with their current state. The 10 GbE interface connected
to the |ttx-name| can be identified by its state changing from ``DOWN`` to ``UP`` when the
cable is plugged in. Note the interface name (e.g. ``enp3s0`` or ``eth0``);
it is used in all subsequent commands.

Configuring the |MTU|
*********************

Set the |MTU| to |ttx-sfp-mtu-default|, replacing ``<interface>`` with the name identified above:

.. code:: bash

    sudo ip link set <interface> mtu 9000

Verify that the |MTU| has been applied:

.. code:: bash

    ip link show <interface>

The output should contain ``mtu 9000``. This change takes effect immediately but might not be
persistent across reboots.

Finding the IPv4 address and |MAC| address
******************************************

.. code:: bash

    ip addr show <interface>

The |MAC| address appears on the ``link/ether`` line and the IPv4 address on the ``inet`` line.
These are the values to supply to the Time Tagger API.


Enabling 10 GbE streaming
-------------------------

After the receiving network interface has been prepared, 10 GbE streaming is enabled via the
Time Tagger API. The |ttx-name| must first be connected and initialized over |USB|, as in normal operation.

Call :cpp:func:`~TimeTagger::xtra_enableStreamInterfaceUDP` with the following parameters:

- ``pc_ip``: the IPv4 address of the receiving network interface, identified in the previous section.
- ``pc_mac``: the |MAC| address of the receiving network interface, identified in the previous section.
- ``tagger_ip``: a user-chosen IPv4 address for the |ttx-name|'s SFP+ interface.
  This address must be on the same subnet as ``pc_ip``.
  A simple choice is to take the PC's IP address and change the last octet to an unused address in the same subnet,
  as shown in the example below.
- ``MTU``: the |MTU| in bytes (default: |ttx-sfp-mtu-default|). Must not exceed the |MTU| configured on the receiving adapter
  and is limited to a maximum of 9500 bytes.

Replace the values in the example below with the values used in your setup.

.. code:: python

    from Swabian import TimeTagger

    PC_IP     = '169.254.99.217'
    PC_MAC    = '24:5E:BE:44:26:AF'
    TAGGER_IP = '169.254.99.216'  # user-chosen; must be on the same subnet as PC_IP

    tagger = TimeTagger.createTimeTagger()

    tagger.xtra_enableStreamInterfaceUDP(pc_ip=PC_IP,
                                         pc_mac=PC_MAC,
                                         tagger_ip=TAGGER_IP,
                                         MTU=9000)

Once the |UDP| stream is enabled, Time Tagger measurements can be started as usual.

When 10 GbE streaming is no longer required, call :cpp:func:`~TimeTagger::xtra_disableStreamInterfaceUDP`
to return the time-tag data path to the default |USB| interface.

.. code:: python

    tagger.xtra_disableStreamInterfaceUDP()



Performance considerations
--------------------------

At high data rates, each time tag is compressed to 32 bits,
enabling a maximum transfer rate of |ttx-tagrate-sfp+| over the 10 Gbit/s link.
On reception, the software expands each tag to a 128-bit internal representation,
producing a sustained in-memory data rate of approximately 4.8 GB/s, a substantial volume
that underlines the scale of the data being processed.

The raw time-tag transfer rate given in the hardware comparison table is
measured with a minimal custom measurement that accumulates the size of each received tag block
without processing individual tags.
Under these conditions, the achievable rate reflects the capacity of the network adapter
and the host to sustain high-speed data reception.

When measurements are running, every received time tag must be processed by each active measurement
object. Running multiple measurements simultaneously means their processing costs add up within
each data block, and the combined cost determines the achievable throughput.
The Time Tagger measurement backend is highly optimized to handle this stream efficiently,
sustaining high throughput across most measurement types even when individual measurements
do not reach the maximum transfer rate.

The computational cost per tag varies substantially across measurement types.
The throughput figures below were obtained on an AMD Ryzen 9 9950X with an Intel X520 |PCIe| adapter
and should be interpreted as indicative rather than as specifications;
actual values depend on hardware, operating system, and measurement parameters.

:ref:`api-FileWriter` achieves the full hardware transfer rate, independently of the number of channels.
Although it does not perform per-tag analysis, it must batch time tags into blocks,
compress each block, and write the result to disk. Its throughput is sensitive to thread
scheduling: when simultaneous multi-threading (SMT) is enabled, the data-conversion and the
file-writing threads may be placed on logical cores sharing the same physical core, reducing
throughput substantially down to 200-250 MTags/s. Disabling SMT recovers the full transfer rate.

Lightweight measurements such as :ref:`api.Counter` and :ref:`api.Countrate` add minimal
overhead and achieve close to the full hardware transfer rate.

:ref:`api.Correlation` is considerably more expensive. For each incoming tag, the algorithm compares it
against all previously received tags on the correlated channel that fall within the correlation
time window, defined as ``binwidth × n_bins``. The computational cost per tag therefore scales
with the number of historical tags retained in that window.
When multiple peaks of the correlation function fall within this window, all inter-peak tags
must be retained and examined for each new tag, multiplying the cost accordingly.
Even in a single-peak configuration, throughput is typically limited to approximately 240 MTags/s.
To maximize throughput, set the time window to the minimum value that covers the region of interest
and ensure that at most one peak falls within the window.

:ref:`api.FrequencyStability` can approach the full transfer rate when the averaging parameter is large
(e.g. ``average = 1000``); throughput is slightly dependent on the number of active channels.
:ref:`api.PhaseNoise` performs an |FFT|-based analysis; throughput improves as more channels share the
computation, with a practical upper limit of approximately 200 MTags/s.

:ref:`api.HistogramLogBins` must search across all bin edges for each detected stop event, a cost
that grows with the number of bins and the span of the time axis; despite a multi-threaded
implementation, it remains one of the most computationally intensive common measurement types.

The achievable throughput for a given setup depends on a combination of factors: the network
adapter and host hardware, the operating system, the number of active channels, the number
and type of simultaneous measurements, and their individual parameters.
For guidance on a specific configuration, please contact `support@swabianinstruments.com`.