Writing Data

Learn how to write data to channels in Synnax, including real-time streaming and historical writes.

This guide covers writing data to Synnax channels. For live data acquisition, using a Writer is the recommended approach. Historical writes are useful for backfilling data or ingesting data from files.

If you’d like a conceptual overview of how writes work in Synnax, check out the writes concepts guide. The rules of writes are especially important to understand.

Writing with Writers

Writers are designed for streaming data as it’s acquired. This is the recommended approach for live data acquisition, control sequences, and real-time data processing. Writers maintain a file-like interface governed by transactions. To learn more about transactions and how writes work in Synnax, see the concepts page.

Opening a Writer

To open a writer, use the open_writer method with a starting timestamp and a list of channels to write to.

# Python's context manager is recommended
with client.open_writer(
    start=sy.TimeStamp.now(),
    channels=["time", "temperature"],
) as writer:
    for i in range(100):
        writer.write({
            "time": sy.TimeStamp.now(),
            "temperature": i * 0.1,
        })
        sy.sleep(0.1)
    writer.commit()

These examples write 100 samples to the temperature channel, each spaced roughly 100ms apart, and commit all writes when finished. It’s typical to write and commit millions of samples over the course of hours or days, intermittently calling commit to persist data to the cluster.

Persistence/Streaming Mode

By default, writers are opened in stream + persist mode. To change the mode of a writer, specify the mode argument when opening the writer. The available modes are:

• persist - Only persist data to the database (no streaming to subscribers)
• stream - Only stream data to subscribers (no persistence)
• persist_stream - Both persist and stream (default)

For example, to open a writer that only persists data:

writer = client.open_writer(
    start=sy.TimeStamp.now(),
    channels=["time", "temperature"],
    mode="persist",
)

Writing Data

The write method accepts several argument formats. Use the one that best fits your use case.

# Write a single sample for a channel
writer.write("temperature", 25.5)

# Write multiple samples for a channel
writer.write("temperature", [25.5, 26.0, 26.5])

# Write a single sample for several channels
writer.write({
    "time": sy.TimeStamp.now(),
    "temperature": 25.5,
})

# Write multiple samples using lists or numpy arrays
import numpy as np
start = sy.TimeStamp.now()
writer.write({
    "time": [start, start + sy.TimeSpan.SECOND],
    "temperature": np.array([25.5, 26.0], dtype=np.float32),
})

# Write using a list of channels and corresponding series
writer.write(
    ["time", "temperature"],
    [timestamps, temperatures],
)

# Write a pandas DataFrame
import pandas as pd
df = pd.DataFrame({
    "time": [sy.TimeStamp.now(), sy.TimeStamp.now() + sy.TimeSpan.SECOND],
    "temperature": [25.5, 26.0],
})
writer.write(df)

# Write a Synnax Frame
frame = sy.Frame({ "time": timestamps, "temperature": temperatures })
writer.write(frame)

Closing a Writer

After you’re done writing, it’s essential to close the writer to release network connections and other resources. If a writer is not closed, other writers may not be able to write to the same channels.

writer.close()

Using structured cleanup patterns ensures the writer is always closed, even if an exception is thrown.

# Python's context manager is the recommended approach
with client.open_writer(
    start=sy.TimeStamp.now(),
    channels=["time", "temperature"],
) as writer:
    # Write data here

# Alternatively, use a try/finally block
writer = client.open_writer(
    start=sy.TimeStamp.now(),
    channels=["time", "temperature"],
)
try:
    # Write data here
finally:
    writer.close()

Historical Writes

Historical writes are useful for backfilling data, ingesting data from files, or writing data at specific timestamps that have already passed.

Writing to a Channel

start = sy.TimeStamp.now()
times = [
    start,
    start + 1 * sy.TimeSpan.MINUTE,
    start + 2 * sy.TimeSpan.MINUTE,
    start + 3 * sy.TimeSpan.MINUTE,
    start + 4 * sy.TimeSpan.MINUTE,
]
temperatures = [55, 55.1, 55.7, 57.2, 58.1]

# Write the timestamps to the index
time_channel.write(start, times)

# Write the data to the channel
temp_channel.write(start, temperatures)

Index and Data Alignment

Notice how the two arrays are aligned using the common start timestamp. This tells Synnax that the first sample in the temperatures array is associated with the first timestamp in the timestamps array.

Synnax will raise a ValidationError if the index channel does not contain a corresponding timestamp for every sample in the data channel. After all, it wouldn’t make sense to have a temperature reading without an associated timestamp.

Writing Multiple Data Channels

It’s common to have multiple data channels that share the same index. For example, a weather station might record temperature, humidity, and pressure all at the same timestamps.

Single Time Index

When writing a single sample to multiple channels at the same instant, use a dictionary with scalar values.

now = sy.TimeStamp.now()

# Write a single sample to multiple channels
client.write(now, {
    "time": now,
    "temperature": 22.5,
    "humidity": 45.0,
    "pressure": 1013.2,
})

Multiple Time Indices

When writing multiple samples over time, use arrays for both the timestamps and data values.

start = sy.TimeStamp.now()
timestamps = [
    start,
    start + 1 * sy.TimeSpan.SECOND,
    start + 2 * sy.TimeSpan.SECOND,
    start + 3 * sy.TimeSpan.SECOND,
    start + 4 * sy.TimeSpan.SECOND,
]
temperatures = [22.5, 22.6, 22.8, 23.1, 23.0]
humidities = [45.0, 45.2, 45.1, 44.8, 44.9]
pressures = [1013.2, 1013.3, 1013.1, 1013.0, 1012.9]

# Write all channels in a single call
client.write(start, {
    "time": timestamps,
    "temperature": temperatures,
    "humidity": humidities,
    "pressure": pressures,
})

All data channels use the same start timestamp for alignment, which tells Synnax they share the same index. The arrays must all have the same length as the timestamps array.

Common Pitfalls

There are several common pitfalls to avoid when writing data to Synnax. These can lead to performance degradation and/or control issues.

Using Many Individual Write Calls Instead of a Writer

When writing large volumes of data in a streaming fashion (or in batches), use a writer instead of making individual write calls to a channel. Calls to write on a channel use an entirely new transaction for each call - constantly creating, committing, and closing transactions has a dramatic impact on performance.

Avoid this pattern - Writing directly to channels in a loop:

time = client.channels.retrieve("time")
temperature = client.channels.retrieve("temperature")

for i in range(100):
    ts = sy.TimeStamp.now()
    time.write(ts, ts)
    temperature.write(ts, i * 0.1)

Also avoid this pattern - Opening and closing a writer for every write:

for i in range(100):
    # Open/close writer each iteration
    with client.open_writer(
        start=sy.TimeStamp.now(),
        channels=["time", "temperature"],
    ) as writer:
        sy.sleep(0.1)
        writer.write({
            "time": sy.TimeStamp.now(),
            "temperature": i * 0.1,
        })

Recommended approach - Repeatedly call write on a single writer:

# Open/close the writer once
with client.open_writer(
    start=sy.TimeStamp.now(),
    channels=["time", "temperature"],
) as writer:
    sy.sleep(0.1)
    for i in range(100):
        writer.write({
            "time": sy.TimeStamp.now(),
            "temperature": i * 0.1,
        })