Class PulsarSink

java.lang.Object
io.numaproj.numaflow.sinker.Sinker
io.numaproj.pulsar.producer.PulsarSink
public class PulsarSink extends io.numaproj.numaflow.sinker.Sinker
Numaflow user-defined sink that publishes messages to a Pulsar topic. Supports Schema.BYTES (raw bytes) or Schema.AUTO_PRODUCE_BYTES (broker-validated payloads). The target topic must already exist; existence is checked on startup by PulsarProducerConfig.

  • Constructor Summary

    Constructors
    Constructor
    Description
    PulsarSink(org.apache.pulsar.client.api.Producer<byte[]> producer, org.apache.pulsar.client.api.PulsarClient pulsarClient, PulsarProducerProperties producerProperties)
    Creates a new sink.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    cleanup()
    Flushes buffered records and closes the producer and client.
    io.numaproj.numaflow.sinker.ResponseList
    processMessages(io.numaproj.numaflow.sinker.DatumIterator datumIterator)
    Publishes every datum in the batch to Pulsar and waits for all acks before returning.
    void
    startServer()
    Starts the Numaflow gRPC server and blocks until it terminates.

    Methods inherited from class java.lang.Object

    equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Constructor Details

    • PulsarSink

      public PulsarSink(org.apache.pulsar.client.api.Producer<byte[]> producer, org.apache.pulsar.client.api.PulsarClient pulsarClient, PulsarProducerProperties producerProperties)
      Creates a new sink.
      Parameters:
      producer - the Pulsar producer bound to the target topic
      pulsarClient - the Pulsar client, closed during cleanup
      producerProperties - parsed pulsar.producer config

  • Method Details

    • startServer

      public void startServer() throws Exception
      Starts the Numaflow gRPC server and blocks until it terminates.
      Throws:
      Exception - if the gRPC server fails

    • processMessages

      public io.numaproj.numaflow.sinker.ResponseList processMessages(io.numaproj.numaflow.sinker.DatumIterator datumIterator)
      Publishes every datum in the batch to Pulsar and waits for all acks before returning. Schema validation failures are either dropped or surfaced as per-message failures depending on producerProperties.dropInvalidMessages.
      Specified by:
      processMessages in class io.numaproj.numaflow.sinker.Sinker
      Parameters:
      datumIterator - the datum iterator
      Returns:
      per-message responses to report back to Numaflow

    • cleanup

      public void cleanup()
      Flushes buffered records and closes the producer and client. Errors are logged rather than thrown.