Chris Collins

Tools · infra

Kafka client

Work with a real Kafka cluster from the browser through a REST proxy (Confluent REST Proxy or Redpanda): brokers and configs, topics with partitions, ISR and offsets, produce single records or batches, consume by group or partition with live tail, consumer group lag and offset resets — and a log of every HTTP call, with copy as curl.

Added

How to use it

A browser cannot speak the Kafka protocol: it has no raw TCP sockets. So this tool talks to an HTTP gateway that sits next to your cluster — the Confluent REST Proxy (its v3 cluster API, plus v2 for consuming) or Redpanda’s HTTP Proxy (pandaproxy, v2-compatible), with the Redpanda Admin API for what the proxy leaves out. Enter the proxy’s URL, press Connect, and the tool works out which one it is. Every request it makes is listed on the Requests tab, so the page doubles as a way to learn the REST APIs.

To learn how Kafka works without a cluster, use the Kafka simulator instead; it animates partitions, replication, consumer groups and delivery guarantees.

How to use it

Cluster shows the proxy type, cluster id, controller and brokers. Configs lists a broker’s settings (v3), or the cluster config from the Redpanda Admin API.

Topics lists topics with partition counts and replication factors. Click one to see each partition’s leader, replicas, in-sync replicas (out-of-sync ones in red), earliest and latest offsets and how many records it holds, then its configs — key ones such as cleanup.policy, retention.ms and min.insync.replicas first, explained, with durations and sizes in plain units. With the v3 API you can edit a config, create a topic (partitions, replication factor, configs) and delete one (you type its name to confirm).

Produce sends one record: a key and a value as text, JSON (validated) or base64 bytes, headers (name: value per line), and a partition — or leave it empty to let the key choose. Null (tombstone) sends a null value, which deletes the key from a compacted topic. The result shows the partition and offset the record was written at. Batch produce takes pasted lines — one value per line, key:value lines, or NDJSON records such as {"key": "order-1", "value": {"qty": 1}, "partition": 0, "headers": {"source": "web"}} — checks them, sends them, and lists where each one landed.

Consume reads a topic two ways. Consumer group subscribes a consumer in a named group: the group starts from its committed offsets, Earliest/Latest apply only where it has none, and nothing is committed until you press Commit. Partitions reads chosen partitions (all, 0, or 0,2) from the earliest offset, the latest, or an offset you give, without joining a group. Without Live tail, reading stops once it has caught up (Resume reads more); with it, new records keep arriving until you pause or stop. The table shows partition, offset, key, value and size; click a row for the pretty-printed JSON, copy buttons and Re-produce…. Search keys, values and headers (plain text or a regular expression), filter by partition, sort, and export what is shown as NDJSON or CSV. The tool keeps the newest 500, 2,000 or 10,000 records.

Groups lists consumer groups and their state; click one for its members and their assignments, and each partition’s committed offset, log-end offset and lag (the v3 lag API). Reset offsets moves a group to the earliest or latest offset, a given offset, or shifts it by a number of records, for a whole topic or some partitions. Preview it, then apply. Kafka only accepts new offsets while the group has no active members, so stop its consumers first. With Redpanda, which cannot list groups, enter a group and a topic to read its committed offsets and lag.

Requests lists every HTTP call: time, method, URL, status and duration. Click one to see its headers and bodies. Copy as curl gives a command you can run in a terminal, with $KAFKA_USER, $KAFKA_PASSWORD or $KAFKA_TOKEN in place of your credentials, and Download as script saves them all.

History keeps the topics you used and the messages you produced (up to 50) in this browser only; Load into producer sends one again.

Running a proxy

The page is served over HTTPS from https://geekychris.github.io, so the proxy must answer that origin with CORS headers. Chrome, Edge and Firefox let an HTTPS page call http://localhost; any other plain http:// address is blocked, so use https:// there. Safari may refuse http://localhost too. Chrome may ask whether this site may reach devices on your local network: allow it.

Confluent REST Proxy next to a Kafka on localhost:9092 (Linux, where --network host shares the host’s network):

docker run -d --name kafka-rest --network host \
  -e KAFKA_REST_BOOTSTRAP_SERVERS=localhost:9092 \
  -e KAFKA_REST_LISTENERS=http://0.0.0.0:8082 \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_ORIGIN=https://geekychris.github.io \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_METHODS=GET,POST,PUT,DELETE,OPTIONS,HEAD \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_HEADERS=origin,content-type,accept,authorization \
  confluentinc/cp-kafka-rest:7.7.1

On macOS and Windows use -p 8082:8082 and KAFKA_REST_BOOTSTRAP_SERVERS=host.docker.internal:9092 instead. That only works if the broker advertises an address the container can reach. A broker that advertises localhost:9092 hands the proxy an address inside its own container. Add a listener advertised as host.docker.internal, or run the REST proxy from the Confluent Platform download (kafka-rest-start kafka-rest.properties) with these lines in its properties file:

bootstrap.servers=localhost:9092
listeners=http://0.0.0.0:8082
access.control.allow.origin=https://geekychris.github.io
access.control.allow.methods=GET,POST,PUT,DELETE,OPTIONS,HEAD
access.control.allow.headers=origin,content-type,accept,authorization

access.control.allow.origin takes a comma-separated list (add http://localhost:1313 for a local preview of this site), and access.control.allow.headers must include authorization if you use credentials.

A throwaway Kafka (KRaft, no ZooKeeper) and REST proxy, nothing exposed but the proxy on 8082:

docker network create kafka-lab
docker run -d --name kafka --network kafka-lab \
  -e KAFKA_NODE_ID=1 -e KAFKA_PROCESS_ROLES=broker,controller \
  -e KAFKA_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093 \
  -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092 \
  -e KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER \
  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT \
  -e KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:9093 \
  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
  -e KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1 -e KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1 \
  apache/kafka:3.8.0
docker run -d --name kafka-rest --network kafka-lab -p 8082:8082 \
  -e KAFKA_REST_BOOTSTRAP_SERVERS=kafka:9092 -e KAFKA_REST_LISTENERS=http://0.0.0.0:8082 \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_ORIGIN=https://geekychris.github.io \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_METHODS=GET,POST,PUT,DELETE,OPTIONS,HEAD \
  -e KAFKA_REST_ACCESS_CONTROL_ALLOW_HEADERS=origin,content-type,accept,authorization \
  confluentinc/cp-kafka-rest:7.7.1

Remove it with docker rm -f kafka kafka-rest && docker network rm kafka-lab.

Redpanda, single node, with its HTTP Proxy on 8082 and Admin API on 9644 (its Kafka API is on 19092, so it does not clash with a Kafka on 9092):

docker run -d --name redpanda -p 19092:19092 -p 8082:8082 -p 9644:9644 \
  redpandadata/redpanda:latest redpanda start --mode dev-container --smp 1 \
  --kafka-addr internal://0.0.0.0:9092,external://0.0.0.0:19092 \
  --advertise-kafka-addr internal://localhost:9092,external://localhost:19092 \
  --pandaproxy-addr 0.0.0.0:8082 --advertise-pandaproxy-addr localhost:8082

With the gitwebsite app. If the gitwebsite app (or gw serve) is running, set your proxy URLs once in its Companion settings (kafka_rest, redpanda_proxy, redpanda_admin), choose ⚡ gitwebsite app above the proxy URL and Connect: the app relays the REST proxy and the Admin API (and turns POST
X-Method-Override: GET into the GET-with-body Redpanda needs), so no CORS relay is needed. Basic or bearer credentials you enter still go to the proxy, beside the app’s own token.

Not needed if the gitwebsite app is running — otherwise: Redpanda’s HTTP Proxy and Admin API send no CORS headers, so run this small relay beside them (Node.js, no packages). It adds CORS for https://geekychris.github.io and serves the Admin API under /admin. Then connect to http://localhost:8090 with Admin API http://localhost:8090/admin, or open this preset. The tool also offers the script under Admin API and authentication → Download the relay.

node kafka-cors-relay.js 8090 http://localhost:8082 http://localhost:9644
// kafka-cors-relay.js: CORS in front of a Kafka REST proxy (and the Redpanda Admin API).
// Not needed when the gitwebsite app is running: its companion relays the proxy and the Admin API.
// node kafka-cors-relay.js [port] [proxy URL] [admin URL] [allowed origins, comma-separated]
const http = require('http'), https = require('https');
const [port = 8090, proxy = 'http://localhost:8082', admin = '',
  origins = 'https://geekychris.github.io'] = process.argv.slice(2);
const allowed = origins.split(',');
http.createServer((req, res) => {
  const origin = req.headers.origin;
  if (origin && (allowed.includes(origin) || allowed.includes('*'))) {
    res.setHeader('Access-Control-Allow-Origin', origin);
    res.setHeader('Vary', 'Origin');
    res.setHeader('Access-Control-Allow-Methods', 'GET, POST, PUT, DELETE, OPTIONS');
    res.setHeader('Access-Control-Allow-Headers', 'Authorization, Content-Type, Accept, X-Method-Override');
    res.setHeader('Access-Control-Expose-Headers', 'Server');
    res.setHeader('Access-Control-Allow-Private-Network', 'true');
  }
  if (req.method === 'OPTIONS') return res.writeHead(204).end();
  const toAdmin = admin && req.url.startsWith('/admin/');
  const target = new URL(toAdmin ? req.url.slice(6) : req.url, toAdmin ? admin : proxy);
  const headers = { ...req.headers, host: target.host };
  delete headers.origin; delete headers.referer; delete headers['x-method-override'];
  const method = req.headers['x-method-override'] || req.method;
  const up = (target.protocol === 'https:' ? https : http).request(target, { method, headers }, r => {
    res.writeHead(r.statusCode, r.headers);
    r.pipe(res);
  });
  up.on('error', e => res.writeHead(502, { 'Content-Type': 'application/json' })
    .end(JSON.stringify({ error_code: 502, message: 'relay: ' + e.message })));
  req.pipe(up);
}).listen(+port, '127.0.0.1', () => console.log('CORS relay on http://localhost:' + port + ' -> ' + proxy + (admin ? ', /admin -> ' + admin : '')));

The relay also helps with any proxy you cannot reconfigure. It listens on 127.0.0.1 only.

Credentials. Basic auth (user and password) and bearer tokens are sent in an Authorization header to the proxy and nowhere else. They stay in the page’s memory unless you tick Remember credentials in this browser, which keeps them in local storage. They are never put in links, the request log or copied curl commands. The Confluent REST Proxy does basic auth with authentication.method=BASIC and a JAAS realm. Redpanda’s HTTP Proxy authenticates with its SASL users when authentication_method: http_basic is set on the listener.

Limits

  • Neither proxy returns a record’s timestamp or headers when consuming (their v2 consumer API has no such fields), so those columns appear only if a proxy sends them, and reading from a timestamp is not possible.

  • Record headers can only be produced through the Confluent v3 API.

  • Creating, deleting and configuring topics, listing consumer groups and their members, and the lag API need the Confluent v3 API. Redpanda’s HTTP Proxy has none of them: use rpk.

  • Redpanda’s HTTP Proxy answers a batch with the first offset of each partition it wrote to, not where each record went, and does not place records by key the way Kafka clients do. Tick One request per record to see every record’s partition and offset.

  • Redpanda only accepts earliest as a new consumer’s starting point. To start a new group at Latest, the tool first commits each partition’s current end for that group (this needs the Admin API URL). Its partition offsets come from the Admin API’s debug view (Raft offsets less its configuration entries), checked with one read; treat them as estimates.

  • Confluent’s v2 commit stores the offset after the one given; the tool allows for this, so offsets shown and entered are always the next record to read, as Kafka means them.

  • A consumer instance lives on the proxy. The tool deletes it when you stop or leave the page, and the proxy drops idle ones after about five minutes.

  • Values are shown as text, JSON or hex. Avro and Protobuf records (Schema Registry) are shown as raw bytes.

  • The tool talks only to the proxy you enter; nothing passes through this site.

Related: Kafka simulator (how Kafka works, no cluster needed), HTTP client, WebSocket tester, JSON formatter.

Linking to an example

Share link copies a link with the tool’s settings: the topic, consume and produce settings, and the proxy URL and Admin API URL with any user:password@, query string and fragment removed. Credentials and history are never included. A link never connects by itself. Put the tab after #: cluster, topics, produce, consume, groups, log (requests) or history.

Parameter Meaning Example

example

one of the presets below; parameters written beside it override it

example=tail-latest

proxy

the REST proxy URL (credentials and query strings are dropped)

proxy=http://localhost:8082

admin

the Redpanda Admin API URL

admin=http://localhost:8090/admin

backend

confluent, redpanda or v2 instead of detecting

backend=redpanda

topic

the topic used by every tab

topic=orders

mode

consume by group or by assign (chosen partitions)

mode=assign

group

consumer group (Consume and Groups)

group=billing

partition

partitions to read: all, 0 or 0,2

partition=0

from

earliest, latest or offset

from=offset

offset

the offset to start from

offset=1200

live

1 for live tail

live=1

filter

search text for the records table

filter=error

key, kfmt

the key to produce and its format: text, json or base64

key=order-1

value, fmt

the value to produce and its format

fmt=json

to

the partition to produce to (empty: by key)

to=2

headers

headers to produce, name: value per line

headers=source%3A%20web

batch, bmode

lines for batch produce and how to read them: lines, kv or ndjson

bmode=ndjson

Presets:

example= Opens

local-confluent

the connection set to a Confluent REST Proxy on localhost:8082 (open)

local-redpanda

the connection set to Redpanda through the relay on localhost:8090, with its Admin API (open)

produce-order

a JSON order keyed by order id, with a header, ready to produce to orders (open)

batch-ndjson

three NDJSON records for batch produce; two share a key, so they land in one partition (open)

tail-latest

live tail of new records on orders as group gw-tail (open)

replay-partition

partition 0 of orders from offset 0, without a group (open)

from-beginning

every partition of orders from the earliest offset (open)

group-lag

members, committed offsets and lag of group orders-service (open)

topic-configs

the partitions, offsets and configs of orders (open)

A hand-written link: ?proxy=http://localhost:8082&topic=orders&mode=assign&partition=0&from=offset&offset=100#consume.

Runs entirely in your browser — network requests go only to the services named on this page.