Logstash http output plugin: difference between json and json_batch format

Written by - 0 comments

Published on - Listed in ELK Observability Logstash Monitoring

Logstash using the http output plugin sending by default json format.

When you use Logstash's http output plugin, you can send logs to a HTTP endpoint (e.g. to a HTTP API), sometimes also named logsink.

   if [@metadata][target] == "externalanalysis" {
    http {
      url => "https://logsink.example.com"
      http_method => "post"
      format => "json"
      headers => { "Authorization" => "Basic YWxhZGRpbjpvcGVuc2VzYW1l" }
    }
  }

Note: The snippet above is placed inside Logstash's configuration, within the output {} context. The meta field [@metadata][target] was added using a mutate filter.

The "format" option allows to define different kinds of formats, how the log event should be sent to the HTTP API. The default setting is "json".

There's also "json_batch" as an option. The official documentation describes json_batch in the following way:

If json_batch, each batch of events received by this output will be placed into a single JSON array and sent in one request. This is particularly useful for high throughput scenarios such as sending data between Logstash instaces.

That sounds very theoretical, without an actual output example. The most important information is that an event is placed into a single JSON array. But what does that mean?

To better understand the actual difference between "json" and "json_batch", here are two examples how logs are shipped from Logstash to a HTTP endpoint.

json:

{"request_user_agent":"\"curl/7.68.0\"","timestamp":"04/Jan/2025:00:15:57 +0100","message":"Hello World","app":"Example","env":"Production"}

json_batch:

[{"request_user_agent":"\"curl/7.68.0\"","timestamp":"04/Jan/2025:00:15:57 +0100","message":"Hello World","app":"Example","env":"Production"}]

The json message is now placed inside a single array, using the [] square brackets.

Whether a json_batch (array) format is accepted, depends on the HTTP endpoint.


More recent articles:

RSS feed

Blog Tags:

  AWS   Android   Ansible   Apache   Apple   Atlassian   BSD   Backup   Bash   Bluecoat   CMS   Chef   Cloud   Coding   Consul   Containers   CouchDB   DB   DNS   Database   Databases   Docker   ELK   Elasticsearch   Filebeat   FreeBSD   Galera   Git   GlusterFS   Grafana   Graphics   HAProxy   HTML   Hacks   Hardware   Icinga   Influx   Internet   Java   KVM   Kibana   Kodi   Kubernetes   LVM   LXC   Linux   Logstash   Mac   Macintosh   Mail   MariaDB   Minio   MongoDB   Monitoring   Multimedia   MySQL   NFS   Nagios   Network   Nginx   OSSEC   OTRS   Observability   Office   OpenSearch   PGSQL   PHP   Perl   Personal   PostgreSQL   Postgres   PowerDNS   Proxmox   Proxy   Python   Rancher   Rant   Redis   Roundcube   SSL   Samba   Seafile   Security   Shell   SmartOS   Solaris   Surveillance   Systemd   TLS   Tomcat   Ubuntu   Unix   VMWare   VMware   Varnish   Virtualization   Windows   Wireless   Wordpress   Wyse   ZFS   Zoneminder