WHAT ARE YOU LOOKING FOR?

Suricata keyword update: viewing entropy values

Suricata Affichages : 383

Introduction

In my blog introducing the Entropy keyword, I mentioned that we were working on a way to include the calculated entropy value in the output to assist in tuning entropy values specified with the entropy keyword.  Suricata has been extended to include the calculated entropy value in the log output. Although the Cyberchef entropy calculator is available to calculate entropy values, the Suricata log output may be more accessible.

This blog builds upon the Emotet malware example from the previous blog and demonstrates the entropy output in the JSON logs generated by Suricata.

To briefly recap, the entropy keyword in Suricata enables detection based on the Shannon entropy of inspected content or sticky buffers, allowing detection based on different levels of randomness (entropy) in the data. By specifying an entropy threshold and optionally anchoring the evaluation using depth and offset, rule authors can detect anomalous or suspicious patterns, often characteristic of encrypted, compressed, or obfuscated content.

With Suricata’s 8.0.0 rc1 release, flows subjected to a rule containing the entropy keyword will also include the calculated entropy in the log output, regardless of whether an alert was generated.

Output format

When a calculated entropy value is available, i.e., it has been calculated for a flow while evaluating a rule containing the entropy keyword, the calculated entropy value and the sticky buffer that provided the data are included in every log event type for the flow.
The calculated entropy value will be in the metadata section and includes the sticky buffer anchoring the data on which entropy was calculated – here’s an example  from a record with event type flow (note that the same format is used for all event types):

"metadata": {
    "entropy": {
      "file_data": 7.923338628443025
    }
}

Note that all log activity for the same flow will have the calculated entropy value recorded in the log as shown. The following list shows common event types that will include an entropy value:

  • alert (When the calculated entropy value “matches” the value specification.)
  • flow
  • http
  • fileinfo
  • smb
  • smtp

Note: throughout this post, the JSON samples will be collapsed, to ease readability.

Choosing an entropy value

Entropy values can continue to be calculated using Cyberchef’s entropy calculator. Another way is to view the values in Suricata’s log output. We’ll review the Emotet example from the previous blog, which introduced the entropy keyword. Recall that the Shannon entropy calculation produces values between 0 and 8, with values of 0 showing the least entropy and values of 8 indicating the maximum entropy.

As a review,  Emotet is an information stealer that was first reported in 2014. It has since evolved – see the footnote for more information. 

The Emotet process is multi-step and eventually involves downloading a Windows binary to establish a command and control (C2) endpoint.

Here’s a straightforward set of rules to detect a high-entropy file object in a web server response. The IPs are not redacted – the Emotet pcaps are publicly available.

Previously, we demonstrated how Proofpoint ET Info rules detected Emotet activity and presented a modified rule incorporating the entropy keyword to enhance confidence.  

Here are the ET rules:

alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"ET INFO GET Minimal HTTP Headers Flowbit Set"; flow:established,to_server; flowbits:set,min.gethttp; flowbits:noalert; http.method; content:"GET"; http.header_names; content:!"Accept"; content:!"If-"; content:!"Referer"; content:!"User-Agent"; content:!"Content"; classtype:bad-unknown; sid:2016537; rev:4; metadata:created_at 2013_03_06, updated_at 2020_08_28;)

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download"; flowbits:isset,min.gethttp; flow:established,to_client; file_data; content:"MZ"; within:2; content:"PE|00 00|"; distance:0; classtype:bad-unknown; sid:2016538; rev:3; metadata:created_at 2013_03_06, updated_at 2013_03_06;)

This rule extends the second listed rule with the entropy keyword to increase confidence that the downloaded executable has high entropy. The initial entropy value range of 6-8 was chosen using the Cyberchef calculator:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download"; flowbits:isset,min.gethttp; flow:established,to_client; file_data; content:"MZ"; within:2; content:"PE|00 00|"; distance:0; entropy: value 6-8; classtype:bad-unknown; sid:2016538; rev:4; metadata:created_at 2013_03_06, updated_at 2025_05_18;)

Running Suricata with these rules and the Emotet pcap produces:

Alert log [click to expand JSON] 
{
  "timestamp": "2021-01-06T16:41:15.583105+0000",
  "flow_id": 1596500678558203,
  "event_type": "alert",
  "src_ip": "103.92.235.25",
  "src_port": 80,
  "dest_ip": "10.1.6.206",
  "dest_port": 49775,
  "proto": "TCP",
  "ip_v": 4,
  "pkt_src": "stream (flow timeout)",
  "metadata": {
    "flowbits": [
      "min.gethttp"
    ],
    "entropy": {
      "file_data": 7.470398128939753
    }
  },
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 111112,
    "rev": 4,
    "signature": "ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download",
    "category": "Potentially Bad Traffic",
    "severity": 2,
    "metadata": {
      "created_at": [
        "2013_03_06"
      ],
      "updated_at": [
        "2025_05_18"
      ]
    }
  },
  "ts_progress": "request_complete",
  "tc_progress": "response_complete",
  "http": {
    "hostname": "seo.udaipurkart.com",
    "url": "/rx-5700-6hnr7/Sgms/",
    "http_content_type": "application/octet-stream",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 192099
  },
  "files": [
    {
      "filename": "nDUrg8uFD5hl.dll",
      "gaps": false,
      "state": "CLOSED",
      "sha256": "8e37a82ff94c03a5be3f9dd76b9dfc335a0f70efc0d8fd3dca9ca34dd287de1b",
      "stored": false,
      "storing": true,
      "size": 192000,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 30,
    "pkts_toclient": 142,
    "bytes_toserver": 1720,
    "bytes_toclient": 200343,
    "start": "2021-01-06T16:41:17.699394+0000",
    "src_ip": "10.1.6.206",
    "dest_ip": "103.92.235.25",
    "src_port": 49775,
    "dest_port": 80
  }
}
Flow log [click to expand JSON]
 
{
  "timestamp": "2021-01-06T16:41:15.583105+0000",
  "flow_id": 1596500678558203,
  "event_type": "flow",
  "src_ip": "10.1.6.206",
  "src_port": 49775,
  "dest_ip": "103.92.235.25",
  "dest_port": 80,
  "ip_v": 4,
  "proto": "TCP",
  "app_proto": "http",
  "flow": {
    "pkts_toserver": 30,
    "pkts_toclient": 142,
    "bytes_toserver": 1720,
    "bytes_toclient": 200343,
    "start": "2021-01-06T16:41:17.699394+0000",
    "end": "2021-01-06T16:42:28.579445+0000",
    "age": 71,
    "state": "closed",
    "reason": "shutdown",
    "alerted": true,
    "tx_cnt": 1
  },
  "metadata": {
    "flowbits": [
      "min.gethttp"
    ],
    "entropy": {
      "file_data": 7.470398128939753
    }
  },
  "tcp": {
    "tcp_flags": "1e",
    "tcp_flags_ts": "1e",
    "tcp_flags_tc": "1a",
    "syn": true,
    "rst": true,
    "psh": true,
    "ack": true,
    "state": "closed",
    "ts_max_regions": 1,
    "tc_max_regions": 1
  }
}
Fileinfo log [click to expand JSON] 
{
  "timestamp": "2021-01-06T16:41:15.583105+0000",
  "flow_id": 1596500678558203,
  "event_type": "fileinfo",
  "src_ip": "103.92.235.25",
  "src_port": 80,
  "dest_ip": "10.1.6.206",
  "dest_port": 49775,
  "proto": "TCP",
  "ip_v": 4,
  "pkt_src": "stream (flow timeout)",
  "metadata": {
    "flowbits": [
      "min.gethttp"
    ],
    "entropy": {
      "file_data": 7.470398128939753
    }
  },
  "http": {
    "hostname": "seo.udaipurkart.com",
    "url": "/rx-5700-6hnr7/Sgms/",
    "http_content_type": "application/octet-stream",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 192099
  },
  "app_proto": "http",
  "fileinfo": {
    "filename": "nDUrg8uFD5hl.dll",
    "gaps": false,
    "state": "CLOSED",
    "sha256": "8e37a82ff94c03a5be3f9dd76b9dfc335a0f70efc0d8fd3dca9ca34dd287de1b",
    "stored": true,
    "file_id": 6,
    "size": 192000,
    "tx_id": 0
  }
}

Wrap-up

To conclude this post, we’ll use this information to tighten our rule. We started with entropy: value 6-8. Since the calculated entropy value is now known, we’ll tighten the value range: value 7.4-7.5.

Here’s the modified rule:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download"; flowbits:isset,min.gethttp; flow:established,to_client; file_data; content:"MZ"; within:2; content:"PE|00 00|"; distance:0; entropy: value 7.4-7.5; classtype:bad-unknown; sid:111112; rev:4; metadata:created_at 2013_03_06, updated_at 2025_05_18;)
Expand the alert JSON the rule generates – note the sid is 111112 
{
  "timestamp": "2021-01-06T16:41:15.583105+0000",
  "flow_id": 1596501949760458,
  "event_type": "alert",
  "src_ip": "103.92.235.25",
  "src_port": 80,
  "dest_ip": "10.1.6.206",
  "dest_port": 49775,
  "proto": "TCP",
  "ip_v": 4,
  "pkt_src": "stream (flow timeout)",
  "metadata": {
    "flowbits": [
      "min.gethttp"
    ],
    "entropy": {
      "file_data": 7.470398128939753
    }
  },
  "tx_id": 0,
  "alert": {
    "action": "allowed",
    "gid": 1,
    "signature_id": 111112,
    "rev": 4,
    "signature": "ET INFO Executable Retrieved With Minimal HTTP Headers - Potential Second Stage Download",
    "category": "Potentially Bad Traffic",
    "severity": 2,
    "metadata": {
      "created_at": [
        "2013_03_06"
      ],
      "updated_at": [
        "2025_05_18"
      ]
    }
  },
  "ts_progress": "request_complete",
  "tc_progress": "response_complete",
  "http": {
    "hostname": "seo.udaipurkart.com",
    "url": "/rx-5700-6hnr7/Sgms/",
    "http_content_type": "application/octet-stream",
    "http_method": "GET",
    "protocol": "HTTP/1.1",
    "status": 200,
    "length": 192099
  },
  "files": [
    {
      "filename": "nDUrg8uFD5hl.dll",
      "gaps": false,
      "state": "CLOSED",
      "sha256": "8e37a82ff94c03a5be3f9dd76b9dfc335a0f70efc0d8fd3dca9ca34dd287de1b",
      "stored": false,
      "storing": true,
      "size": 192000,
      "tx_id": 0
    }
  ],
  "app_proto": "http",
  "direction": "to_client",
  "flow": {
    "pkts_toserver": 30,
    "pkts_toclient": 142,
    "bytes_toserver": 1720,
    "bytes_toclient": 200343,
    "start": "2021-01-06T16:41:17.699394+0000",
    "src_ip": "10.1.6.206",
    "dest_ip": "103.92.235.25",
    "src_port": 49775,
    "dest_port": 80
  }
}

References

Related links for further reading:

The post Suricata keyword update: viewing entropy values appeared first on Suricata.

Image
Back To Top

Pensée du jour :

Ce que l'homme a fait ,

l'homme peut le défaire.

 

"No secure path in the world"

Category

Popular Sections

About