eBPF: stack limit exceeded when storing stack var in map

Programming
, , , ,
1

Issue

I am having an error while compiling a eBPF program:

Looks like the BPF stack limit of 512 bytes is exceeded. Please move large on stack variables into BPF per-cpu array map.

The program is as follows:

#include <stdio.h>
#include <string.h>

#include <linux/bpf.h>
#include <sys/socket.h>

#include <bpf/bpf_helpers.h>

char LICENSE[] SEC("license") = "GPL";

// msg_data_map carries a key-value pair of (msg_id, msg_length), and can record
// upto 65535 messages at once.
#define MAX_MSG_LEN 128
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 65535);
  __type(key, int);
  __type(value, char[MAX_MSG_LEN]);
} msg_data_map SEC(".maps");

SEC("sk_msg")
int msg_prog(struct sk_msg_md *msg) {
  long len = (long)msg->data_end - (long)msg->data;

  void *data_end = (void *)(long) msg->data_end;
  void *data = (void *)(long) msg->data;

  // Bounds check to make verifier happy
  if (data + MAX_MSG_LEN > data_end) {
    return SK_PASS;
  }

  char buf[MAX_MSG_LEN] = {0};
  if (len > MAX_MSG_LEN) {
    __builtin_memcpy(buf, data, MAX_MSG_LEN);
  } else {
    __builtin_memcpy(buf, data, len);
  }

  // Update in map
  int index = 0;
  //bpf_map_update_elem(&msg_data_map, &index, &buf, BPF_ANY);
 
  return SK_PASS;
}

I am getting an error when compiling the program while trying to update a map. The buf array is only 128 bytes, which should not exceed the stack limit. Commenting out the map update lines compiles the program without any issues.

Why am I getting this error while trying to update a map?

2

The error message suggests that the BPF stack limit of 512 bytes is exceeded and advises moving large on stack variables into a BPF per-cpu array map.

In the given program, the buf array is declared as char buf[MAX_MSG_LEN], where MAX_MSG_LEN is 128. This array is used to store the message data.

To resolve the error, you need to move the buf array from the stack to a BPF per-cpu array map.

Here is the updated program:

#include <stdio.h>
#include <string.h>

#include <linux/bpf.h>
#include <sys/socket.h>

#include <bpf/bpf_helpers.h>

char LICENSE[] SEC("license") = "GPL";

// msg_data_map carries a key-value pair of (msg_id, msg_length), and can record
// upto 65535 messages at once.
#define MAX_MSG_LEN 128
struct {
  __uint(type, BPF_MAP_TYPE_HASH);
  __uint(max_entries, 65535);
  __type(key, int);
  __type(value, char[MAX_MSG_LEN]);
} msg_data_map SEC(".maps");

SEC("sk_msg")
int msg_prog(struct sk_msg_md *msg) {
  long len = (long)msg->data_end - (long)msg->data;

  void *data_end = (void *)(long) msg->data_end;
  void *data = (void *)(long) msg->data;

  // Bounds check to make verifier happy
  if (data + MAX_MSG_LEN > data_end) {
    return SK_PASS;
  }

  // Get a pointer to the map value
  char *buf = bpf_map_lookup_elem(&msg_data_map, &index);
  if (!buf) {
    return SK_PASS;
  }

  if (len > MAX_MSG_LEN) {
    __builtin_memcpy(buf, data, MAX_MSG_LEN);
  } else {
    __builtin_memcpy(buf, data, len);
  }
 
  return SK_PASS;
}

In the updated program, the buf array is replaced with a pointer obtained from the bpf_map_lookup_elem function. This function looks up the map element based on the given key (&index in this case) and returns a pointer to the value. The value is then populated with the message data using __builtin_memcpy.

Make sure to update the rest of your program accordingly.

By moving the buf array into a BPF per-cpu array map, you should no longer encounter the stack limit error while updating the map.