Perf Event Array map user-mode API implementation with tests (#4302)

* Perf Event Array map user-mode API implementation with tests

* CR Feedback

* Refactored and Unified the Ring Buffer and perf event array user mode API. Incorporated test fixes from Michael.

* Fixed build and Test issues

* Review Feedback Changes

* Fixed the AV in Perf_Buffer_free

* Used Conditional Variables to wait for callbacks to complete

* Added support for Using multiple subscription rings

* CR Comments

* Rename `_ebpf_map_subscription_ring` to `_ebpf_map_async_query_context`

For better readability, renaming `_ebpf_map_subscription_ring` to `_ebpf_map_async_query_context`.

* More CR comments

* Added Check to make sure ring_buffer_new does not pass more than 1 CPU ID

* More CR Feedback

---------

Co-authored-by: Shankar Seal <74580197+shankarseal@users.noreply.github.com>

Author: kumarvin123

docs/PerfEventArray.md

@@ -25,31 +25,26 @@ There are 3 primary differences between ring buffer maps and perf event arrays:
       - `perf_event_output` takes the bpf context as an argument and the helper implementation copies the payload.
           - The payload is whatever the data pointer of the program context points to (e.g. packet data including headers). The payload does not include the bpf program context structure itself.
 
-The main motivation for this proposal is to efficiently support payload capture from the context in bpf programs.
+The main motivation for this implementation is to efficiently support payload capture from the context in bpf programs.
 - Supporting ring buffer reserve and submit in ebpf-for-windows is currently blocked on verifier support [#273](https://github.com/vbpf/ebpf-verifier/issues/273).
 - Without reserve+submit, using `ringbuf_output` for payload capture requires using a per-CPU array as scratch space to append the payload to the event before calling ringbuf_output.
 - The CTXLEN field in the flags of `perf_event_output` tells the kernel to append bytes from the payload to the record, avoiding the extra copy.
   - On Linux this works for specific program types, on Windows this will work for any program type with a data pointer in the context.
 
 
-## Proposal
+The implementation behaviour matches Linux, but currently only supports user-space consumers and bpf-program producers with a subset of the features.
 
-The proposed behaviour matches Linux, but currently only supports user-space consumers and bpf-program producers with a subset of the features.
+The perf buffers are implemented using the existing per-CPU and ring buffer map support in ebpf-for-windows.
 
-The plan is to implement perf buffers using the existing per-CPU and ring buffer map support in ebpf-for-windows.
-
-To match Linux behaviour, by default the callback will only be called inside calls to `perf_buffer__poll()`.
-If the PERFBUF_FLAG_AUTO_CALLBACK flag is set, the callback will be automatically invoked when there is data available.
-
-1. Implement a new map type `BPF_MAP_TYPE_PERF_EVENT_ARRAY`.
+1. Implements a new map type `BPF_MAP_TYPE_PERF_EVENT_ARRAY`.
     1. Linux-compatible default behaviour.
         - With Linux-compatible behaviour and bpf interfaces, additional features from Linux should be possible to add in the future.
     2. Only support the perf ringbuffer (not other Linux perf features).
         - Only support bpf program producers with a single user-space consumer per event array.
         - Features not supported include perf counters, hardware-generated perf events,
           attaching bpf programs to perf events, and sending events from user-space to bpf programs.
     3. In addition to the Linux behaviour, automatically invoke the callback if the auto callback flag is set.
-2. Implement `perf_event_output` bpf helper function.
+2. Implements `perf_event_output` bpf helper function.
     1. Only support writing to the current CPU (matches current Linux restrictions).
         - Specify current CPU in flags using BPF_F_INDEX_MASK or pass BPF_F_CURRENT_CPU.
     2. Support BPF_F_CTXLEN_MASK flags for any bpf program types with a data pointer in the context.
@@ -58,15 +53,10 @@ If the PERFBUF_FLAG_AUTO_CALLBACK flag is set, the callback will be automaticall
           - The extension-provided ebpf_context_descriptor_t includes the offset of the data pointer.
         - Passing a non-zero value in BPF_F_CTXLEN_MASK will return an operation not supported error for program types
           without a data pointer in the context.
-2. Implement libbpf support for perf event arrays.
+2. Implements libbpf support for perf event arrays.
     1. `perf_buffer__new` - Create a new perfbuf manager (attaches callback).
         - Attaches to all CPUs automatically.
-    2. `perf_buffer__new_raw` - Not supported initially (can be future work).
-        - This function gives extra control over the perfbuf manager creation (e.g. which CPUs to attach).
     2. `perf_buffer__free` - Free perfbuf manager (detaches callback).
-    3. `perf_buffer__poll` - Wait the buffer to be non-empty (or timeout), then invoke callback for each ready record.
-        - By default (without `PERFBUF_FLAG_AUTO_CALLBACK`), the callback will not be called except inside poll() calls.
-        - poll() should not be called if the auto callback flag is set.
 
 ## bpf helpers
 ```c
@@ -102,48 +92,26 @@ typedef void (*perf_buffer_lost_fn)(void *ctx, int cpu, __u64 cnt);
 // Perf buffer manager options.
 struct perf_buffer_opts {
 	size_t sz;
-    uint64_t flags;
-};
-#define perf_buffer_opts__last_field flags
-
-// Flags for configuring perf buffer manager.
-enum perf_buffer_flags {
-    PERFBUF_FLAG_AUTO_CALLBACK = (uint64_t)1 << 0 /* Automatically invoke callback for each record */
 };
+#define perf_buffer_opts__last_field sz
 
 /**
- * @brief **perf_buffer__new()** creates BPF perfbuf manager for a specified
+ * @brief **perf_buffer__new()** creates BPF perfbuffer manager for a specified
  * BPF_PERF_EVENT_ARRAY map
  * @param map_fd FD of BPF_PERF_EVENT_ARRAY BPF map that will be used by BPF
  * code to send data over to user-space
- * @param page_cnt number of memory pages allocated for each per-CPU buffer
+ * @param page_cnt number of memory pages allocated for each per-CPU buffer. Should be set to 0.
  * @param sample_cb function called on each received data record
  * @param lost_cb function called when record loss has occurred
  * @param ctx user-provided extra context passed into *sample_cb* and *lost_cb*
- * @return a new instance of struct perf_buffer on success, NULL on error with
- * *errno* containing an error code
+ * @param opts perfbuffer manager options. Not supported currently. Should be null.
+ * @return a new instance of struct perf_buffer on success, NULL on error.
  */
 LIBBPF_API struct perf_buffer *
 perf_buffer__new(int map_fd, size_t page_cnt,
 		 perf_buffer_sample_fn sample_cb, perf_buffer_lost_fn lost_cb, void *ctx,
 		 const struct perf_buffer_opts *opts);
 
-/**
- * @brief poll perfbuf for new data
- * Poll for available data and consume records, if any are available.
- *
- * Must be called to receive callbacks by default (without auto callbacks).
- * NOT supported when PERFBUF_FLAG_AUTO_CALLBACK is set.
- *
- * If timeout_ms is zero, poll will not wait but only invoke the callback on records that are ready.
- * If timeout_ms is -1, poll will wait until data is ready (no timeout).
- *
- * @param[in] pb Pointer to perf buffer manager.
- * @param[in] timeout_ms maximum time to wait for (in milliseconds).
- *
- * @returns number of records consumed, INT_MAX, or a negative number on error
- */
-int perf_buffer__poll(struct perf_buffer *pb, int timeout_ms);
 /**
  * @brief Frees a perf buffer manager.
  *

ebpfapi/Source.def

@@ -133,6 +133,7 @@ EXPORTS
     ebpf_object_load_native_by_fds
     ebpf_object_set_execution_type
     ebpf_object_unpin
+    ebpf_perf_event_array_map_write
     ebpf_program_attach
     ebpf_program_attach_by_fd
     ebpf_program_attach_by_fds
@@ -151,5 +152,7 @@ EXPORTS
     libbpf_num_possible_cpus
     libbpf_prog_type_by_name
     libbpf_strerror
+    perf_buffer__free
+    perf_buffer__new
     ring_buffer__free
     ring_buffer__new

include/bpf/libbpf.h

@@ -943,6 +943,34 @@ bpf_program__flags(const struct bpf_program* prog);
 int
 bpf_program__set_flags(struct bpf_program* prog, __u32 flags);
 
+/**
+ * @brief Create BPF perfbuf manager.
+ *
+ * @param[in] map_fd File descriptor to perf event array map.
+ * @param[in] page_cnt Number of memory pages allocated for each per-CPU buffer. This should be set to 0.
+ * @param[in] sample_cb Pointer to perf buffer notification callback function.
+ * @param[in] lost_cb Function pointer for callback when record loss has occurred.
+ * @param[in] ctx User provided extra context passed into sample_cb and lost_cb.
+ * @param[in] opts The perf buffer manager options. This should be set to NULL.
+ * @return Pointer to perf buffer manager.
+ */
+LIBBPF_API struct perf_buffer*
+perf_buffer__new(
+    int map_fd,
+    size_t page_cnt,
+    perf_buffer_sample_fn sample_cb,
+    perf_buffer_lost_fn lost_cb,
+    void* ctx,
+    const struct perf_buffer_opts* opts);
+
+/**
+ * @brief Free a perf buffer manager.
+ *
+ * @param[in] rb Pointer to perf buffer manager to be freed.
+ */
+LIBBPF_API void
+perf_buffer__free(struct perf_buffer* pb);
+
 #else
 #pragma warning(push)
 #pragma warning(disable : 4200) // Zero-sized array in struct/union

libs/api/api_internal.h

@@ -13,8 +13,7 @@
 
 struct bpf_object;
 
-typedef struct _ebpf_ring_buffer_subscription ring_buffer_subscription_t;
-typedef struct _ebpf_perf_event_array_subscription perf_event_array_subscription_t;
+typedef struct _ebpf_map_subscription ebpf_map_subscription_t;
 
 typedef struct bpf_program
 {
@@ -607,64 +606,38 @@ ebpf_object_load(_Inout_ struct bpf_object* object) noexcept;
 EBPF_API_LOCKING _Must_inspect_result_ ebpf_result_t
 ebpf_object_unload(_Inout_ struct bpf_object* object) noexcept;
 
-typedef int (*ring_buffer_sample_fn)(void* ctx, void* data, size_t size);
-
 /**
- * @brief Subscribe for notifications from the input ring buffer map.
+ * @brief Subscribe for notifications from the input perf event array or a ring buffer map.
  *
- * @param[in] ring_buffer_map_fd File descriptor to the ring buffer map.
- * @param[in, out] sample_callback_context Pointer to supplied context to be passed in notification callback.
- * @param[in] sample_callback Function pointer to notification handler.
- * @param[out] subscription Opaque pointer to ring buffer subscription object.
- *
- * @retval EBPF_SUCCESS The operation was successful.
- * @retval EBPF_NO_MEMORY Out of memory.
- */
-_Must_inspect_result_ ebpf_result_t
-ebpf_ring_buffer_map_subscribe(
-    fd_t ring_buffer_map_fd,
-    _Inout_opt_ void* sample_callback_context,
-    ring_buffer_sample_fn sample_callback,
-    _Outptr_ ring_buffer_subscription_t** subscription) noexcept;
-
-/**
- * @brief Unsubscribe from the ring buffer map event notifications.
- *
- * @param[in] subscription Pointer to ring buffer subscription to be canceled.
- */
-bool
-ebpf_ring_buffer_map_unsubscribe(_In_ _Post_invalid_ ring_buffer_subscription_t* subscription) noexcept;
-
-typedef void (*perf_buffer_sample_fn)(void* ctx, int cpu, void* data, uint32_t size);
-typedef void (*perf_buffer_lost_fn)(void* ctx, int cpu, uint64_t cnt);
-
-/**
- * @brief Subscribe for notifications from the input perf event array map.
- *
- * @param[in] perf_event_array_map_fd File descriptor to the perf event array map.
- * @param[in, out] callback_context Pointer to supplied context to be passed in notification callback.
+ * @param[in] map_fd File descriptor to the perf event array or a ring buffer map.
+ * @param[in] cpu_ids The CPU Ids corresponding to this subscription. For a ring buffer map this is a single value with
+ * Id 0.
+ * @param[in] cpu_id_count The count of the elements in the cpu_ids parameter.
+ * @param[in] callback_context Pointer to supplied context to be passed in notification callback.
  * @param[in] sample_callback Function pointer to notification handler.
  * @param[in] lost_callback Function pointer to lost record notification handler.
- * @param[out] subscription Opaque pointer to perf event array subscription object.
+ * @param[out] subscription Opaque pointer to the subscription object.
  *
  * @retval EBPF_SUCCESS The operation was successful.
  * @retval EBPF_NO_MEMORY Out of memory.
  */
 _Must_inspect_result_ ebpf_result_t
-ebpf_perf_event_array_map_subscribe(
-    fd_t perf_event_array_map_fd,
+ebpf_map_subscribe(
+    fd_t map_fd,
+    _In_reads_(cpu_id_count) uint32_t* cpu_ids,
+    _In_ size_t cpu_id_count,
     _Inout_opt_ void* callback_context,
-    perf_buffer_sample_fn sample_callback,
-    perf_buffer_lost_fn lost_callback,
-    _Outptr_ perf_event_array_subscription_t** subscription) noexcept;
+    _In_ const void* sample_callback,
+    _In_opt_ const void* lost_callback,
+    _Outptr_ ebpf_map_subscription_t** subscription) noexcept;
 
 /**
- * @brief Unsubscribe from the perf event array map event notifications.
+ * @brief Unsubscribe from the map event notifications.
  *
- * @param[in] subscription Pointer to perf event array subscription to be canceled.
+ * @param[in] subscription Pointer to subscription to be canceled.
  */
 bool
-ebpf_perf_event_array_map_unsubscribe(_In_ _Post_invalid_ perf_event_array_subscription_t* subscription) noexcept;
+ebpf_map_unsubscribe(_In_ _Post_invalid_ ebpf_map_subscription_t* subscription) noexcept;
 
 /**
  * @brief Get list of programs and stats in an ELF eBPF file.