Setting up Cloudflare

Logging

Forwarding logs to our platform is a necessary step in setting up your Valyu integration. This process enables us to provide comprehensive analytics on bot traffic and other metrics, offering a centralised view of your bot traffic. Check below for the instructions to set up analytics for Cloudflare.

Before getting started

Before setting up logging make sure you have onboarded your domain on the Valyu Exchange platform. Go to Add Data and onboard your domain.

Getting started

To forward logs from Cloudflare, first go to your Cloudflare Dashboard and go to the Workers & Pages section on the left sidebar and click Overview.

Once you are on the Overview page click Create Worker. This will take you to the page shown in the image below. Set the name to "valyu-logging" and then click Deploy. We will modify the code after the worker has deployed.

Configuring the worker

Once the worker has been deployed, click Edit code.

In the worker.js file, delete all existing code and copy the code found below over exactly. Make sure to replace YOUR_API_KEY_HERE with the secret key you can find in your Api Keys Section.

const CF_APP_VERSION = "1.0.0";

const valyuLogEndpoint = "https://internal.api.valyu.network/v1/cloudflare";
const valyuApiKey = "YOUR_API_KEY_HERE";

/**
 * Utility function to pause execution for a specified number of milliseconds
 * @param ms Number of milliseconds to sleep
 */
const sleep = (ms) => {
  return new Promise((resolve) => {
    setTimeout(resolve, ms);
  });
};

/**
 * Generates a random ID string of specified length using alphanumeric characters (excluding 'O')
 * @param length Length of the ID to generate
 */
const makeid = (length) => {
  let text = "";
  const possible = "ABCDEFGHIJKLMNPQRSTUVWXYZ0123456789";
  for (let i = 0; i < length; i += 1) {
    text += possible.charAt(Math.floor(Math.random() * possible.length));
  }
  return text;
};

/**
 * Creates a standardized log object from request and response data
 * Captures geographic, request, and response information for logging
 * @param request The incoming request object
 * @param response The outgoing response object
 */
const buildLogMessage = (request, response) => {
  const logObject = {
    timestamp: new Date().toISOString(),
    client_ip: "", // worker only is able to get cloudflare edge IP, leaving blank
    geo_country: request.cf["country"],
    geo_city: request.cf["city"],
    geo_postal_code: request.cf["postalCode"],
    geo_latitude: request.cf["latitude"],
    geo_longitude: request.cf["longitude"],
    host: request.headers.get("host"),
    url: request.url.replace("https://" + request.headers.get("host"), ""),
    request_method: request.method,
    request_protocol: request.cf["httpProtocol"],
    request_user_agent: request.headers.get("user-agent"),
    request_latency: null, // cloudflare does not have latency information
    response_state: null,
    response_status: response.status,
    response_reason: response.statusText,
    response_body_size: response.contentLength,
  };
  return logObject;
};

// Batching
const BATCH_INTERVAL_MS = 20000; // 20 seconds
const MAX_REQUESTS_PER_BATCH = 500; // 500 logs
const WORKER_ID = makeid(6);

let workerTimestamp;

let batchTimeoutReached = true;
let logEventsBatch = [];

// Backoff
const BACKOFF_INTERVAL = 10000;
let backoff = 0;

/**
 * Adds a log event to the batch queue and triggers processing if batch size limit is reached
 * @param body The log event to add to the batch
 * @param event The Cloudflare worker event
 */
async function addToBatch(body, event) {
  logEventsBatch.push(body);

  if (logEventsBatch.length >= MAX_REQUESTS_PER_BATCH) {
    event.waitUntil(postBatch(event));
  }

  return true;
}

/**
 * Main request handler that processes incoming requests
 * Creates a log event for each request and adds it to the batch queue
 * @param event The Cloudflare worker event
 */
async function handleRequest(event) {
  const { request } = event;

  const response = await fetch(request);
  const rCf = request.cf;
  delete rCf.tlsClientAuth;
  delete rCf.tlsExportedAuthenticator;

  const eventBody = buildLogMessage(request, response);
  event.waitUntil(addToBatch(eventBody, event));

  return response;
}

/**
 * Sends the batch of logs to the API endpoint with backoff handling
 * Implements rate limiting protection by setting a backoff timer on 403/429 responses
 * @param lfRequest The request to send to the logging API
 * @param event The Cloudflare worker event
 */
const fetchAndSetBackOff = async (lfRequest, event) => {
  if (backoff <= Date.now()) {
    const resp = await fetch(valyuLogEndpoint, lfRequest);
    if (resp.status === 403 || resp.status === 429) {
      backoff = Date.now() + BACKOFF_INTERVAL;
    }
  }

  event.waitUntil(scheduleBatch(event));

  return true;
};

/**
 * Processes and sends the current batch of log events to the API
 * Clears the batch queue after sending
 * @param event The Cloudflare worker event
 */
const postBatch = async (event) => {
  const batchInFlight = [...logEventsBatch.map((e) => JSON.stringify(e))];
  logEventsBatch = [];
  const body = batchInFlight.join("\n");
  const request = {
    method: "POST",
    headers: {
      "x-api-key": `${valyuApiKey}`,
      "Content-Type": "application/json",
    },
    body,
  };
  event.waitUntil(fetchAndSetBackOff(request, event));
};

/**
 * Schedules periodic processing of batched log events
 * Ensures batches are sent even if they haven't reached the size limit
 * @param event The Cloudflare worker event
 */
const scheduleBatch = async (event) => {
  if (batchTimeoutReached) {
    batchTimeoutReached = false;
    await sleep(BATCH_INTERVAL_MS);
    if (logEventsBatch.length > 0) {
      event.waitUntil(postBatch(event));
    }
    batchTimeoutReached = true;
  }
  return true;
};

addEventListener("fetch", (event) => {
  event.passThroughOnException();

  if (!workerTimestamp) {
    workerTimestamp = new Date().toISOString();
  }

  event.waitUntil(scheduleBatch(event));
  event.respondWith(handleRequest(event));
});

Once pasted, click Save and deploy on the upper righthand corner once you are finished.

Link worker to CloudFlare HTTP Logs

Go to Websites on the left sidebar and choose the website that you would like to forward logs for, and click into it. On the left panel, click on Worker Routes, and then click Add route.

Set the route to ".<your-site.com>/" which is the default recommend path by Cloudflare, or a custom path if you only want to forward logs for certain URL patterns.

Under workers, choose the valyu-logging worker that you just created.

Click Save once you are done.

Once this has set up, your logs will be forwarded to Valyu and you will be able to view your Bot Traffic on the Valyu Exchange platform.