Axios streams data b/w servers

Programming
, ,
1

I’m trying to stream data between two backends using Axios functions - downloadContent and uploadContent. The downloadContent function returns the data as a stream and the uploadContent function uses the stream to upload it.

However, when using the code below, the process memory usage is going up until it reaches the file size (5-10 GB+).

async function downloadContent(downloadUrl) {
  return await axios
    .get(downloadUrl, {
      responseType: "stream",
    })
    .then((r) => ({
      file: r.data,
      someOtherProps: {},
    }));
}

async function uploadContent() {
  const download = await downloadContent("download/url");

  return await axios.post("upload/url", download.file, {
    headers: {
      "Content-Type": "specific-content-type",
    },
  });
}

await uploadContent();

Is there something wrong with my approach? How can I achieve streaming between two servers to minimize the memory footprint with Axios?

2

There is an issue with your approach. The problem is that when you use await with downloadContent, it buffers the entire response into memory before returning it. This is why your process memory usage increases.

To minimize the memory footprint and achieve streaming between two servers using Axios, you can use the response.data stream directly to pipe it to the uploadContent function. Here’s an updated version of your code:

async function downloadContent(downloadUrl) {
  const response = await axios.get(downloadUrl, {
    responseType: "stream",
  });

  return {
    file: response.data,
    // Add other properties if needed
  };
}

async function uploadContent() {
  const download = await downloadContent("download/url");
  const uploadUrl = "upload/url";

  return new Promise((resolve, reject) => {
    download.file.pipe(
      axios.post(uploadUrl, null, {
        headers: {
          "Content-Type": "specific-content-type",
        },
      })
    )
    .then(resolve)
    .catch(reject);
  });
}

await uploadContent();

In this updated code, responseType: "stream" is used to get the response as a stream. Then, the response.data stream is directly piped to the axios.post request, avoiding buffering the entire response in memory.

This approach will help minimize the memory footprint when streaming data between two servers using Axios.