Nate's Blog

This blog post outlines the steps I took to automate uploading local media to Supabase and referencing the remote file.

When I first started this markdown based blog, I referenced local images and uploaded them to github. I realized this was a problem when I tried adding a large number of images to git and received the error that my commit was too large - filesizewise.

I migrated to using Supabase Buckets - a user friendly wrapper on top of an Amazon S3 bucket - to upload my media to and reference in my posts. This way, I'm not committing my media - just the remote reference to it. This adds a few additional steps though - manually uploading the file to Supabase, copying the URL and referencing the remote url in my markdown. This automates that process using a node script and a pre-commit hook that will run when I fire off a git commit command.

My development steps as as follow - using Bun to run TypeScript

1. upload media files to Supabase bucket
2. find local media references
3. replace local media references with remote references in blog
4. delete local media files
5. Put together full script to find local files, upload them to Supabase, replace the local references to remote and delete all the local files
6. Orchestrate script with pre-commit hook

Full script here

Uploading Media Files to Supabase Bucket

In this section, we'll focus on the uploadFile function, which handles the process of uploading media files to a Supabase bucket.

1import { createClient, SupabaseClient } from "@supabase/supabase-js";
2
3const supabaseUrl = process.env.NEXT_PUBLIC_SUPABASE_URL;
4const supabaseKey = process.env.NEXT_PUBLIC_SUPABASE_KEY;
5const bucketName = "natespilmanblog";
6const storageEndpoint = `${process.env.NEXT_PUBLIC_SUPABASE_URL}/storage/v1/object/public/${bucketName}`;
7
8if (!supabaseUrl || !supabaseKey) {
9  throw new Error(
10    "SUPABASE_URL and SUPABASE_KEY must be set in environment variables"
11  );
12}
13
14const bucketName = 'natespilmanblog'
15
16const uploadFile = async ({
17  filepath,
18  destinationDirName,
19}: {
20  filepath: string;
21  destinationDirName: string;
22}) => {
23  try {
24    const fileContent = await fs.readFile(`/${filepath}`);
25
26    const filename = filepath.split("/")[filepath.split("/").length - 1];
27    const uploadFilePath = `${destinationDirName}/${filename}`;
28
29    const { data, error } = await supabase.storage
30      .from(bucketName)
31      .upload(uploadFilePath, fileContent, {
32        contentType: "image/jpeg",
33        cacheControl: "3600",
34        upsert: true,
35      });
36
37    const successPayload = {
38      status: "success" as const,
39      localFilepath: filepath,
40      remoteFilepath: `${storageEndpoint}/${uploadFilePath}`,
41    };
42
43    if (error) {
44      if (error.message === "The resource already exists") {
45        console.log("File uploaded successfully:", data);
46
47        return successPayload;
48      } else {
49        console.error("Error uploading file:", error);
50      }
51      return {
52        status: "error" as const,
53        message: error.message,
54        localFilepath: filepath,
55      };
56    } else {
57      console.log("File uploaded successfully:", data);
58      return successPayload;
59    }
60  } catch (error) {
61    console.error("Error reading or uploading file:", error);
62    return {
63      status: "error" as const,
64      message: "Error reading or uploading file",
65      localFilepath: filepath,
66    };
67  }
68};
typescript

This function reads the file content, constructs the upload path, and uses the Supabase client to upload the file to the specified bucket. It handles both successful uploads and potential errors, returning a status object with relevant information.

Step 2 - Finding Local Media References

The findLocalMediaReferences function scans Markdown files for local media references:

1import fs from "fs/promises";
2import path from "path";
3
4async function findLocalMediaReferences(contentDir: string): Promise<
5  {
6    fullPath: string;
7    destinationDirName: string;
8  }[]
9> {
10  const mediaFiles: {
11    fullPath: string;
12    destinationDirName: string;
13  }[] = [];
14
15  // Find all .md files in the content directory
16  const allFiles = await walkDir(contentDir);
17  const mdFiles = allFiles.filter((file) => path.extname(file) === ".md");
18
19  for (const file of mdFiles) {
20    const content = await fs.readFile(file, "utf-8");
21
22    // Regular expression to match Markdown image and link syntax
23    const mediaRegex =
24      /!\[.*?\]\(((?!http|https:\/\/).*?)\)|(?<!!)\[.*?\]\(((?!http|https:\/\/).*?)\)/g;
25
26    const destinationDirName = file.split("/")[1].split(".")[0];
27
28    let match;
29    while ((match = mediaRegex.exec(content)) !== null) {
30      const mediaPath = match[1] || match[2];
31      if (isMediaFile(mediaPath)) {
32        const fullPath = path
33          .resolve(path.dirname(file), mediaPath)
34          .split("/")
35          .slice(1)
36          .join("/")
37          .split(" ")[0];
38        mediaFiles.push({ fullPath, destinationDirName });
39      }
40    }
41  }
42
43  return mediaFiles;
44}
typescript

This function uses a regular expression to find local media references in Markdown files, excluding external URLs. It returns an array of objects containing the full path and destination directory name for each media file.

Step 3 - Replacing Local Media References with Remote References

The replaceLocalReferenceWithRemote function updates the Markdown files to use the new remote URLs:

javascript

Copy

1async function walkDir(dir: string): Promise<string[]> {
2  const files = await fs.readdir(dir);
3  const paths = await Promise.all(
4    files.map(async (file) => {
5      const filePath = path.join(dir, file);
6      const stats = await fs.stat(filePath);
7      if (stats.isDirectory()) return walkDir(filePath);
8      else if (stats.isFile()) return filePath;
9      return [];
10    })
11  );
12  return paths.flat();
13}
14
15async function replaceLocalReferenceWithRemote(
16  status:
17    | {
18        status: "success";
19        localFilepath: string;
20        remoteFilepath: string;
21      }
22    | {
23        status: "error";
24        localFilepath: string;
25        message: string;
26      }
27) {
28  if (status.status !== "success") {
29    console.log(
30      `Skipping update for ${status.localFilepath} due to non-success status`
31    );
32    return;
33  }
34
35  const contentDir = "./blog"; // Adjust this to your content directory
36  const allFiles = await walkDir(contentDir);
37  const mdFiles = allFiles.filter((file) => path.extname(file) === ".md");
38
39  for (const file of mdFiles) {
40    let content = await fs.readFile(file, "utf-8");
41
42    // Escape special characters in the local filepath for use in regex
43    const filename =
44      status.localFilepath.split("/")[
45        status.localFilepath.split("/").length - 1
46      ];
47    const escapedLocalPath = filename.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
48
49    // Create a regex that matches the local filepath, accounting for possible relative paths and an optional preceding slash
50    const localPathRegex = new RegExp(
51      `(\\.\\.?\\/)*\\/?${escapedLocalPath}`,
52      "g"
53    );
54
55    if (localPathRegex.test(content)) {
56      // Replace all occurrences of the local path with the remote path, preserving any preceding slash
57      content = content.replace(localPathRegex, (match) => {
58        const hasLeadingSlash = match.startsWith("/");
59        return hasLeadingSlash
60          ? `/${status.remoteFilepath}`
61          : status.remoteFilepath;
62      });
63
64      // Write updated content back to the file
65      await fs.writeFile(file, content, "utf-8");
66      console.log(
67        `Updated references to ${status.localFilepath} in file: ${file}`
68      );
69      deleteLocalFile(`/${status.localFilepath}`);
70    }
71  }
72}
typescript

This function replaces all occurrences of the local file path with the new remote URL in the Markdown files, preserving any leading slashes.

Step 4 - Deleting Local Media Files

After uploading and updating references, we delete the local media files:

1async function deleteLocalFile(filepath: string) {
2  try {
3    await fs.unlink(filepath);
4    console.log(`Deleted local file: ${filepath}`);
5  } catch (error) {
6    console.error(`Error deleting local file ${filepath}:`, error);
7  }
8}
typescript

This function is called after successfully updating the references in the Markdown files.

Step 5 - Putting Together the Full Script

The main function orchestrates the entire process:

1const main = async () => {
2  const contentDirectory = "./blog"; // Adjust this to your content directory
3  await findLocalMediaReferences(contentDirectory)
4    .then((files) =>
5      files.forEach(async ({ fullPath, destinationDirName }) => {
6        const status = await uploadFile({
7          filepath: fullPath,
8          destinationDirName,
9        });
10        replaceLocalReferenceWithRemote(status);
11      })
12    )
13    .catch((error) => console.error("Error finding local media files:", error));
14};
typescript

This function ties together all the previous steps: finding local media references, uploading files, replacing references, and deleting local files.

Step 6 - Orchestrating the Script with a Pre-commit Hook

To run this script automatically before each commit, you can set up a pre-commit hook in your Git repository. Create a file named pre-commit in the .git/hooks/ directory of your repository with the following content:

1#!/bin/bash
2
3# Exit on any error
4set -e
5
6# Run your existing script to replace image references and delete images
7bun run upload-media
8
9# Stage all changes
10git add -A
11
12# Exit successfully
13exit 0
bash

Make sure to make the hook executable:

chmod +x .git/hooks/pre-commit

Now, every time I attempt to make a commit, this script will run automatically, ensuring that all media files are uploaded to Supabase and references are updated before the commit is made.

And Tadaaaaa - right now this image is a local reference on my computer, but will be sent to Supabase and referenced there as soon as I commit my changes to git.