Go OverlayFS

go-bds/overlayfs is a Go library that provides an abstraction for creating and managing overlay or merge filesystems. It aims to use native operating system functionalities where possible and offers a pure Go fallback implementation.

This allows you to merge multiple directory layers (some read-only, one potentially read-write) into a single, unified view.

Features

• Cross-Platform Support (with caveats):
  • Linux: Utilizes the kernel's native OverlayFS.
  • Windows (amd64, 386, arm64): Uses WinFsp via github.com/aegistudio/go-winfsp to provide merge filesystem capabilities.
  • FreeBSD: Attempts to use unionfs.ko if available.
  • Other Platforms: Falls back to a pure Go implementation of a merge filesystem (MergeFS).
• Read-Write & Read-Only Modes: Supports creating read-write overlays (with an "upper" directory for changes) or read-only merged views.
• io/fs.FS Compatibility: The pure Go MergeFS implementation fully supports the io/fs.FS interface, including:
  • fs.FS
  • fs.ReadFileFS
  • fs.ReadDirFS
  • fs.StatFS
  • fs.GlobFS
  • fs.SubFS
• File Operations for Go MergeFS: The Go-native MergeFS also implements common file system operations:
  • Open, Create, ReadFile, WriteFile
  • Stat, Lstat, ReadDir
  • Mkdir, MkdirAll, Remove, RemoveAll
  • Chmod, Chown
  • Rename, Symlink, Readlink, Truncate
  • OpenFile
• Whiteout Support: The Go-native MergeFS handles .wh. (whiteout) files in the upper directory to mark files/directories from lower layers as deleted.

Installation

go get -u sirherobrine23.com.br/go-bds/overlayfs@latest

Usage

Creating an OverlayFS Instance

import "sirherobrine23.com.br/go-bds/overlayfs"

// For a read-write overlay
ovReadWrite := overlayfs.NewOverlayFS(
    "/mnt/merged_target", // Target mount point
    "/path/to/upper_rw",  // Upper, read-write layer
    "/path/to/workdir",   // Workdir (required for Linux kernel overlayfs)
    "/path/to/lower1_ro", // Lower, read-only layer 1
    "/path/to/lower2_ro", // Lower, read-only layer 2
)

// For a read-only merge
ovReadOnly := overlayfs.NewOverlayFS(
    "/mnt/readonly_target", // Target mount point
    "",                     // No upper layer means read-only
    "",                     // No workdir needed if no upper layer (except Linux if it still requires it)
    "/path/to/base1",
    "/path/to/base2",
)

Mounting and Unmounting (Platform-Specific)

// Ensure directories exist
// os.MkdirAll(ovReadWrite.Target, 0755)
// os.MkdirAll(ovReadWrite.Upper, 0755)
// os.MkdirAll(ovReadWrite.Workdir, 0755)
// ... and for lower dirs

// Mount
err := ovReadWrite.Mount()
if err != nil {
    if errors.Is(err, overlayfs.ErrNotOverlayAvaible) {
        fmt.Println("Native overlayfs not available on this system. Consider using Mergefs() for Go-native operations.")
    } else if errors.Is(err, overlayfs.ErrNoCGOAvaible) {
        // Specific to FreeBSD unionfs check in this library
        fmt.Println("CGO is disabled, cannot perform syscalls for mounting.")
    } else {
        log.Fatalf("Failed to mount: %v", err)
    }
}
fmt.Println("Mounted successfully at:", ovReadWrite.Target)

// ... perform operations on the mounted filesystem ...

// Unmount
err = ovReadWrite.Unmount()
if err != nil {
    log.Fatalf("Failed to unmount: %v", err)
}
fmt.Println("Unmounted successfully.")

Notes:

• Permissions: Mounting usually requires elevated privileges (e.g., root on Linux/FreeBSD, Administrator on Windows).
• WinFsp: On Windows, WinFsp must be installed for Mount() to succeed.
• FreeBSD: unionfs.ko kernel module needs to be available.
• Linux: Workdir must be an empty directory on the same filesystem as the Upper directory.

Using the Go-Native MergeFS (io/fs.FS)

If native mounting is not available or not desired, you can use the Go-native MergeFS implementation which provides an io/fs.FS interface and direct file operation methods.

package main

import (
    "fmt"
    "io/fs"
    "log"
    "os"
    "path/filepath"

    "sirherobrine23.com.br/go-bds/overlayfs"
)

func main() {
    // Setup temporary directories for example
    tmpDir, err := os.MkdirTemp("", "mergefs-example")
    if err != nil {
        log.Fatal(err)
    }
    defer os.RemoveAll(tmpDir)

    upperDir := filepath.Join(tmpDir, "upper")
    lower1Dir := filepath.Join(tmpDir, "lower1")
    lower2Dir := filepath.Join(tmpDir, "lower2")

    os.Mkdir(upperDir, 0755)
    os.Mkdir(lower1Dir, 0755)
    os.Mkdir(lower2Dir, 0755)

    // Create some files
    os.WriteFile(filepath.Join(lower1Dir, "file1.txt"), []byte("Content from lower1"), 0644)
    os.WriteFile(filepath.Join(lower2Dir, "file2.txt"), []byte("Content from lower2"), 0644)
    os.WriteFile(filepath.Join(lower1Dir, "override.txt"), []byte("Original in lower1"), 0644)
    os.WriteFile(filepath.Join(upperDir, "file3.txt"), []byte("Content from upper"), 0644)
    os.WriteFile(filepath.Join(upperDir, "override.txt"), []byte("Overridden in upper"), 0644)

    // Create Overlayfs instance (Target and Workdir are not used by MergeFS directly)
    ov := overlayfs.NewOverlayFS(
        "", // Target not used by MergeFS methods directly
        upperDir,
        "", // Workdir not used by MergeFS
        lower1Dir,
        lower2Dir,
    )

    // Get the io/fs.FS compatible interface
    mergedFS := ov.Mergefs()

    // Example: Read a file
    content, err := fs.ReadFile(mergedFS, "file1.txt")
    if err != nil {
        log.Fatalf("ReadFile (file1.txt): %v", err)
    }
    fmt.Printf("file1.txt: %s\n", content) // Output: Content from lower1

    content, err = fs.ReadFile(mergedFS, "file3.txt")
    if err != nil {
        log.Fatalf("ReadFile (file3.txt): %v", err)
    }
    fmt.Printf("file3.txt: %s\n", content) // Output: Content from upper

    content, err = fs.ReadFile(mergedFS, "override.txt")
    if err != nil {
        log.Fatalf("ReadFile (override.txt): %v", err)
    }
    fmt.Printf("override.txt: %s\n", content) // Output: Overridden in upper

    // Example: List directory
    entries, err := fs.ReadDir(mergedFS, ".")
    if err != nil {
        log.Fatalf("ReadDir (.): %v", err)
    }
    fmt.Println("\nDirectory listing:")
    for _, entry := range entries {
        fmt.Printf("- %s (Is dir: %t)\n", entry.Name(), entry.IsDir())
    }

    // Example: Using direct MergeFS methods for writing
    err = ov.WriteFile("newfile.txt", []byte("This is a new file written via MergeFS"), 0644)
    if err != nil {
        log.Fatalf("ov.WriteFile (newfile.txt): %v", err)
    }
    fmt.Println("\nCreated newfile.txt in upper layer.")

    // Verify newfile.txt exists in the actual upper directory
    _, err = os.Stat(filepath.Join(upperDir, "newfile.txt"))
    if err != nil {
        log.Fatalf("Stat on actual upperDir/newfile.txt failed: %v", err)
    }

    // Example: Deleting a file (will create a .wh. file in upper)
    err = ov.Remove("file1.txt") // file1.txt was in lower1
    if err != nil {
        log.Fatalf("ov.Remove (file1.txt): %v", err)
    }
    fmt.Println("Removed file1.txt (created whiteout in upper layer).")

    // Try to read it again, should fail
    _, err = fs.ReadFile(mergedFS, "file1.txt")
    if err == nil || !errors.Is(err, fs.ErrNotExist) {
        log.Fatalf("ReadFile (file1.txt) after remove should be ErrNotExist, got: %v", err)
    }
    fmt.Println("Attempting to read file1.txt after removal correctly results in 'not exist'.")

    // Check for whiteout file
    _, err = os.Stat(filepath.Join(upperDir, ".wh.file1.txt"))
    if err != nil {
        log.Fatalf("Whiteout file .wh.file1.txt not found in upperDir: %v", err)
    }
    fmt.Println("Whiteout file .wh.file1.txt found in upper layer.")
}

Contributing

Contributions are welcome! Please feel free to submit pull requests or open issues.