fs

The package fs offers a more complete altenative to the io/fs package of the Go standard library.

The idea was to find common abstractions for paths and filesystems, no matter, if they are local, remote, backed by memory, drives of cloud servers.

API docs

see https://pkg.go.dev/gitlab.com/golang-utils/fs

design decisions

If became clear soon, that such a general fs library can't be done without rethinking and integrating the concept of a path.

Paths should work in a similar way, no matter, if they are local windows paths, windows UNC paths or unix paths. No matter if the path is part of a URL or local, if it has a drive letter or a network share.

The fs/path package makes Path a proper type and has relative and absolute paths as different types. Every path can have a relative part and every path can be represented by a string.

Therefor the common path.Path interface is as follows:

type Path interface {
	String() string
	Relative() Relative
}

where path.Relative is just a string:

type Relative string

However, since a path.Relative is also a path.Path, it implements that interface:

func (r Relative) String() string {
	return string(r)
}

func (r Relative) Relative() Relative {
	return r
}

Very simple. It became clear, that an absolute path is always related to a filesystem, while a relative path is independant of the filesystem. Therefor path.Absolute became an interface, since the differences between local os paths and also remote paths show up in their absolute part:

type Absolute interface {
	Path
	Head() string
}

So a path.Absolute is just a path.Path with a Head. The easiest implementation of this path.Absolute is a local path:

type Local [2]string

func (a Local) Head() string {
	return a[0]
}

func (a Local) String() string {
	return a[0] + a[1]
}

func (a Local) Relative() Relative {
	return Relative(a[1])
}

Here some design decision come into place:

1. the relative part of a path always uses the slash / as a separator.
2. directories always end with a /. this makes it easy to check, if a path refering to a directory without the the need of a filesystem.
3. the head of an absolute path is always a directory and therefor it always ends with a slash /
4. parts of paths are joined together simply by glueing them together. Since a directory must end in a slash / this naturally leads to correct paths.
5. the head of an absolute path is depending on the filesystem that it refers to: e.g.
   • a local windows paths starts with a drive letter, e.g. c:/
   • a windows share in UNC starts with the host followed by the share name, e.g. \\example.com/share/
   • a url starts with a schema, followed by a host http://example.com/
6. absolute paths can be written differently, e.g.
   • c:/ can also be written as C:\
   • \\example.com/share/ can also be written as \\example.com\share\ therefor we need one unique internal representation, while allowing the path to be generated by parsing also the alternative ways of writing. this leads to parsers for converting an absolute path string to a path.Absolute.
7. While having a unified syntax behind the scenes, path.Local can be converted to the most typical string notation that is used on the running system, by calling the ToSystem method, so that it can easily be integrated with external tools
8. The only time where a local absolute path is being created via solo a relative path, is when the relative path is relative to the current working directory of the system. This case is handled like every other way to convert a string to a path.Local by the path.ParseLocal() function (see below)

Local and Remote

Not only because of the different ways the local and remote absolute paths are written, but also because of the very different performance characteristics and optimization opportunities, it makes sense to be able to distinguish between local and remote paths, while still being able to handle them both as absolute paths.

This is taken into account via having path.Local as well as path.Remote implement the path.Absolute interface. This way we end up with two parsers:

• ParseLocal(string) handling local windows, UNC and unix paths, also paths relativ to the working directory, e.g. ./a/
• ParseRemote(string) handling URLs

A path.Remote is basically just a wrapper around an url.URL that is implementing the path.Absolute interface.

type Remote struct {
	*url.URL
}

func (u *Remote) Relative() Relative {...}
func (u *Remote) Head() string {...}

There are some helpers, to get the common string notation for windows, UNC and so on back, so that everything does integrate well.

filesystems

Since it became clear that absolute paths are associated with a filesystem, this lead to filesystems being initiated via an path.Absolute.

For good integration with the existing io/fs.FS interface which provides only a solution for reading access, we started with the fs.ReadOnly filesystem interface:

type ReadOnly interface {
	Reader(p path.Relative) (io.ReadCloser, error)
	Exists(p path.Relative) bool
	ModTime(p path.Relative) (time.Time, error)
	Abs(p path.Relative) path.Absolute
	Size(p path.Relative) int64
}

It is very easy to convert an existing io/fs.FS implementation to the fs.ReadOnly interface via the wrapfs package:

dir := "/etc"
fsys := os.DirFS(dir)
ro, err := wrapfs.New(fsys, path.MustLocal(dir+"/")) 
size := ro.Size("fstab")

For os.DirFS there is an easier way via the localfs package:

fs, err := localfs.New(path.MustLocal("/etc/"))
size := fs.Size("fstab")

The real power comes with the more general fs.FS interface:

type FS interface {
	ReadOnly
	ExtWriteable
	ExtDeleteable
}

It adds to the ReadOnly interface the ability to write and delete files and folders. This is also implemented by the localfs package:

fs, err := localfs.New(path.MustLocal(`C:\`))
recursive := true
err = fs.Delete(path.Relative("Windows/"), recursive)

But the same powerful interface is also implemented by the httpsfs package which accesses a filesystem via http:

fs, err := httpsfs.New(path.MustRemote(`http://localhost:3030/data/`))
createDirs := true
err = fs.Write(path.Relative("myblog/january/something-new.txt"), fs.ReadCloser(strings.NewReader("some text")), createDirs)

Finally we have some properties specifically for local filesystems that we don't have for remote filesystems and vice versa:

fs, err := localfs.New(path.MustLocal(`/`))
err = fs.SetMode(path.Relative("etc/"), 0750)

fs, err := httpsfs.New(path.MustRemote(`http://localhost:3030/data/`))
meta := map[string][]byte{"Content-Type": []byte("application/json")}
data := fs.ReadCloser(strings.NewReader(`{key: "val"}`))
err = fs.WriteWithMeta(path.Relative("sth.json"), data, meta, true) 

So we have this hierarchy of FS interfaces where the last ones a more specific but also more powerfull and the first ones are more general and easier to implement:

type ReadOnly interface {
	Reader(p path.Relative) (io.ReadCloser, error)
	Exists(p path.Relative) bool
	ModTime(p path.Relative) (time.Time, error)
	Abs(p path.Relative) path.Absolute
	Size(p path.Relative) int64 
}

type FS interface {
	ReadOnly
	ExtWriteable
	ExtDeleteable
}

type Local interface {
	FS
	ExtMoveable
	ExtModeable
	ExtRenameable
	ExtSpaceReporter
}

type Remote interface {
	FS
	ExtMeta
	ExtURL
}

Finally we have TestFS interface that is covering everything, so that can easily test our packages against all features:

type TestFS interface {
	Local
	Remote
}

The mockfs package offers an implementation of the fs.TestFS interface that is backed by a map for easy testing.

For implementors

For implementors there is a large test suite that can be easily integrated into your package testing to ensure that your filesystem behaves correctly according to the specifications. Here an example how to use it, based on the mockfs package:

package mockfs

import (
	"testing"

	"gitlab.com/golang-utils/fs"
	"gitlab.com/golang-utils/fs/path"
	"gitlab.com/golang-utils/fs/spec"
)

func mustNew(loc path.Absolute) fs.TestFS {
	f, err := New(loc)
	if err != nil {
		panic(err.Error())
	}
	return f
}

func TestSpec(t *testing.T) {
	var c spec.Config
	s := spec.TestFS(c, mustNew)
	s.Run("", t)
}