diff options
| author | Bjørn Erik Pedersen <[email protected]> | 2023-12-24 19:11:05 +0100 |
|---|---|---|
| committer | Bjørn Erik Pedersen <[email protected]> | 2024-01-27 16:28:14 +0100 |
| commit | 7285e74090852b5d52f25e577850fa75f4aa8573 (patch) | |
| tree | 54d07cb4a7de2db5c89f2590266595f0aca6cbd6 /hugofs | |
| parent | 5fd1e7490305570872d3899f5edda950903c5213 (diff) | |
| download | hugo-7285e74090852b5d52f25e577850fa75f4aa8573.tar.gz hugo-7285e74090852b5d52f25e577850fa75f4aa8573.zip | |
all: Rework page store, add a dynacache, improve partial rebuilds, and some general spring cleaningdevelop2024
There are some breaking changes in this commit, see #11455.
Closes #11455
Closes #11549
This fixes a set of bugs (see issue list) and it is also paying some technical debt accumulated over the years. We now build with Staticcheck enabled in the CI build.
The performance should be about the same as before for regular sized Hugo sites, but it should perform and scale much better to larger data sets, as objects that uses lots of memory (e.g. rendered Markdown, big JSON files read into maps with transform.Unmarshal etc.) will now get automatically garbage collected if needed. Performance on partial rebuilds when running the server in fast render mode should be the same, but the change detection should be much more accurate.
A list of the notable new features:
* A new dependency tracker that covers (almost) all of Hugo's API and is used to do fine grained partial rebuilds when running the server.
* A new and simpler tree document store which allows fast lookups and prefix-walking in all dimensions (e.g. language) concurrently.
* You can now configure an upper memory limit allowing for much larger data sets and/or running on lower specced PCs.
We have lifted the "no resources in sub folders" restriction for branch bundles (e.g. sections).
Memory Limit
* Hugos will, by default, set aside a quarter of the total system memory, but you can set this via the OS environment variable HUGO_MEMORYLIMIT (in gigabytes). This is backed by a partitioned LRU cache used throughout Hugo. A cache that gets dynamically resized in low memory situations, allowing Go's Garbage Collector to free the memory.
New Dependency Tracker: Hugo has had a rule based coarse grained approach to server rebuilds that has worked mostly pretty well, but there have been some surprises (e.g. stale content). This is now revamped with a new dependency tracker that can quickly calculate the delta given a changed resource (e.g. a content file, template, JS file etc.). This handles transitive relations, e.g. $page -> js.Build -> JS import, or $page1.Content -> render hook -> site.GetPage -> $page2.Title, or $page1.Content -> shortcode -> partial -> site.RegularPages -> $page2.Content -> shortcode ..., and should also handle changes to aggregated values (e.g. site.Lastmod) effectively.
This covers all of Hugo's API with 2 known exceptions (a list that may not be fully exhaustive):
Changes to files loaded with template func os.ReadFile may not be handled correctly. We recommend loading resources with resources.Get
Changes to Hugo objects (e.g. Page) passed in the template context to lang.Translate may not be detected correctly. We recommend having simple i18n templates without too much data context passed in other than simple types such as strings and numbers.
Note that the cachebuster configuration (when A changes then rebuild B) works well with the above, but we recommend that you revise that configuration, as it in most situations should not be needed. One example where it is still needed is with TailwindCSS and using changes to hugo_stats.json to trigger new CSS rebuilds.
Document Store: Previously, a little simplified, we split the document store (where we store pages and resources) in a tree per language. This worked pretty well, but the structure made some operations harder than they needed to be. We have now restructured it into one Radix tree for all languages. Internally the language is considered to be a dimension of that tree, and the tree can be viewed in all dimensions concurrently. This makes some operations re. language simpler (e.g. finding translations is just a slice range), but the idea is that it should also be relatively inexpensive to add more dimensions if needed (e.g. role).
Fixes #10169
Fixes #10364
Fixes #10482
Fixes #10630
Fixes #10656
Fixes #10694
Fixes #10918
Fixes #11262
Fixes #11439
Fixes #11453
Fixes #11457
Fixes #11466
Fixes #11540
Fixes #11551
Fixes #11556
Fixes #11654
Fixes #11661
Fixes #11663
Fixes #11664
Fixes #11669
Fixes #11671
Fixes #11807
Fixes #11808
Fixes #11809
Fixes #11815
Fixes #11840
Fixes #11853
Fixes #11860
Fixes #11883
Fixes #11904
Fixes #7388
Fixes #7425
Fixes #7436
Fixes #7544
Fixes #7882
Fixes #7960
Fixes #8255
Fixes #8307
Fixes #8863
Fixes #8927
Fixes #9192
Fixes #9324
Diffstat (limited to 'hugofs')
28 files changed, 1303 insertions, 1907 deletions
diff --git a/hugofs/component_fs.go b/hugofs/component_fs.go new file mode 100644 index 000000000..c55f15957 --- /dev/null +++ b/hugofs/component_fs.go @@ -0,0 +1,284 @@ +// Copyright 2024 The Hugo Authors. All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package hugofs + +import ( + iofs "io/fs" + "os" + "path" + "runtime" + "sort" + + "github.com/gohugoio/hugo/common/herrors" + "github.com/gohugoio/hugo/common/hstrings" + "github.com/gohugoio/hugo/common/paths" + "github.com/gohugoio/hugo/hugofs/files" + "github.com/spf13/afero" + "golang.org/x/text/unicode/norm" +) + +// NewComponentFs creates a new component filesystem. +func NewComponentFs(opts ComponentFsOptions) *componentFs { + if opts.Component == "" { + panic("ComponentFsOptions.PathParser.Component must be set") + } + if opts.Fs == nil { + panic("ComponentFsOptions.Fs must be set") + } + bfs := NewBasePathFs(opts.Fs, opts.Component) + return &componentFs{Fs: bfs, opts: opts} +} + +var _ FilesystemUnwrapper = (*componentFs)(nil) + +// componentFs is a filesystem that holds one of the Hugo components, e.g. content, layouts etc. +type componentFs struct { + afero.Fs + + opts ComponentFsOptions +} + +func (fs *componentFs) UnwrapFilesystem() afero.Fs { + return fs.Fs +} + +type componentFsDir struct { + *noOpRegularFileOps + DirOnlyOps + name string // the name passed to Open + fs *componentFs +} + +// ReadDir reads count entries from this virtual directorie and +// sorts the entries according to the component filesystem rules. +func (f *componentFsDir) ReadDir(count int) ([]iofs.DirEntry, error) { + fis, err := f.DirOnlyOps.(iofs.ReadDirFile).ReadDir(-1) + if err != nil { + return nil, err + } + + // Filter out any symlinks. + n := 0 + for _, fi := range fis { + // IsDir will always be false for symlinks. + keep := fi.IsDir() + if !keep { + // This is unfortunate, but is the only way to determine if it is a symlink. + info, err := fi.Info() + if err != nil { + if herrors.IsNotExist(err) { + continue + } + return nil, err + } + if info.Mode()&os.ModeSymlink == 0 { + keep = true + } + } + if keep { + fis[n] = fi + n++ + } + } + + fis = fis[:n] + + for _, fi := range fis { + s := path.Join(f.name, fi.Name()) + _ = f.fs.applyMeta(fi, s) + + } + + sort.Slice(fis, func(i, j int) bool { + fimi, fimj := fis[i].(FileMetaInfo), fis[j].(FileMetaInfo) + if fimi.IsDir() != fimj.IsDir() { + return fimi.IsDir() + } + fimim, fimjm := fimi.Meta(), fimj.Meta() + + if fimim.ModuleOrdinal != fimjm.ModuleOrdinal { + switch f.fs.opts.Component { + case files.ComponentFolderI18n: + // The way the language files gets loaded means that + // we need to provide the least important files first (e.g. the theme files). + return fimim.ModuleOrdinal > fimjm.ModuleOrdinal + default: + return fimim.ModuleOrdinal < fimjm.ModuleOrdinal + } + } + + pii, pij := fimim.PathInfo, fimjm.PathInfo + if pii != nil { + basei, basej := pii.Base(), pij.Base() + exti, extj := pii.Ext(), pij.Ext() + if f.fs.opts.Component == files.ComponentFolderContent { + // Pull bundles to the top. + if pii.IsBundle() != pij.IsBundle() { + return pii.IsBundle() + } + } + + if exti != extj { + // This pulls .md above .html. + return exti > extj + } + + if basei != basej { + return basei < basej + } + } + + if fimim.Weight != fimjm.Weight { + return fimim.Weight > fimjm.Weight + } + + return fimi.Name() < fimj.Name() + }) + + if f.fs.opts.Component == files.ComponentFolderContent { + // Finally filter out any duplicate content files, e.g. page.md and page.html. + n := 0 + seen := map[hstrings.Tuple]bool{} + for _, fi := range fis { + fim := fi.(FileMetaInfo) + pi := fim.Meta().PathInfo + keep := fim.IsDir() || !pi.IsContent() + + if !keep { + baseLang := hstrings.Tuple{First: pi.Base(), Second: fim.Meta().Lang} + if !seen[baseLang] { + keep = true + seen[baseLang] = true + } + } + + if keep { + fis[n] = fi + n++ + } + } + + fis = fis[:n] + } + + return fis, nil +} + +func (f *componentFsDir) Stat() (iofs.FileInfo, error) { + fi, err := f.DirOnlyOps.Stat() + if err != nil { + return nil, err + } + return f.fs.applyMeta(fi, f.name), nil +} + +func (fs *componentFs) Stat(name string) (os.FileInfo, error) { + fi, err := fs.Fs.Stat(name) + if err != nil { + return nil, err + } + return fs.applyMeta(fi, name), nil +} + +func (fs *componentFs) applyMeta(fi FileNameIsDir, name string) FileMetaInfo { + if runtime.GOOS == "darwin" { + name = norm.NFC.String(name) + } + fim := fi.(FileMetaInfo) + meta := fim.Meta() + meta.PathInfo = fs.opts.PathParser.Parse(fs.opts.Component, name) + if !fim.IsDir() { + if fileLang := meta.PathInfo.Lang(); fileLang != "" { + // A valid lang set in filename. + // Give priority to myfile.sv.txt inside the sv filesystem. + meta.Weight++ + meta.Lang = fileLang + } + } + + if meta.Lang == "" { + meta.Lang = fs.opts.DefaultContentLanguage + } + + langIdx, found := fs.opts.PathParser.LanguageIndex[meta.Lang] + if !found { + panic("no language found for " + meta.Lang) + } + meta.LangIndex = langIdx + + if fi.IsDir() { + meta.OpenFunc = func() (afero.File, error) { + return fs.Open(name) + } + } + + return fim +} + +func (f *componentFsDir) Readdir(count int) ([]os.FileInfo, error) { + panic("not supported: Use ReadDir") +} + +func (f *componentFsDir) Readdirnames(count int) ([]string, error) { + dirsi, err := f.DirOnlyOps.(iofs.ReadDirFile).ReadDir(count) + if err != nil { + return nil, err + } + + dirs := make([]string, len(dirsi)) + for i, d := range dirsi { + dirs[i] = d.Name() + } + return dirs, nil +} + +type ComponentFsOptions struct { + // The filesystem where one or more components are mounted. + Fs afero.Fs + + // The component name, e.g. "content", "layouts" etc. + Component string + + DefaultContentLanguage string + + // The parser used to parse paths provided by this filesystem. + PathParser paths.PathParser +} + +func (fs *componentFs) Open(name string) (afero.File, error) { + f, err := fs.Fs.Open(name) + if err != nil { + return nil, err + } + + fi, err := f.Stat() + if err != nil { + if err != errIsDir { + f.Close() + return nil, err + } + } else if !fi.IsDir() { + return f, nil + } + + return &componentFsDir{ + DirOnlyOps: f, + name: name, + fs: fs, + }, nil +} + +func (fs *componentFs) ReadDir(name string) ([]os.FileInfo, error) { + panic("not implemented") +} diff --git a/hugofs/decorators.go b/hugofs/decorators.go index 47b4266df..405c81ce4 100644 --- a/hugofs/decorators.go +++ b/hugofs/decorators.go @@ -15,63 +15,25 @@ package hugofs import ( "fmt" + "io/fs" "os" "path/filepath" - "strings" - "github.com/gohugoio/hugo/common/herrors" "github.com/spf13/afero" ) -var ( - _ FilesystemUnwrapper = (*baseFileDecoratorFs)(nil) -) +var _ FilesystemUnwrapper = (*baseFileDecoratorFs)(nil) func decorateDirs(fs afero.Fs, meta *FileMeta) afero.Fs { ffs := &baseFileDecoratorFs{Fs: fs} - decorator := func(fi os.FileInfo, name string) (os.FileInfo, error) { + decorator := func(fi FileNameIsDir, name string) (FileNameIsDir, error) { if !fi.IsDir() { // Leave regular files as they are. return fi, nil } - return decorateFileInfo(fi, fs, nil, "", "", meta), nil - } - - ffs.decorate = decorator - - return ffs -} - -func decoratePath(fs afero.Fs, createPath func(name string) string) afero.Fs { - ffs := &baseFileDecoratorFs{Fs: fs} - - decorator := func(fi os.FileInfo, name string) (os.FileInfo, error) { - path := createPath(name) - - return decorateFileInfo(fi, fs, nil, "", path, nil), nil - } - - ffs.decorate = decorator - - return ffs -} - -// DecorateBasePathFs adds Path info to files and directories in the -// provided BasePathFs, using the base as base. -func DecorateBasePathFs(base *afero.BasePathFs) afero.Fs { - basePath, _ := base.RealPath("") - if !strings.HasSuffix(basePath, filepathSeparator) { - basePath += filepathSeparator - } - - ffs := &baseFileDecoratorFs{Fs: base} - - decorator := func(fi os.FileInfo, name string) (os.FileInfo, error) { - path := strings.TrimPrefix(name, basePath) - - return decorateFileInfo(fi, base, nil, "", path, nil), nil + return decorateFileInfo(fi, nil, "", meta), nil } ffs.decorate = decorator @@ -84,7 +46,7 @@ func DecorateBasePathFs(base *afero.BasePathFs) afero.Fs { func NewBaseFileDecorator(fs afero.Fs, callbacks ...func(fi FileMetaInfo)) afero.Fs { ffs := &baseFileDecoratorFs{Fs: fs} - decorator := func(fi os.FileInfo, filename string) (os.FileInfo, error) { + decorator := func(fi FileNameIsDir, filename string) (FileNameIsDir, error) { // Store away the original in case it's a symlink. meta := NewFileMeta() meta.Name = fi.Name() @@ -92,38 +54,24 @@ func NewBaseFileDecorator(fs afero.Fs, callbacks ...func(fi FileMetaInfo)) afero if fi.IsDir() { meta.JoinStatFunc = func(name string) (FileMetaInfo, error) { joinedFilename := filepath.Join(filename, name) - fi, _, err := lstatIfPossible(fs, joinedFilename) + fi, err := fs.Stat(joinedFilename) if err != nil { return nil, err } - - fi, err = ffs.decorate(fi, joinedFilename) + fim, err := ffs.decorate(fi, joinedFilename) if err != nil { return nil, err } - return fi.(FileMetaInfo), nil - } - } - - isSymlink := isSymlink(fi) - if isSymlink { - meta.OriginalFilename = filename - var link string - var err error - link, fi, err = evalSymlinks(fs, filename) - if err != nil { - return nil, err + return fim.(FileMetaInfo), nil } - filename = link - meta.IsSymlink = true } opener := func() (afero.File, error) { return ffs.open(filename) } - fim := decorateFileInfo(fi, ffs, opener, filename, "", meta) + fim := decorateFileInfo(fi, opener, filename, meta) for _, cb := range callbacks { cb(fim) @@ -136,23 +84,9 @@ func NewBaseFileDecorator(fs afero.Fs, callbacks ...func(fi FileMetaInfo)) afero return ffs } -func evalSymlinks(fs afero.Fs, filename string) (string, os.FileInfo, error) { - link, err := filepath.EvalSymlinks(filename) - if err != nil { - return "", nil, err - } - - fi, err := fs.Stat(link) - if err != nil { - return "", nil, err - } - - return link, fi, nil -} - type baseFileDecoratorFs struct { afero.Fs - decorate func(fi os.FileInfo, filename string) (os.FileInfo, error) + decorate func(fi FileNameIsDir, name string) (FileNameIsDir, error) } func (fs *baseFileDecoratorFs) UnwrapFilesystem() afero.Fs { @@ -165,29 +99,11 @@ func (fs *baseFileDecoratorFs) Stat(name string) (os.FileInfo, error) { return nil, err } - return fs.decorate(fi, name) -} - -func (fs *baseFileDecoratorFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - var ( - fi os.FileInfo - err error - ok bool - ) - - if lstater, isLstater := fs.Fs.(afero.Lstater); isLstater { - fi, ok, err = lstater.LstatIfPossible(name) - } else { - fi, err = fs.Fs.Stat(name) - } - + fim, err := fs.decorate(fi, name) if err != nil { - return nil, false, err + return nil, err } - - fi, err = fs.decorate(fi, name) - - return fi, ok, err + return fim.(os.FileInfo), nil } func (fs *baseFileDecoratorFs) Open(name string) (afero.File, error) { @@ -207,35 +123,32 @@ type baseFileDecoratorFile struct { fs *baseFileDecoratorFs } -func (l *baseFileDecoratorFile) Readdir(c int) (ofi []os.FileInfo, err error) { - dirnames, err := l.File.Readdirnames(c) +func (l *baseFileDecoratorFile) ReadDir(n int) ([]fs.DirEntry, error) { + fis, err := l.File.(fs.ReadDirFile).ReadDir(-1) if err != nil { return nil, err } - fisp := make([]os.FileInfo, 0, len(dirnames)) + fisp := make([]fs.DirEntry, len(fis)) - for _, dirname := range dirnames { - filename := dirname - - if l.Name() != "" && l.Name() != filepathSeparator { - filename = filepath.Join(l.Name(), dirname) + for i, fi := range fis { + filename := fi.Name() + if l.Name() != "" { + filename = filepath.Join(l.Name(), fi.Name()) } - // We need to resolve any symlink info. - fi, _, err := lstatIfPossible(l.fs.Fs, filename) - if err != nil { - if herrors.IsNotExist(err) { - continue - } - return nil, err - } - fi, err = l.fs.decorate(fi, filename) + fid, err := l.fs.decorate(fi, filename) if err != nil { return nil, fmt.Errorf("decorate: %w", err) } - fisp = append(fisp, fi) + + fisp[i] = fid.(fs.DirEntry) + } return fisp, err } + +func (l *baseFileDecoratorFile) Readdir(c int) (ofi []os.FileInfo, err error) { + panic("not supported: Use ReadDir") +} diff --git a/hugofs/language_merge.go b/hugofs/dirsmerger.go index a2fa411a9..392353e27 100644 --- a/hugofs/language_merge.go +++ b/hugofs/dirsmerger.go @@ -1,4 +1,4 @@ -// Copyright 2022 The Hugo Authors. All rights reserved. +// Copyright 2024 The Hugo Authors. All rights reserved. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -14,12 +14,14 @@ package hugofs import ( - "os" + "io/fs" + + "github.com/bep/overlayfs" ) // LanguageDirsMerger implements the overlayfs.DirsMerger func, which is used // to merge two directories. -var LanguageDirsMerger = func(lofi, bofi []os.FileInfo) []os.FileInfo { +var LanguageDirsMerger overlayfs.DirsMerger = func(lofi, bofi []fs.DirEntry) []fs.DirEntry { for _, fi1 := range bofi { fim1 := fi1.(FileMetaInfo) var found bool @@ -37,3 +39,27 @@ var LanguageDirsMerger = func(lofi, bofi []os.FileInfo) []os.FileInfo { return lofi } + +// AppendDirsMerger merges two directories keeping all regular files +// with the first slice as the base. +// Duplicate directories in the secnond slice will be ignored. +// This strategy is used for the i18n and data fs where we need all entries. +var AppendDirsMerger overlayfs.DirsMerger = func(lofi, bofi []fs.DirEntry) []fs.DirEntry { + for _, fi1 := range bofi { + var found bool + // Remove duplicate directories. + if fi1.IsDir() { + for _, fi2 := range lofi { + if fi2.IsDir() && fi2.Name() == fi1.Name() { + found = true + break + } + } + } + if !found { + lofi = append(lofi, fi1) + } + } + + return lofi +} diff --git a/hugofs/fileinfo.go b/hugofs/fileinfo.go index 773352ea8..6d6122c0c 100644 --- a/hugofs/fileinfo.go +++ b/hugofs/fileinfo.go @@ -16,21 +16,25 @@ package hugofs import ( "errors" + "fmt" + "io" + "io/fs" "os" "path/filepath" "reflect" "runtime" "sort" - "strings" + "sync" "time" "github.com/gohugoio/hugo/hugofs/glob" - "github.com/gohugoio/hugo/hugofs/files" "golang.org/x/text/unicode/norm" + "github.com/gohugoio/hugo/common/herrors" "github.com/gohugoio/hugo/common/hreflect" "github.com/gohugoio/hugo/common/htime" + "github.com/gohugoio/hugo/common/paths" "github.com/spf13/afero" ) @@ -39,48 +43,37 @@ func NewFileMeta() *FileMeta { return &FileMeta{} } -// PathFile returns the relative file path for the file source. -func (f *FileMeta) PathFile() string { - if f.BaseDir == "" { - return "" - } - return strings.TrimPrefix(strings.TrimPrefix(f.Filename, f.BaseDir), filepathSeparator) -} - type FileMeta struct { - Name string - Filename string - Path string - PathWalk string - OriginalFilename string - BaseDir string - - SourceRoot string - MountRoot string - Module string - - Weight int - IsOrdered bool - IsSymlink bool - IsRootFile bool - IsProject bool - Watch bool - - Classifier files.ContentClass - - SkipDir bool - - Lang string - TranslationBaseName string - TranslationBaseNameWithExt string - Translations []string - - Fs afero.Fs + PathInfo *paths.Path + Name string + Filename string + + BaseDir string + SourceRoot string + Module string + ModuleOrdinal int + Component string + + Weight int + IsProject bool + Watch bool + + // The lang associated with this file. This may be + // either the language set in the filename or + // the language defined in the source mount configuration. + Lang string + // The language index for the above lang. This is the index + // in the sorted list of languages/sites. + LangIndex int + OpenFunc func() (afero.File, error) JoinStatFunc func(name string) (FileMetaInfo, error) // Include only files or directories that match. InclusionFilter *glob.FilenameFilter + + // Rename the name part of the file (not the directory). + Rename func(name string, toFrom bool) string } func (m *FileMeta) Copy() *FileMeta { @@ -120,6 +113,15 @@ func (f *FileMeta) Open() (afero.File, error) { return f.OpenFunc() } +func (f *FileMeta) ReadAll() ([]byte, error) { + file, err := f.Open() + if err != nil { + return nil, err + } + defer file.Close() + return io.ReadAll(file) +} + func (f *FileMeta) JoinStat(name string) (FileMetaInfo, error) { if f.JoinStatFunc == nil { return nil, os.ErrNotExist @@ -128,50 +130,123 @@ func (f *FileMeta) JoinStat(name string) (FileMetaInfo, error) { } type FileMetaInfo interface { - os.FileInfo - // Meta is for internal use. + fs.DirEntry + MetaProvider + + // This is a real hybrid as it also implements the fs.FileInfo interface. + FileInfoOptionals +} + +type MetaProvider interface { Meta() *FileMeta } -type fileInfoMeta struct { - os.FileInfo +type FileInfoOptionals interface { + Size() int64 + Mode() fs.FileMode + ModTime() time.Time + Sys() any +} - m *FileMeta +type FileNameIsDir interface { + Name() string + IsDir() bool } -type filenameProvider interface { - Filename() string +type FileInfoProvider interface { + FileInfo() FileMetaInfo } -var _ filenameProvider = (*fileInfoMeta)(nil) +// DirOnlyOps is a subset of the afero.File interface covering +// the methods needed for directory operations. +type DirOnlyOps interface { + io.Closer + Name() string + Readdir(count int) ([]os.FileInfo, error) + Readdirnames(n int) ([]string, error) + Stat() (os.FileInfo, error) +} + +type dirEntryMeta struct { + fs.DirEntry + m *FileMeta + + fi fs.FileInfo + fiInit sync.Once +} + +func (fi *dirEntryMeta) Meta() *FileMeta { + return fi.m +} // Filename returns the full filename. -func (fi *fileInfoMeta) Filename() string { +func (fi *dirEntryMeta) Filename() string { return fi.m.Filename } -// Name returns the file's name. Note that we follow symlinks, -// if supported by the file system, and the Name given here will be the -// name of the symlink, which is what Hugo needs in all situations. -func (fi *fileInfoMeta) Name() string { +func (fi *dirEntryMeta) fileInfo() fs.FileInfo { + var err error + fi.fiInit.Do(func() { + fi.fi, err = fi.DirEntry.Info() + }) + if err != nil { + panic(err) + } + return fi.fi +} + +func (fi *dirEntryMeta) Size() int64 { + return fi.fileInfo().Size() +} + +func (fi *dirEntryMeta) Mode() fs.FileMode { + return fi.fileInfo().Mode() +} + +func (fi *dirEntryMeta) ModTime() time.Time { + return fi.fileInfo().ModTime() +} + +func (fi *dirEntryMeta) Sys() any { + return fi.fileInfo().Sys() +} + +// Name returns the file's name. +func (fi *dirEntryMeta) Name() string { if name := fi.m.Name; name != "" { return name } - return fi.FileInfo.Name() + return fi.DirEntry.Name() } -func (fi *fileInfoMeta) Meta() *FileMeta { - return fi.m +// dirEntry is an adapter from os.FileInfo to fs.DirEntry +type dirEntry struct { + fs.FileInfo } -func NewFileMetaInfo(fi os.FileInfo, m *FileMeta) FileMetaInfo { +var _ fs.DirEntry = dirEntry{} + +func (d dirEntry) Type() fs.FileMode { return d.FileInfo.Mode().Type() } + +func (d dirEntry) Info() (fs.FileInfo, error) { return d.FileInfo, nil } + +func NewFileMetaInfo(fi FileNameIsDir, m *FileMeta) FileMetaInfo { if m == nil { panic("FileMeta must be set") } - if fim, ok := fi.(FileMetaInfo); ok { + if fim, ok := fi.(MetaProvider); ok { m.Merge(fim.Meta()) } - return &fileInfoMeta{FileInfo: fi, m: m} + switch v := fi.(type) { + case fs.DirEntry: + return &dirEntryMeta{DirEntry: v, m: m} + case fs.FileInfo: + return &dirEntryMeta{DirEntry: dirEntry{v}, m: m} + case nil: + return &dirEntryMeta{DirEntry: dirEntry{}, m: m} + default: + panic(fmt.Sprintf("Unsupported type: %T", fi)) + } } type dirNameOnlyFileInfo struct { @@ -212,7 +287,6 @@ func newDirNameOnlyFileInfo(name string, meta *FileMeta, fileOpener func() (afer m.Filename = name } m.OpenFunc = fileOpener - m.IsOrdered = false return NewFileMetaInfo( &dirNameOnlyFileInfo{name: base, modTime: htime.Now()}, @@ -220,16 +294,10 @@ func newDirNameOnlyFileInfo(name string, meta *FileMeta, fileOpener func() (afer ) } -func decorateFileInfo( - fi os.FileInfo, - fs afero.Fs, opener func() (afero.File, error), - filename, filepath string, inMeta *FileMeta, -) FileMetaInfo { +func decorateFileInfo(fi FileNameIsDir, opener func() (afero.File, error), filename string, inMeta *FileMeta) FileMetaInfo { var meta *FileMeta var fim FileMetaInfo - filepath = strings.TrimPrefix(filepath, filepathSeparator) - var ok bool if fim, ok = fi.(FileMetaInfo); ok { meta = fim.Meta() @@ -241,14 +309,8 @@ func decorateFileInfo( if opener != nil { meta.OpenFunc = opener } - if fs != nil { - meta.Fs = fs - } - nfilepath := normalizeFilename(filepath) + nfilename := normalizeFilename(filename) - if nfilepath != "" { - meta.Path = nfilepath - } if nfilename != "" { meta.Filename = nfilename } @@ -258,14 +320,11 @@ func decorateFileInfo( return fim } -func isSymlink(fi os.FileInfo) bool { - return fi != nil && fi.Mode()&os.ModeSymlink == os.ModeSymlink -} - -func fileInfosToFileMetaInfos(fis []os.FileInfo) []FileMetaInfo { +func DirEntriesToFileMetaInfos(fis []fs.DirEntry) []FileMetaInfo { fims := make([]FileMetaInfo, len(fis)) for i, v := range fis { - fims[i] = v.(FileMetaInfo) + fim := v.(FileMetaInfo) + fims[i] = fim } return fims } @@ -281,17 +340,49 @@ func normalizeFilename(filename string) string { return filename } -func fileInfosToNames(fis []os.FileInfo) []string { - names := make([]string, len(fis)) - for i, d := range fis { - names[i] = d.Name() - } - return names -} - -func sortFileInfos(fis []os.FileInfo) { +func sortDirEntries(fis []fs.DirEntry) { sort.Slice(fis, func(i, j int) bool { fimi, fimj := fis[i].(FileMetaInfo), fis[j].(FileMetaInfo) return fimi.Meta().Filename < fimj.Meta().Filename }) } + +// AddFileInfoToError adds file info to the given error. +func AddFileInfoToError(err error, fi FileMetaInfo, fs afero.Fs) error { + if err == nil { + return nil + } + + meta := fi.Meta() + filename := meta.Filename + + // Check if it's already added. + for _, ferr := range herrors.UnwrapFileErrors(err) { + pos := ferr.Position() + errfilename := pos.Filename + if errfilename == "" { + pos.Filename = filename + ferr.UpdatePosition(pos) + } + + if errfilename == "" || errfilename == filename { + if filename != "" && ferr.ErrorContext() == nil { + f, ioerr := fs.Open(filename) + if ioerr != nil { + return err + } + defer f.Close() + ferr.UpdateContent(f, nil) + } + return err + } + } + + lineMatcher := herrors.NopLineMatcher + + if textSegmentErr, ok := err.(*herrors.TextSegmentError); ok { + lineMatcher = herrors.ContainsMatcher(textSegmentErr.Segment) + } + + return herrors.NewFileErrorFromFile(err, filename, fs, lineMatcher) +} diff --git a/hugofs/fileinfo_test.go b/hugofs/fileinfo_test.go index 8d6a2ff7a..715798b34 100644 --- a/hugofs/fileinfo_test.go +++ b/hugofs/fileinfo_test.go @@ -25,7 +25,6 @@ func TestFileMeta(t *testing.T) { c.Run("Merge", func(c *qt.C) { src := &FileMeta{ Filename: "fs1", - Path: "ps1", } dst := &FileMeta{ Filename: "fd1", @@ -33,19 +32,16 @@ func TestFileMeta(t *testing.T) { dst.Merge(src) - c.Assert(dst.Path, qt.Equals, "ps1") c.Assert(dst.Filename, qt.Equals, "fd1") }) c.Run("Copy", func(c *qt.C) { src := &FileMeta{ Filename: "fs1", - Path: "ps1", } dst := src.Copy() c.Assert(dst, qt.Not(qt.Equals), src) c.Assert(dst, qt.DeepEquals, src) }) - } diff --git a/hugofs/filename_filter_fs.go b/hugofs/filename_filter_fs.go index c101309c2..5bae4b876 100644 --- a/hugofs/filename_filter_fs.go +++ b/hugofs/filename_filter_fs.go @@ -14,6 +14,7 @@ package hugofs import ( + "io/fs" "os" "strings" "syscall" @@ -45,17 +46,6 @@ func (fs *filenameFilterFs) UnwrapFilesystem() afero.Fs { return fs.fs } -func (fs *filenameFilterFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - fi, b, err := fs.fs.(afero.Lstater).LstatIfPossible(name) - if err != nil { - return nil, false, err - } - if !fs.filter.Match(name, fi.IsDir()) { - return nil, false, os.ErrNotExist - } - return fi, b, nil -} - func (fs *filenameFilterFs) Open(name string) (afero.File, error) { fi, err := fs.fs.Stat(name) if err != nil { @@ -87,8 +77,14 @@ func (fs *filenameFilterFs) OpenFile(name string, flag int, perm os.FileMode) (a } func (fs *filenameFilterFs) Stat(name string) (os.FileInfo, error) { - fi, _, err := fs.LstatIfPossible(name) - return fi, err + fi, err := fs.fs.Stat(name) + if err != nil { + return nil, err + } + if !fs.filter.Match(name, fi.IsDir()) { + return nil, os.ErrNotExist + } + return fi, nil } type filenameFilterDir struct { @@ -97,31 +93,35 @@ type filenameFilterDir struct { filter *glob.FilenameFilter } -func (f *filenameFilterDir) Readdir(count int) ([]os.FileInfo, error) { - fis, err := f.File.Readdir(-1) +func (f *filenameFilterDir) ReadDir(n int) ([]fs.DirEntry, error) { + des, err := f.File.(fs.ReadDirFile).ReadDir(n) if err != nil { return nil, err } - - var result []os.FileInfo - for _, fi := range fis { - fim := fi.(FileMetaInfo) - if f.filter.Match(strings.TrimPrefix(fim.Meta().Filename, f.base), fim.IsDir()) { - result = append(result, fi) + i := 0 + for _, de := range des { + fim := de.(FileMetaInfo) + rel := strings.TrimPrefix(fim.Meta().Filename, f.base) + if f.filter.Match(rel, de.IsDir()) { + des[i] = de + i++ } } + return des[:i], nil +} - return result, nil +func (f *filenameFilterDir) Readdir(count int) ([]os.FileInfo, error) { + panic("not supported: Use ReadDir") } func (f *filenameFilterDir) Readdirnames(count int) ([]string, error) { - dirsi, err := f.Readdir(count) + des, err := f.ReadDir(count) if err != nil { return nil, err } - dirs := make([]string, len(dirsi)) - for i, d := range dirsi { + dirs := make([]string, len(des)) + for i, d := range des { dirs[i] = d.Name() } return dirs, nil diff --git a/hugofs/filename_filter_fs_test.go b/hugofs/filename_filter_fs_test.go index b3e97a6a6..7b31f0f82 100644 --- a/hugofs/filename_filter_fs_test.go +++ b/hugofs/filename_filter_fs_test.go @@ -36,12 +36,12 @@ func TestFilenameFilterFs(t *testing.T) { for _, letter := range []string{"a", "b", "c"} { for i := 1; i <= 3; i++ { - c.Assert(afero.WriteFile(fs, filepath.Join(base, letter, fmt.Sprintf("my%d.txt", i)), []byte("some text file for"+letter), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join(base, letter, fmt.Sprintf("my%d.json", i)), []byte("some json file for"+letter), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(base, letter, fmt.Sprintf("my%d.txt", i)), []byte("some text file for"+letter), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(base, letter, fmt.Sprintf("my%d.json", i)), []byte("some json file for"+letter), 0o755), qt.IsNil) } } - fs = afero.NewBasePathFs(fs, base) + fs = NewBasePathFs(fs, base) filter, err := glob.NewFilenameFilter(nil, []string{"/b/**.txt"}) c.Assert(err, qt.IsNil) @@ -69,15 +69,16 @@ func TestFilenameFilterFs(t *testing.T) { assertExists("/b/my1.txt", false) dirB, err := fs.Open("/b") - defer dirB.Close() c.Assert(err, qt.IsNil) + defer dirB.Close() dirBEntries, err := dirB.Readdirnames(-1) + c.Assert(err, qt.IsNil) c.Assert(dirBEntries, qt.DeepEquals, []string{"my1.json", "my2.json", "my3.json"}) dirC, err := fs.Open("/c") - defer dirC.Close() c.Assert(err, qt.IsNil) + defer dirC.Close() dirCEntries, err := dirC.Readdirnames(-1) + c.Assert(err, qt.IsNil) c.Assert(dirCEntries, qt.DeepEquals, []string{"my1.json", "my1.txt", "my2.json", "my2.txt", "my3.json", "my3.txt"}) - } diff --git a/hugofs/files/classifier.go b/hugofs/files/classifier.go index bdac2d686..a8d231f73 100644 --- a/hugofs/files/classifier.go +++ b/hugofs/files/classifier.go @@ -14,16 +14,10 @@ package files import ( - "bufio" - "fmt" - "io" "os" "path/filepath" "sort" "strings" - "unicode" - - "github.com/spf13/afero" ) const ( @@ -80,99 +74,14 @@ func IsIndexContentFile(filename string) bool { return strings.HasPrefix(base, "index.") || strings.HasPrefix(base, "_index.") } -func IsHTMLFile(filename string) bool { - return htmlFileExtensionsSet[strings.TrimPrefix(filepath.Ext(filename), ".")] +func IsHTML(ext string) bool { + return htmlFileExtensionsSet[ext] } func IsContentExt(ext string) bool { return contentFileExtensionsSet[ext] } -type ContentClass string - -const ( - ContentClassLeaf ContentClass = "leaf" - ContentClassBranch ContentClass = "branch" - ContentClassFile ContentClass = "zfile" // Sort below - ContentClassContent ContentClass = "zcontent" -) - -func (c ContentClass) IsBundle() bool { - return c == ContentClassLeaf || c == ContentClassBranch -} - -func ClassifyContentFile(filename string, open func() (afero.File, error)) ContentClass { - if !IsContentFile(filename) { - return ContentClassFile - } - - if IsHTMLFile(filename) { - // We need to look inside the file. If the first non-whitespace - // character is a "<", then we treat it as a regular file. - // Eearlier we created pages for these files, but that had all sorts - // of troubles, and isn't what it says in the documentation. - // See https://github.com/gohugoio/hugo/issues/7030 - if open == nil { - panic(fmt.Sprintf("no file opener provided for %q", filename)) - } - - f, err := open() - if err != nil { - return ContentClassFile - } - ishtml := isHTMLContent(f) - f.Close() - if ishtml { - return ContentClassFile - } - - } - - if strings.HasPrefix(filename, "_index.") { - return ContentClassBranch - } - - if strings.HasPrefix(filename, "index.") { - return ContentClassLeaf - } - - return ContentClassContent -} - -var htmlComment = []rune{'<', '!', '-', '-'} - -func isHTMLContent(r io.Reader) bool { - br := bufio.NewReader(r) - i := 0 - for { - c, _, err := br.ReadRune() - if err != nil { - break - } - - if i > 0 { - if i >= len(htmlComment) { - return false - } - - if c != htmlComment[i] { - return true - } - - i++ - continue - } - - if !unicode.IsSpace(c) { - if i == 0 && c != '<' { - return false - } - i++ - } - } - return true -} - const ( ComponentFolderArchetypes = "archetypes" ComponentFolderStatic = "static" diff --git a/hugofs/files/classifier_test.go b/hugofs/files/classifier_test.go index 84036b870..f2fad56ca 100644 --- a/hugofs/files/classifier_test.go +++ b/hugofs/files/classifier_test.go @@ -15,7 +15,6 @@ package files import ( "path/filepath" - "strings" "testing" qt "github.com/frankban/quicktest" @@ -31,16 +30,6 @@ func TestIsContentFile(t *testing.T) { c.Assert(IsContentExt("json"), qt.Equals, false) } -func TestIsHTMLContent(t *testing.T) { - c := qt.New(t) - - c.Assert(isHTMLContent(strings.NewReader(" <html>")), qt.Equals, true) - c.Assert(isHTMLContent(strings.NewReader(" <!--\n---")), qt.Equals, false) - c.Assert(isHTMLContent(strings.NewReader(" <!--")), qt.Equals, true) - c.Assert(isHTMLContent(strings.NewReader(" ---<")), qt.Equals, false) - c.Assert(isHTMLContent(strings.NewReader(" foo <")), qt.Equals, false) -} - func TestComponentFolders(t *testing.T) { c := qt.New(t) diff --git a/hugofs/filter_fs.go b/hugofs/filter_fs.go deleted file mode 100644 index 1b020738a..000000000 --- a/hugofs/filter_fs.go +++ /dev/null @@ -1,344 +0,0 @@ -// Copyright 2019 The Hugo Authors. All rights reserved. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package hugofs - -import ( - "fmt" - "io" - "os" - "path/filepath" - "sort" - "strings" - "syscall" - "time" - - "github.com/gohugoio/hugo/hugofs/files" - - "github.com/spf13/afero" -) - -var ( - _ afero.Fs = (*FilterFs)(nil) - _ afero.Lstater = (*FilterFs)(nil) - _ afero.File = (*filterDir)(nil) -) - -func NewLanguageFs(langs map[string]int, fs afero.Fs) (afero.Fs, error) { - applyMeta := func(fs *FilterFs, name string, fis []os.FileInfo) { - for i, fi := range fis { - if fi.IsDir() { - filename := filepath.Join(name, fi.Name()) - fis[i] = decorateFileInfo(fi, fs, fs.getOpener(filename), "", "", nil) - continue - } - - meta := fi.(FileMetaInfo).Meta() - lang := meta.Lang - - fileLang, translationBaseName, translationBaseNameWithExt := langInfoFrom(langs, fi.Name()) - weight := meta.Weight - - if fileLang != "" { - if fileLang == lang { - // Give priority to myfile.sv.txt inside the sv filesystem. - weight++ - } - lang = fileLang - } - - fim := NewFileMetaInfo( - fi, - &FileMeta{ - Lang: lang, - Weight: weight, - TranslationBaseName: translationBaseName, - TranslationBaseNameWithExt: translationBaseNameWithExt, - Classifier: files.ClassifyContentFile(fi.Name(), meta.OpenFunc), - }) - - fis[i] = fim - } - } - - all := func(fis []os.FileInfo) { - // Maps translation base name to a list of language codes. - translations := make(map[string][]string) - trackTranslation := func(meta *FileMeta) { - name := meta.TranslationBaseNameWithExt - translations[name] = append(translations[name], meta.Lang) - } - for _, fi := range fis { - if fi.IsDir() { - continue - } - meta := fi.(FileMetaInfo).Meta() - - trackTranslation(meta) - - } - - for _, fi := range fis { - fim := fi.(FileMetaInfo) - langs := translations[fim.Meta().TranslationBaseNameWithExt] - if len(langs) > 0 { - fim.Meta().Translations = sortAndRemoveStringDuplicates(langs) - } - } - } - - return &FilterFs{ - fs: fs, - applyPerSource: applyMeta, - applyAll: all, - }, nil -} - -func NewFilterFs(fs afero.Fs) (afero.Fs, error) { - applyMeta := func(fs *FilterFs, name string, fis []os.FileInfo) { - for i, fi := range fis { - if fi.IsDir() { - fis[i] = decorateFileInfo(fi, fs, fs.getOpener(fi.(FileMetaInfo).Meta().Filename), "", "", nil) - } - } - } - - ffs := &FilterFs{ - fs: fs, - applyPerSource: applyMeta, - } - - return ffs, nil -} - -var ( - _ FilesystemUnwrapper = (*FilterFs)(nil) -) - -// FilterFs is an ordered composite filesystem. -type FilterFs struct { - fs afero.Fs - - applyPerSource func(fs *FilterFs, name string, fis []os.FileInfo) - applyAll func(fis []os.FileInfo) -} - -func (fs *FilterFs) Chmod(n string, m os.FileMode) error { - return syscall.EPERM -} - -func (fs *FilterFs) Chtimes(n string, a, m time.Time) error { - return syscall.EPERM -} - -func (fs *FilterFs) Chown(n string, uid, gid int) error { - return syscall.EPERM -} - -func (fs *FilterFs) UnwrapFilesystem() afero.Fs { - return fs.fs -} - -func (fs *FilterFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - fi, b, err := lstatIfPossible(fs.fs, name) - if err != nil { - return nil, false, err - } - - if fi.IsDir() { - return decorateFileInfo(fi, fs, fs.getOpener(name), "", "", nil), false, nil - } - - parent := filepath.Dir(name) - fs.applyFilters(parent, -1, fi) - - return fi, b, nil -} - -func (fs *FilterFs) Mkdir(n string, p os.FileMode) error { - return syscall.EPERM -} - -func (fs *FilterFs) MkdirAll(n string, p os.FileMode) error { - return syscall.EPERM -} - -func (fs *FilterFs) Name() string { - return "WeightedFileSystem" -} - -func (fs *FilterFs) Open(name string) (afero.File, error) { - f, err := fs.fs.Open(name) - if err != nil { - return nil, err - } - - return &filterDir{ - File: f, - ffs: fs, - }, nil -} - -func (fs *FilterFs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error) { - return fs.fs.Open(name) -} - -func (fs *FilterFs) ReadDir(name string) ([]os.FileInfo, error) { - panic("not implemented") -} - -func (fs *FilterFs) Remove(n string) error { - return syscall.EPERM -} - -func (fs *FilterFs) RemoveAll(p string) error { - return syscall.EPERM -} - -func (fs *FilterFs) Rename(o, n string) error { - return syscall.EPERM -} - -func (fs *FilterFs) Stat(name string) (os.FileInfo, error) { - fi, _, err := fs.LstatIfPossible(name) - return fi, err -} - -func (fs *FilterFs) Create(n string) (afero.File, error) { - return nil, syscall.EPERM -} - -func (fs *FilterFs) getOpener(name string) func() (afero.File, error) { - return func() (afero.File, error) { - return fs.Open(name) - } -} - -func (fs *FilterFs) applyFilters(name string, count int, fis ...os.FileInfo) ([]os.FileInfo, error) { - if fs.applyPerSource != nil { - fs.applyPerSource(fs, name, fis) - } - - seen := make(map[string]bool) - var duplicates []int - for i, dir := range fis { - if !dir.IsDir() { - continue - } - if seen[dir.Name()] { - duplicates = append(duplicates, i) - } else { - seen[dir.Name()] = true - } - } - - // Remove duplicate directories, keep first. - if len(duplicates) > 0 { - for i := len(duplicates) - 1; i >= 0; i-- { - idx := duplicates[i] - fis = append(fis[:idx], fis[idx+1:]...) - } - } - - if fs.applyAll != nil { - fs.applyAll(fis) - } - - if count > 0 && len(fis) >= count { - return fis[:count], nil - } - - return fis, nil -} - -type filterDir struct { - afero.File - ffs *FilterFs -} - -func (f *filterDir) Readdir(count int) ([]os.FileInfo, error) { - fis, err := f.File.Readdir(-1) - if err != nil { - return nil, err - } - return f.ffs.applyFilters(f.Name(), count, fis...) -} - -func (f *filterDir) Readdirnames(count int) ([]string, error) { - dirsi, err := f.Readdir(count) - if err != nil { - return nil, err - } - - dirs := make([]string, len(dirsi)) - for i, d := range dirsi { - dirs[i] = d.Name() - } - return dirs, nil -} - -// Try to extract the language from the given filename. -// Any valid language identifier in the name will win over the -// language set on the file system, e.g. "mypost.en.md". -func langInfoFrom(languages map[string]int, name string) (string, string, string) { - var lang string - - baseName := filepath.Base(name) - ext := filepath.Ext(baseName) - translationBaseName := baseName - - if ext != "" { - translationBaseName = strings.TrimSuffix(translationBaseName, ext) - } - - fileLangExt := filepath.Ext(translationBaseName) - fileLang := strings.TrimPrefix(fileLangExt, ".") - - if _, found := languages[fileLang]; found { - lang = fileLang - translationBaseName = strings.TrimSuffix(translationBaseName, fileLangExt) - } - - translationBaseNameWithExt := translationBaseName - - if ext != "" { - translationBaseNameWithExt += ext - } - - return lang, translationBaseName, translationBaseNameWithExt -} - -func printFs(fs afero.Fs, path string, w io.Writer) { - if fs == nil { - return - } - afero.Walk(fs, path, func(path string, info os.FileInfo, err error) error { - fmt.Println("p:::", path) - return nil - }) -} - -func sortAndRemoveStringDuplicates(s []string) []string { - ss := sort.StringSlice(s) - ss.Sort() - i := 0 - for j := 1; j < len(s); j++ { - if !ss.Less(i, j) { - continue - } - i++ - s[i] = s[j] - } - - return s[:i+1] -} diff --git a/hugofs/filter_fs_test.go b/hugofs/filter_fs_test.go deleted file mode 100644 index 524d957d6..000000000 --- a/hugofs/filter_fs_test.go +++ /dev/null @@ -1,46 +0,0 @@ -// Copyright 2019 The Hugo Authors. All rights reserved. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package hugofs - -import ( - "path/filepath" - "testing" - - qt "github.com/frankban/quicktest" -) - -func TestLangInfoFrom(t *testing.T) { - langs := map[string]int{ - "sv": 10, - "en": 20, - } - - c := qt.New(t) - - tests := []struct { - input string - expected []string - }{ - {"page.sv.md", []string{"sv", "page", "page.md"}}, - {"page.en.md", []string{"en", "page", "page.md"}}, - {"page.no.md", []string{"", "page.no", "page.no.md"}}, - {filepath.FromSlash("tc-lib-color/class-Com.Tecnick.Color.Css"), []string{"", "class-Com.Tecnick.Color", "class-Com.Tecnick.Color.Css"}}, - {filepath.FromSlash("class-Com.Tecnick.Color.sv.Css"), []string{"sv", "class-Com.Tecnick.Color", "class-Com.Tecnick.Color.Css"}}, - } - - for _, test := range tests { - v1, v2, v3 := langInfoFrom(langs, test.input) - c.Assert([]string{v1, v2, v3}, qt.DeepEquals, test.expected) - } -} diff --git a/hugofs/fs.go b/hugofs/fs.go index 5b8a3adb2..fc0ea71c6 100644 --- a/hugofs/fs.go +++ b/hugofs/fs.go @@ -111,7 +111,7 @@ func newFs(source, destination afero.Fs, workingDir, publishDir string) *Fs { // If this does not exist, it will be created later. absPublishDir := paths.AbsPathify(workingDir, publishDir) - pubFs := afero.NewBasePathFs(destination, absPublishDir) + pubFs := NewBasePathFs(destination, absPublishDir) return &Fs{ Source: source, @@ -126,16 +126,16 @@ func newFs(source, destination afero.Fs, workingDir, publishDir string) *Fs { func getWorkingDirFsReadOnly(base afero.Fs, workingDir string) afero.Fs { if workingDir == "" { - return afero.NewReadOnlyFs(base) + return NewReadOnlyFs(base) } - return afero.NewBasePathFs(afero.NewReadOnlyFs(base), workingDir) + return NewBasePathFs(NewReadOnlyFs(base), workingDir) } func getWorkingDirFsWritable(base afero.Fs, workingDir string) afero.Fs { if workingDir == "" { return base } - return afero.NewBasePathFs(base, workingDir) + return NewBasePathFs(base, workingDir) } func isWrite(flag int) bool { @@ -171,14 +171,11 @@ func MakeReadableAndRemoveAllModulePkgDir(fs afero.Fs, dir string) (int, error) func IsOsFs(fs afero.Fs) bool { var isOsFs bool WalkFilesystems(fs, func(fs afero.Fs) bool { - switch base := fs.(type) { + switch fs.(type) { case *afero.MemMapFs: isOsFs = false case *afero.OsFs: isOsFs = true - case *afero.BasePathFs: - _, supportsLstat, _ := base.LstatIfPossible("asdfasdfasdf") - isOsFs = supportsLstat } return isOsFs }) @@ -225,3 +222,30 @@ func WalkFilesystems(fs afero.Fs, fn WalkFn) bool { return false } + +var _ FilesystemUnwrapper = (*filesystemsWrapper)(nil) + +// NewBasePathFs creates a new BasePathFs. +func NewBasePathFs(source afero.Fs, path string) afero.Fs { + return WrapFilesystem(afero.NewBasePathFs(source, path), source) +} + +// NewReadOnlyFs creates a new ReadOnlyFs. +func NewReadOnlyFs(source afero.Fs) afero.Fs { + return WrapFilesystem(afero.NewReadOnlyFs(source), source) +} + +// WrapFilesystem is typically used to wrap a afero.BasePathFs to allow +// access to the underlying filesystem if needed. +func WrapFilesystem(container, content afero.Fs) afero.Fs { + return filesystemsWrapper{Fs: container, content: content} +} + +type filesystemsWrapper struct { + afero.Fs + content afero.Fs +} + +func (w filesystemsWrapper) UnwrapFilesystem() afero.Fs { + return w.content +} diff --git a/hugofs/fs_test.go b/hugofs/fs_test.go index b2ed2e86e..660ddd14c 100644 --- a/hugofs/fs_test.go +++ b/hugofs/fs_test.go @@ -28,8 +28,8 @@ func TestIsOsFs(t *testing.T) { c.Assert(IsOsFs(Os), qt.Equals, true) c.Assert(IsOsFs(&afero.MemMapFs{}), qt.Equals, false) - c.Assert(IsOsFs(afero.NewBasePathFs(&afero.MemMapFs{}, "/public")), qt.Equals, false) - c.Assert(IsOsFs(afero.NewBasePathFs(Os, t.TempDir())), qt.Equals, true) + c.Assert(IsOsFs(NewBasePathFs(&afero.MemMapFs{}, "/public")), qt.Equals, false) + c.Assert(IsOsFs(NewBasePathFs(Os, t.TempDir())), qt.Equals, true) } func TestNewDefault(t *testing.T) { @@ -43,9 +43,8 @@ func TestNewDefault(t *testing.T) { c.Assert(f.Source, hqt.IsSameType, new(afero.OsFs)) c.Assert(f.Os, qt.IsNotNil) c.Assert(f.WorkingDirReadOnly, qt.IsNotNil) - c.Assert(f.WorkingDirReadOnly, hqt.IsSameType, new(afero.BasePathFs)) - c.Assert(IsOsFs(f.Source), qt.IsTrue) c.Assert(IsOsFs(f.WorkingDirReadOnly), qt.IsTrue) + c.Assert(IsOsFs(f.Source), qt.IsTrue) c.Assert(IsOsFs(f.PublishDir), qt.IsTrue) c.Assert(IsOsFs(f.Os), qt.IsTrue) } diff --git a/hugofs/glob.go b/hugofs/glob.go index 1b649a283..6a6d999ce 100644 --- a/hugofs/glob.go +++ b/hugofs/glob.go @@ -31,6 +31,9 @@ func Glob(fs afero.Fs, pattern string, handle func(fi FileMetaInfo) (bool, error return nil } root := glob.ResolveRootDir(pattern) + if !strings.HasPrefix(root, "/") { + root = "/" + root + } pattern = strings.ToLower(pattern) g, err := glob.GetGlob(pattern) @@ -44,7 +47,7 @@ func Glob(fs afero.Fs, pattern string, handle func(fi FileMetaInfo) (bool, error // Signals that we're done. done := errors.New("done") - wfn := func(p string, info FileMetaInfo, err error) error { + wfn := func(p string, info FileMetaInfo) error { p = glob.NormalizePath(p) if info.IsDir() { if !hasSuperAsterisk { @@ -69,11 +72,13 @@ func Glob(fs afero.Fs, pattern string, handle func(fi FileMetaInfo) (bool, error return nil } - w := NewWalkway(WalkwayConfig{ - Root: root, - Fs: fs, - WalkFn: wfn, - }) + w := NewWalkway( + WalkwayConfig{ + Root: root, + Fs: fs, + WalkFn: wfn, + FailOnNotExist: true, + }) err = w.Walk() diff --git a/hugofs/glob/filename_filter.go b/hugofs/glob/filename_filter.go index 8e8af554b..6f283de48 100644 --- a/hugofs/glob/filename_filter.go +++ b/hugofs/glob/filename_filter.go @@ -27,6 +27,8 @@ type FilenameFilter struct { dirInclusions []glob.Glob exclusions []glob.Glob isWindows bool + + nested []*FilenameFilter } func normalizeFilenameGlobPattern(s string) string { @@ -101,11 +103,32 @@ func (f *FilenameFilter) Match(filename string, isDir bool) bool { if f == nil { return true } - return f.doMatch(filename, isDir) - /*if f.shouldInclude == nil { - fmt.Printf("Match: %q (%t) => %t\n", filename, isDir, isMatch) + if !f.doMatch(filename, isDir) { + return false + } + + for _, nested := range f.nested { + if !nested.Match(filename, isDir) { + return false + } + } + + return true +} + +// Append appends a filter to the chain. The receiver will be copied if needed. +func (f *FilenameFilter) Append(other *FilenameFilter) *FilenameFilter { + if f == nil { + return other } - return isMatch*/ + + clone := *f + nested := make([]*FilenameFilter, len(clone.nested)+1) + copy(nested, clone.nested) + nested[len(nested)-1] = other + clone.nested = nested + + return &clone } func (f *FilenameFilter) doMatch(filename string, isDir bool) bool { diff --git a/hugofs/glob/filename_filter_test.go b/hugofs/glob/filename_filter_test.go index 8437af858..6398e8a1e 100644 --- a/hugofs/glob/filename_filter_test.go +++ b/hugofs/glob/filename_filter_test.go @@ -36,6 +36,7 @@ func TestFilenameFilter(t *testing.T) { c.Assert(excludeAlmostAllJSON.Match("", true), qt.Equals, true) excludeAllButFooJSON, err := NewFilenameFilter([]string{"/a/**/foo.json"}, []string{"**.json"}) + c.Assert(err, qt.IsNil) c.Assert(excludeAllButFooJSON.Match(filepath.FromSlash("/data/my.json"), false), qt.Equals, false) c.Assert(excludeAllButFooJSON.Match(filepath.FromSlash("/a/b/c/d/e/foo.json"), false), qt.Equals, true) c.Assert(excludeAllButFooJSON.Match(filepath.FromSlash("/a/b/c"), true), qt.Equals, true) @@ -71,5 +72,4 @@ func TestFilenameFilter(t *testing.T) { funcFilter := NewFilenameFilterForInclusionFunc(func(s string) bool { return strings.HasSuffix(s, ".json") }) c.Assert(funcFilter.Match("ab.json", false), qt.Equals, true) c.Assert(funcFilter.Match("ab.bson", false), qt.Equals, false) - } diff --git a/hugofs/glob/glob.go b/hugofs/glob/glob.go index dc9b4fb5b..42aa1fa3b 100644 --- a/hugofs/glob/glob.go +++ b/hugofs/glob/glob.go @@ -69,7 +69,8 @@ func (gc *globCache) GetGlob(pattern string) (glob.Glob, error) { eg = globErr{ globDecorator{ g: g, - isWindows: gc.isWindows}, + isWindows: gc.isWindows, + }, err, } @@ -121,15 +122,6 @@ func (g globDecorator) Match(s string) bool { return g.g.Match(s) } -type globDecoratorDouble struct { - lowerCase glob.Glob - originalCase glob.Glob -} - -func (g globDecoratorDouble) Match(s string) bool { - return g.lowerCase.Match(s) || g.originalCase.Match(s) -} - func GetGlob(pattern string) (glob.Glob, error) { return defaultGlobCache.GetGlob(pattern) } diff --git a/hugofs/glob_test.go b/hugofs/glob_test.go index a6ae85fc8..722e0b441 100644 --- a/hugofs/glob_test.go +++ b/hugofs/glob_test.go @@ -14,6 +14,7 @@ package hugofs import ( + "os" "path/filepath" "testing" @@ -28,14 +29,21 @@ func TestGlob(t *testing.T) { fs := NewBaseFileDecorator(afero.NewMemMapFs()) create := func(filename string) { - err := afero.WriteFile(fs, filepath.FromSlash(filename), []byte("content "+filename), 0777) + filename = filepath.FromSlash(filename) + dir := filepath.Dir(filename) + if dir != "." { + err := fs.MkdirAll(dir, 0o777) + c.Assert(err, qt.IsNil) + } + err := afero.WriteFile(fs, filename, []byte("content "+filename), 0o777) c.Assert(err, qt.IsNil) } collect := func(pattern string) []string { var paths []string h := func(fi FileMetaInfo) (bool, error) { - paths = append(paths, fi.Meta().Path) + p := fi.Meta().PathInfo.Path() + paths = append(paths, p) return false, nil } err := Glob(fs, pattern, h) @@ -43,17 +51,22 @@ func TestGlob(t *testing.T) { return paths } - create("root.json") - create("jsonfiles/d1.json") - create("jsonfiles/d2.json") - create("jsonfiles/sub/d3.json") - create("jsonfiles/d1.xml") - create("a/b/c/e/f.json") - create("UPPER/sub/style.css") - create("root/UPPER/sub/style.css") + create("/root.json") + create("/jsonfiles/d1.json") + create("/jsonfiles/d2.json") + create("/jsonfiles/sub/d3.json") + create("/jsonfiles/d1.xml") + create("/a/b/c/e/f.json") + create("/UPPER/sub/style.css") + create("/root/UPPER/sub/style.css") - c.Assert(collect(filepath.FromSlash("/jsonfiles/*.json")), qt.HasLen, 2) + afero.Walk(fs, "/", func(path string, info os.FileInfo, err error) error { + c.Assert(err, qt.IsNil) + return nil + }) + c.Assert(collect(filepath.FromSlash("/jsonfiles/*.json")), qt.HasLen, 2) + c.Assert(collect("/*.json"), qt.HasLen, 1) c.Assert(collect("**.json"), qt.HasLen, 5) c.Assert(collect("**"), qt.HasLen, 8) c.Assert(collect(""), qt.HasLen, 0) @@ -63,5 +76,4 @@ func TestGlob(t *testing.T) { c.Assert(collect("root/UPPER/sub/style.css"), qt.HasLen, 1) c.Assert(collect("UPPER/sub/style.css"), qt.HasLen, 1) - } diff --git a/hugofs/hasbytes_fs.go b/hugofs/hasbytes_fs.go index 3d32a828f..238fbc9c4 100644 --- a/hugofs/hasbytes_fs.go +++ b/hugofs/hasbytes_fs.go @@ -1,4 +1,4 @@ -// Copyright 2022 The Hugo Authors. All rights reserved. +// Copyright 2024 The Hugo Authors. All rights reserved. // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. @@ -67,7 +67,6 @@ func (fs *hasBytesFs) wrapFile(f afero.File) afero.File { }, hasBytesCallback: fs.hasBytesCallback, } - } func (fs *hasBytesFs) Name() string { diff --git a/hugofs/noop_fs.go b/hugofs/noop_fs.go index 87f2cc9ff..e9def7c99 100644 --- a/hugofs/noop_fs.go +++ b/hugofs/noop_fs.go @@ -22,7 +22,7 @@ import ( ) var ( - errNoOp = errors.New("this is a filesystem that does nothing and this operation is not supported") + errNoOp = errors.New("this operation is not supported") _ afero.Fs = (*noOpFs)(nil) // NoOpFs provides a no-op filesystem that implements the afero.Fs @@ -30,8 +30,7 @@ var ( NoOpFs = &noOpFs{} ) -type noOpFs struct { -} +type noOpFs struct{} func (fs noOpFs) Create(name string) (afero.File, error) { panic(errNoOp) @@ -84,3 +83,47 @@ func (fs noOpFs) Chtimes(name string, atime time.Time, mtime time.Time) error { func (fs *noOpFs) Chown(name string, uid int, gid int) error { panic(errNoOp) } + +// noOpRegularFileOps implements the non-directory operations of a afero.File +// panicking for all operations. +type noOpRegularFileOps struct{} + +func (f *noOpRegularFileOps) Read(p []byte) (n int, err error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) ReadAt(p []byte, off int64) (n int, err error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Seek(offset int64, whence int) (int64, error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Write(p []byte) (n int, err error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) WriteAt(p []byte, off int64) (n int, err error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Readdir(count int) ([]os.FileInfo, error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Readdirnames(n int) ([]string, error) { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Sync() error { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) Truncate(size int64) error { + panic(errNoOp) +} + +func (f *noOpRegularFileOps) WriteString(s string) (ret int, err error) { + panic(errNoOp) +} diff --git a/hugofs/nosymlink_fs.go b/hugofs/nosymlink_fs.go deleted file mode 100644 index af559844f..000000000 --- a/hugofs/nosymlink_fs.go +++ /dev/null @@ -1,160 +0,0 @@ -// Copyright 2018 The Hugo Authors. All rights reserved. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package hugofs - -import ( - "errors" - "os" - "path/filepath" - - "github.com/gohugoio/hugo/common/loggers" - "github.com/spf13/afero" -) - -var ErrPermissionSymlink = errors.New("symlinks not allowed in this filesystem") - -// NewNoSymlinkFs creates a new filesystem that prevents symlinks. -func NewNoSymlinkFs(fs afero.Fs, logger loggers.Logger, allowFiles bool) afero.Fs { - return &noSymlinkFs{Fs: fs, logger: logger, allowFiles: allowFiles} -} - -var ( - _ FilesystemUnwrapper = (*noSymlinkFs)(nil) -) - -// noSymlinkFs is a filesystem that prevents symlinking. -type noSymlinkFs struct { - allowFiles bool // block dirs only - logger loggers.Logger - afero.Fs -} - -type noSymlinkFile struct { - fs *noSymlinkFs - afero.File -} - -func (f *noSymlinkFile) Readdir(count int) ([]os.FileInfo, error) { - fis, err := f.File.Readdir(count) - - filtered := fis[:0] - for _, x := range fis { - filename := filepath.Join(f.Name(), x.Name()) - if _, err := f.fs.checkSymlinkStatus(filename, x); err != nil { - // Log a warning and drop the file from the list - logUnsupportedSymlink(filename, f.fs.logger) - } else { - filtered = append(filtered, x) - } - } - - return filtered, err -} - -func (f *noSymlinkFile) Readdirnames(count int) ([]string, error) { - dirs, err := f.Readdir(count) - if err != nil { - return nil, err - } - return fileInfosToNames(dirs), nil -} - -func (fs *noSymlinkFs) UnwrapFilesystem() afero.Fs { - return fs.Fs -} - -func (fs *noSymlinkFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - return fs.stat(name) -} - -func (fs *noSymlinkFs) Stat(name string) (os.FileInfo, error) { - fi, _, err := fs.stat(name) - return fi, err -} - -func (fs *noSymlinkFs) stat(name string) (os.FileInfo, bool, error) { - var ( - fi os.FileInfo - wasLstat bool - err error - ) - - if lstater, ok := fs.Fs.(afero.Lstater); ok { - fi, wasLstat, err = lstater.LstatIfPossible(name) - } else { - fi, err = fs.Fs.Stat(name) - } - - if err != nil { - return nil, false, err - } - - fi, err = fs.checkSymlinkStatus(name, fi) - - return fi, wasLstat, err -} - -func (fs *noSymlinkFs) checkSymlinkStatus(name string, fi os.FileInfo) (os.FileInfo, error) { - var metaIsSymlink bool - - if fim, ok := fi.(FileMetaInfo); ok { - meta := fim.Meta() - metaIsSymlink = meta.IsSymlink - } - - if metaIsSymlink { - if fs.allowFiles && !fi.IsDir() { - return fi, nil - } - return nil, ErrPermissionSymlink - } - - // Also support non-decorated filesystems, e.g. the Os fs. - if isSymlink(fi) { - // Need to determine if this is a directory or not. - _, sfi, err := evalSymlinks(fs.Fs, name) - if err != nil { - return nil, err - } - if fs.allowFiles && !sfi.IsDir() { - // Return the original FileInfo to get the expected Name. - return fi, nil - } - return nil, ErrPermissionSymlink - } - - return fi, nil -} - -func (fs *noSymlinkFs) Open(name string) (afero.File, error) { - if _, _, err := fs.stat(name); err != nil { - return nil, err - } - return fs.wrapFile(fs.Fs.Open(name)) -} - -func (fs *noSymlinkFs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error) { - if _, _, err := fs.stat(name); err != nil { - return nil, err - } - return fs.wrapFile(fs.Fs.OpenFile(name, flag, perm)) -} - -func (fs *noSymlinkFs) wrapFile(f afero.File, err error) (afero.File, error) { - if err != nil { - return nil, err - } - - return &noSymlinkFile{File: f, fs: fs}, nil -} diff --git a/hugofs/nosymlink_test.go b/hugofs/nosymlink_test.go deleted file mode 100644 index d0a8baaaa..000000000 --- a/hugofs/nosymlink_test.go +++ /dev/null @@ -1,146 +0,0 @@ -// Copyright 2019 The Hugo Authors. All rights reserved. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package hugofs - -import ( - "os" - "path/filepath" - "testing" - - "github.com/bep/logg" - "github.com/gohugoio/hugo/common/loggers" - "github.com/gohugoio/hugo/htesting" - - "github.com/spf13/afero" - - qt "github.com/frankban/quicktest" -) - -func prepareSymlinks(t *testing.T) (string, func()) { - c := qt.New(t) - - workDir, clean, err := htesting.CreateTempDir(Os, "hugo-symlink-test") - c.Assert(err, qt.IsNil) - wd, _ := os.Getwd() - - blogDir := filepath.Join(workDir, "blog") - blogSubDir := filepath.Join(blogDir, "sub") - c.Assert(os.MkdirAll(blogSubDir, 0777), qt.IsNil) - blogFile1 := filepath.Join(blogDir, "a.txt") - blogFile2 := filepath.Join(blogSubDir, "b.txt") - afero.WriteFile(Os, filepath.Join(blogFile1), []byte("content1"), 0777) - afero.WriteFile(Os, filepath.Join(blogFile2), []byte("content2"), 0777) - os.Chdir(workDir) - c.Assert(os.Symlink("blog", "symlinkdedir"), qt.IsNil) - os.Chdir(blogDir) - c.Assert(os.Symlink("sub", "symsub"), qt.IsNil) - c.Assert(os.Symlink("a.txt", "symlinkdedfile.txt"), qt.IsNil) - - return workDir, func() { - clean() - os.Chdir(wd) - } -} - -func TestNoSymlinkFs(t *testing.T) { - if skipSymlink() { - t.Skip("Skip; os.Symlink needs administrator rights on Windows") - } - c := qt.New(t) - workDir, clean := prepareSymlinks(t) - defer clean() - - blogDir := filepath.Join(workDir, "blog") - blogFile1 := filepath.Join(blogDir, "a.txt") - - logger := loggers.NewDefault() - - for _, bfs := range []afero.Fs{NewBaseFileDecorator(Os), Os} { - for _, allowFiles := range []bool{false, true} { - logger.Reset() - fs := NewNoSymlinkFs(bfs, logger, allowFiles) - ls := fs.(afero.Lstater) - symlinkedDir := filepath.Join(workDir, "symlinkdedir") - symlinkedFilename := "symlinkdedfile.txt" - symlinkedFile := filepath.Join(blogDir, symlinkedFilename) - - assertFileErr := func(err error) { - if allowFiles { - c.Assert(err, qt.IsNil) - } else { - c.Assert(err, qt.Equals, ErrPermissionSymlink) - } - } - - assertFileStat := func(name string, fi os.FileInfo, err error) { - t.Helper() - assertFileErr(err) - if err == nil { - c.Assert(fi, qt.Not(qt.IsNil)) - c.Assert(fi.Name(), qt.Equals, name) - } - } - - // Check Stat and Lstat - for _, stat := range []func(name string) (os.FileInfo, error){ - func(name string) (os.FileInfo, error) { - return fs.Stat(name) - }, - func(name string) (os.FileInfo, error) { - fi, _, err := ls.LstatIfPossible(name) - return fi, err - }, - } { - _, err := stat(symlinkedDir) - c.Assert(err, qt.Equals, ErrPermissionSymlink) - fi, err := stat(symlinkedFile) - assertFileStat(symlinkedFilename, fi, err) - - fi, err = stat(filepath.Join(workDir, "blog")) - c.Assert(err, qt.IsNil) - c.Assert(fi, qt.Not(qt.IsNil)) - - fi, err = stat(blogFile1) - c.Assert(err, qt.IsNil) - c.Assert(fi, qt.Not(qt.IsNil)) - } - - // Check Open - _, err := fs.Open(symlinkedDir) - c.Assert(err, qt.Equals, ErrPermissionSymlink) - _, err = fs.OpenFile(symlinkedDir, os.O_RDWR|os.O_APPEND|os.O_CREATE, 0666) - c.Assert(err, qt.Equals, ErrPermissionSymlink) - _, err = fs.OpenFile(symlinkedFile, os.O_RDWR|os.O_APPEND|os.O_CREATE, 0666) - assertFileErr(err) - _, err = fs.Open(symlinkedFile) - assertFileErr(err) - f, err := fs.Open(blogDir) - c.Assert(err, qt.IsNil) - f.Close() - f, err = fs.Open(blogFile1) - c.Assert(err, qt.IsNil) - f.Close() - - // Check readdir - f, err = fs.Open(workDir) - c.Assert(err, qt.IsNil) - // There is at least one unsupported symlink inside workDir - _, err = f.Readdir(-1) - c.Assert(err, qt.IsNil) - f.Close() - c.Assert(logger.LoggCount(logg.LevelWarn), qt.Equals, 1) - - } - } -} diff --git a/hugofs/openfiles_fs.go b/hugofs/openfiles_fs.go new file mode 100644 index 000000000..f363c95f6 --- /dev/null +++ b/hugofs/openfiles_fs.go @@ -0,0 +1,110 @@ +// Copyright 2024 The Hugo Authors. All rights reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package hugofs + +import ( + "io/fs" + "os" + "sync" + + "github.com/spf13/afero" +) + +var _ FilesystemUnwrapper = (*OpenFilesFs)(nil) + +// OpenFilesFs is a wrapper around afero.Fs that keeps track of open files. +type OpenFilesFs struct { + afero.Fs + + mu sync.Mutex + openFiles map[string]int +} + +func (fs *OpenFilesFs) UnwrapFilesystem() afero.Fs { + return fs.Fs +} + +func (fs *OpenFilesFs) Create(name string) (afero.File, error) { + f, err := fs.Fs.Create(name) + if err != nil { + return nil, err + } + return fs.trackAndWrapFile(f), nil +} + +func (fs *OpenFilesFs) Open(name string) (afero.File, error) { + f, err := fs.Fs.Open(name) + if err != nil { + return nil, err + } + return fs.trackAndWrapFile(f), nil +} + +func (fs *OpenFilesFs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error) { + f, err := fs.Fs.OpenFile(name, flag, perm) + if err != nil { + return nil, err + } + return fs.trackAndWrapFile(f), nil +} + +func (fs *OpenFilesFs) trackAndWrapFile(f afero.File) afero.File { + fs.mu.Lock() + defer fs.mu.Unlock() + + if fs.openFiles == nil { + fs.openFiles = make(map[string]int) + } + + fs.openFiles[f.Name()]++ + + return &openFilesFsFile{fs: fs, File: f} +} + +type openFilesFsFile struct { + fs *OpenFilesFs + afero.File +} + +func (f *openFilesFsFile) ReadDir(count int) ([]fs.DirEntry, error) { + return f.File.(fs.ReadDirFile).ReadDir(count) +} + +func (f *openFilesFsFile) Close() (err error) { + f.fs.mu.Lock() + defer f.fs.mu.Unlock() + + err = f.File.Close() + + if f.fs.openFiles == nil { + return + } + + name := f.Name() + + f.fs.openFiles[name]-- + + if f.fs.openFiles[name] <= 0 { + delete(f.fs.openFiles, name) + } + + return +} + +func (fs *OpenFilesFs) OpenFiles() map[string]int { + fs.mu.Lock() + defer fs.mu.Unlock() + + return fs.openFiles +} diff --git a/hugofs/rootmapping_fs.go b/hugofs/rootmapping_fs.go index a37e21a8b..1efb8ee5f 100644 --- a/hugofs/rootmapping_fs.go +++ b/hugofs/rootmapping_fs.go @@ -14,13 +14,20 @@ package hugofs import ( + "errors" "fmt" + iofs "io/fs" "os" + "path" "path/filepath" "strings" "github.com/gohugoio/hugo/common/herrors" + "github.com/gohugoio/hugo/common/paths" + + "github.com/bep/overlayfs" "github.com/gohugoio/hugo/hugofs/files" + "github.com/gohugoio/hugo/hugofs/glob" radix "github.com/armon/go-radix" "github.com/spf13/afero" @@ -28,17 +35,31 @@ import ( var filepathSeparator = string(filepath.Separator) +var _ ReverseLookupProvder = (*RootMappingFs)(nil) + // NewRootMappingFs creates a new RootMappingFs on top of the provided with // root mappings with some optional metadata about the root. // Note that From represents a virtual root that maps to the actual filename in To. func NewRootMappingFs(fs afero.Fs, rms ...RootMapping) (*RootMappingFs, error) { rootMapToReal := radix.New() + realMapToRoot := radix.New() var virtualRoots []RootMapping + addMapping := func(key string, rm RootMapping, to *radix.Tree) { + var mappings []RootMapping + v, found := to.Get(key) + if found { + // There may be more than one language pointing to the same root. + mappings = v.([]RootMapping) + } + mappings = append(mappings, rm) + to.Insert(key, mappings) + } + for _, rm := range rms { (&rm).clean() - fromBase := files.ResolveComponentFolder(rm.From) + rm.FromBase = files.ResolveComponentFolder(rm.From) if len(rm.To) < 2 { panic(fmt.Sprintf("invalid root mapping; from/to: %s/%s", rm.From, rm.To)) @@ -46,21 +67,80 @@ func NewRootMappingFs(fs afero.Fs, rms ...RootMapping) (*RootMappingFs, error) { fi, err := fs.Stat(rm.To) if err != nil { - if herrors.IsNotExist(err) { + if os.IsNotExist(err) { continue } return nil, err } - // Extract "blog" from "content/blog" - rm.path = strings.TrimPrefix(strings.TrimPrefix(rm.From, fromBase), filepathSeparator) + if rm.Meta == nil { rm.Meta = NewFileMeta() } - rm.Meta.SourceRoot = rm.To - rm.Meta.BaseDir = rm.ToBasedir - rm.Meta.MountRoot = rm.path + if !fi.IsDir() { + // We do allow single file mounts. + // However, the file system logic will be much simpler with just directories. + // So, convert this mount into a directory mount with a nameTo filter and renamer. + dirFrom, nameFrom := filepath.Split(rm.From) + dirTo, nameTo := filepath.Split(rm.To) + dirFrom, dirTo = strings.TrimSuffix(dirFrom, filepathSeparator), strings.TrimSuffix(dirTo, filepathSeparator) + rm.From = dirFrom + + fi, err = fs.Stat(rm.To) + if err != nil { + if herrors.IsNotExist(err) { + continue + } + return nil, err + } + + rm.fiSingleFile = NewFileMetaInfo(fi, rm.Meta.Copy()) + rm.To = dirTo + + rm.Meta.Rename = func(name string, toFrom bool) string { + if toFrom { + if name == nameTo { + return nameFrom + } + return name + } + + if name == nameFrom { + return nameTo + } + + return name + } + nameToFilename := filepathSeparator + nameTo + + rm.Meta.InclusionFilter = rm.Meta.InclusionFilter.Append(glob.NewFilenameFilterForInclusionFunc( + func(filename string) bool { + return strings.HasPrefix(nameToFilename, filename) + }, + )) + + // Refresh the FileInfo object. + fi, err = fs.Stat(rm.To) + if err != nil { + if herrors.IsNotExist(err) { + continue + } + return nil, err + } + } + + if rm.FromBase == "" { + panic(" rm.FromBase is empty") + } + + // Extract "blog" from "content/blog" + rm.path = strings.TrimPrefix(strings.TrimPrefix(rm.From, rm.FromBase), filepathSeparator) + + rm.Meta.SourceRoot = fi.(MetaProvider).Meta().Filename + rm.Meta.BaseDir = rm.ToBase rm.Meta.Module = rm.Module + rm.Meta.ModuleOrdinal = rm.ModuleOrdinal + rm.Meta.Component = rm.FromBase rm.Meta.IsProject = rm.IsProject meta := rm.Meta.Copy() @@ -72,15 +152,13 @@ func NewRootMappingFs(fs afero.Fs, rms ...RootMapping) (*RootMappingFs, error) { rm.fi = NewFileMetaInfo(fi, meta) - key := filepathSeparator + rm.From - var mappings []RootMapping - v, found := rootMapToReal.Get(key) - if found { - // There may be more than one language pointing to the same root. - mappings = v.([]RootMapping) + addMapping(filepathSeparator+rm.From, rm, rootMapToReal) + rev := rm.To + if !strings.HasPrefix(rev, filepathSeparator) { + rev = filepathSeparator + rev } - mappings = append(mappings, rm) - rootMapToReal.Insert(key, mappings) + + addMapping(rev, rm, realMapToRoot) virtualRoots = append(virtualRoots, rm) } @@ -90,6 +168,7 @@ func NewRootMappingFs(fs afero.Fs, rms ...RootMapping) (*RootMappingFs, error) { rfs := &RootMappingFs{ Fs: fs, rootMapToReal: rootMapToReal, + realMapToRoot: realMapToRoot, } return rfs, nil @@ -103,9 +182,9 @@ func newRootMappingFsFromFromTo( rms := make([]RootMapping, len(fromTo)/2) for i, j := 0, 0; j < len(fromTo); i, j = i+1, j+2 { rms[i] = RootMapping{ - From: fromTo[j], - To: fromTo[j+1], - ToBasedir: baseDir, + From: fromTo[j], + To: fromTo[j+1], + ToBase: baseDir, } } @@ -114,16 +193,18 @@ func newRootMappingFsFromFromTo( // RootMapping describes a virtual file or directory mount. type RootMapping struct { - From string // The virtual mount. - To string // The source directory or file. - ToBasedir string // The base of To. May be empty if an absolute path was provided. - Module string // The module path/ID. - IsProject bool // Whether this is a mount in the main project. - Meta *FileMeta // File metadata (lang etc.) - - fi FileMetaInfo - path string // The virtual mount point, e.g. "blog". + From string // The virtual mount. + FromBase string // The base directory of the virtual mount. + To string // The source directory or file. + ToBase string // The base of To. May be empty if an absolute path was provided. + Module string // The module path/ID. + ModuleOrdinal int // The module ordinal starting with 0 which is the project. + IsProject bool // Whether this is a mount in the main project. + Meta *FileMeta // File metadata (lang etc.) + fi FileMetaInfo + fiSingleFile FileMetaInfo // Also set when this mounts represents a single file with a rename func. + path string // The virtual mount point, e.g. "blog". } type keyRootMappings struct { @@ -150,9 +231,7 @@ func (r RootMapping) trimFrom(name string) string { return strings.TrimPrefix(name, r.From) } -var ( - _ FilesystemUnwrapper = (*RootMappingFs)(nil) -) +var _ FilesystemUnwrapper = (*RootMappingFs)(nil) // A RootMappingFs maps several roots into one. Note that the root of this filesystem // is directories only, and they will be returned in Readdir and Readdirnames @@ -160,9 +239,10 @@ var ( type RootMappingFs struct { afero.Fs rootMapToReal *radix.Tree + realMapToRoot *radix.Tree } -func (fs *RootMappingFs) Dirs(base string) ([]FileMetaInfo, error) { +func (fs *RootMappingFs) Mounts(base string) ([]FileMetaInfo, error) { base = filepathSeparator + fs.cleanName(base) roots := fs.getRootsWithPrefix(base) @@ -172,17 +252,14 @@ func (fs *RootMappingFs) Dirs(base string) ([]FileMetaInfo, error) { fss := make([]FileMetaInfo, len(roots)) for i, r := range roots { - bfs := afero.NewBasePathFs(fs.Fs, r.To) - bfs = decoratePath(bfs, func(name string) string { - p := strings.TrimPrefix(name, r.To) - if r.path != "" { - // Make sure it's mounted to a any sub path, e.g. blog - p = filepath.Join(r.path, p) - } - p = strings.TrimLeft(p, filepathSeparator) - return p - }) + if r.fiSingleFile != nil { + // A single file mount. + fss[i] = r.fiSingleFile + continue + } + + bfs := NewBasePathFs(fs.Fs, r.To) fs := bfs if r.Meta.InclusionFilter != nil { fs = newFilenameFilterFs(fs, r.To, r.Meta.InclusionFilter) @@ -229,18 +306,9 @@ func (fs RootMappingFs) Filter(f func(m RootMapping) bool) *RootMappingFs { return &fs } -// LstatIfPossible returns the os.FileInfo structure describing a given file. -func (fs *RootMappingFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - fis, err := fs.doLstat(name) - if err != nil { - return nil, false, err - } - return fis[0], false, nil -} - // Open opens the named file for reading. func (fs *RootMappingFs) Open(name string) (afero.File, error) { - fis, err := fs.doLstat(name) + fis, err := fs.doStat(name) if err != nil { return nil, err } @@ -251,8 +319,68 @@ func (fs *RootMappingFs) Open(name string) (afero.File, error) { // Stat returns the os.FileInfo structure describing a given file. If there is // an error, it will be of type *os.PathError. func (fs *RootMappingFs) Stat(name string) (os.FileInfo, error) { - fi, _, err := fs.LstatIfPossible(name) - return fi, err + fis, err := fs.doStat(name) + if err != nil { + return nil, err + } + + return fis[0], nil +} + +type ComponentPath struct { + Component string + Path string + Lang string +} + +func (c ComponentPath) ComponentPathJoined() string { + return path.Join(c.Component, c.Path) +} + +type ReverseLookupProvder interface { + ReverseLookup(filename string, checkExists bool) ([]ComponentPath, error) +} + +// func (fs *RootMappingFs) ReverseStat(filename string) ([]FileMetaInfo, error) +func (fs *RootMappingFs) ReverseLookup(in string, checkExists bool) ([]ComponentPath, error) { + in = fs.cleanName(in) + key := filepathSeparator + in + + s, roots := fs.getRootsReverse(key) + + if len(roots) == 0 { + return nil, nil + } + + var cps []ComponentPath + + base := strings.TrimPrefix(key, s) + dir, name := filepath.Split(base) + + for _, first := range roots { + if first.Meta.Rename != nil { + name = first.Meta.Rename(name, true) + } + + // Now we know that this file _could_ be in this fs. + filename := filepathSeparator + filepath.Join(first.path, dir, name) + + if checkExists { + // Confirm that it exists. + _, err := fs.Stat(first.FromBase + filename) + if err != nil { + continue + } + } + + cps = append(cps, ComponentPath{ + Component: first.FromBase, + Path: paths.ToSlashTrimLeading(filename), + Lang: first.Meta.Lang, + }) + } + + return cps, nil } func (fs *RootMappingFs) hasPrefix(prefix string) bool { @@ -275,21 +403,22 @@ func (fs *RootMappingFs) getRoot(key string) []RootMapping { } func (fs *RootMappingFs) getRoots(key string) (string, []RootMapping) { - s, v, found := fs.rootMapToReal.LongestPrefix(key) - if !found || (s == filepathSeparator && key != filepathSeparator) { + return fs.getRootsIn(key, fs.rootMapToReal) +} + +func (fs *RootMappingFs) getRootsReverse(key string) (string, []RootMapping) { + return fs.getRootsIn(key, fs.realMapToRoot) +} + +func (fs *RootMappingFs) getRootsIn(key string, tree *radix.Tree) (string, []RootMapping) { + s, v, found := tree.LongestPrefix(key) + + if !found { return "", nil } return s, v.([]RootMapping) } -func (fs *RootMappingFs) debug() { - fmt.Println("debug():") - fs.rootMapToReal.Walk(func(s string, v any) bool { - fmt.Println("Key", s) - return false - }) -} - func (fs *RootMappingFs) getRootsWithPrefix(prefix string) []RootMapping { var roots []RootMapping fs.rootMapToReal.WalkPrefix(prefix, func(b string, v any) bool { @@ -316,61 +445,63 @@ func (fs *RootMappingFs) getAncestors(prefix string) []keyRootMappings { } func (fs *RootMappingFs) newUnionFile(fis ...FileMetaInfo) (afero.File, error) { - meta := fis[0].Meta() - f, err := meta.Open() - if err != nil { - return nil, err - } if len(fis) == 1 { - return f, nil + return fis[0].Meta().Open() } - rf := &rootMappingFile{File: f, fs: fs, name: meta.Name, meta: meta} - if len(fis) == 1 { - return rf, err - } - - next, err := fs.newUnionFile(fis[1:]...) - if err != nil { - return nil, err + openers := make([]func() (afero.File, error), len(fis)) + for i := len(fis) - 1; i >= 0; i-- { + fi := fis[i] + openers[i] = func() (afero.File, error) { + meta := fi.Meta() + f, err := meta.Open() + if err != nil { + return nil, err + } + return &rootMappingDir{DirOnlyOps: f, fs: fs, name: meta.Name, meta: meta}, nil + } } - uf := &afero.UnionFile{Base: rf, Layer: next} - - uf.Merger = func(lofi, bofi []os.FileInfo) ([]os.FileInfo, error) { + merge := func(lofi, bofi []iofs.DirEntry) []iofs.DirEntry { // Ignore duplicate directory entries - seen := make(map[string]bool) - var result []os.FileInfo - - for _, fis := range [][]os.FileInfo{bofi, lofi} { - for _, fi := range fis { - - if fi.IsDir() && seen[fi.Name()] { + for _, fi1 := range bofi { + var found bool + for _, fi2 := range lofi { + if !fi2.IsDir() { continue } - - if fi.IsDir() { - seen[fi.Name()] = true + if fi1.Name() == fi2.Name() { + found = true + break } - - result = append(result, fi) + } + if !found { + lofi = append(lofi, fi1) } } - return result, nil + return lofi } - return uf, nil + info := func() (os.FileInfo, error) { + return fis[0], nil + } + + return overlayfs.OpenDir(merge, info, openers...) } func (fs *RootMappingFs) cleanName(name string) string { - return strings.Trim(filepath.Clean(name), filepathSeparator) + name = strings.Trim(filepath.Clean(name), filepathSeparator) + if name == "." { + name = "" + } + return name } -func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) { - prefix = filepathSeparator + fs.cleanName(prefix) +func (rfs *RootMappingFs) collectDirEntries(prefix string) ([]iofs.DirEntry, error) { + prefix = filepathSeparator + rfs.cleanName(prefix) - var fis []os.FileInfo + var fis []iofs.DirEntry seen := make(map[string]bool) // Prevent duplicate directories level := strings.Count(prefix, filepathSeparator) @@ -380,15 +511,17 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) if err != nil { return err } - direntries, err := f.Readdir(-1) + direntries, err := f.(iofs.ReadDirFile).ReadDir(-1) if err != nil { f.Close() return err } for _, fi := range direntries { + meta := fi.(FileMetaInfo).Meta() meta.Merge(rm.Meta) + if !rm.Meta.InclusionFilter.Match(strings.TrimPrefix(meta.Filename, meta.SourceRoot), fi.IsDir()) { continue } @@ -400,11 +533,14 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) } seen[name] = true opener := func() (afero.File, error) { - return fs.Open(filepath.Join(rm.From, name)) + return rfs.Open(filepath.Join(rm.From, name)) } fi = newDirNameOnlyFileInfo(name, meta, opener) + } else if rm.Meta.Rename != nil { + if n := rm.Meta.Rename(fi.Name(), true); n != fi.Name() { + fi.(MetaProvider).Meta().Name = n + } } - fis = append(fis, fi) } @@ -414,7 +550,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) } // First add any real files/directories. - rms := fs.getRoot(prefix) + rms := rfs.getRoot(prefix) for _, rm := range rms { if err := collectDir(rm, rm.fi); err != nil { return nil, err @@ -423,7 +559,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) // Next add any file mounts inside the given directory. prefixInside := prefix + filepathSeparator - fs.rootMapToReal.WalkPrefix(prefixInside, func(s string, v any) bool { + rfs.rootMapToReal.WalkPrefix(prefixInside, func(s string, v any) bool { if (strings.Count(s, filepathSeparator) - level) != 1 { // This directory is not part of the current, but we // need to include the first name part to make it @@ -437,7 +573,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) } seen[name] = true opener := func() (afero.File, error) { - return fs.Open(path) + return rfs.Open(path) } fi := newDirNameOnlyFileInfo(name, nil, opener) @@ -460,7 +596,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) seen[name] = true opener := func() (afero.File, error) { - return fs.Open(rm.From) + return rfs.Open(rm.From) } fi := newDirNameOnlyFileInfo(name, rm.Meta, opener) @@ -473,7 +609,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) }) // Finally add any ancestor dirs with files in this directory. - ancestors := fs.getAncestors(prefix) + ancestors := rfs.getAncestors(prefix) for _, root := range ancestors { subdir := strings.TrimPrefix(prefix, root.key) for _, rm := range root.roots { @@ -491,7 +627,7 @@ func (fs *RootMappingFs) collectDirEntries(prefix string) ([]os.FileInfo, error) return fis, nil } -func (fs *RootMappingFs) doLstat(name string) ([]FileMetaInfo, error) { +func (fs *RootMappingFs) doStat(name string) ([]FileMetaInfo, error) { name = fs.cleanName(name) key := filepathSeparator + name @@ -504,7 +640,7 @@ func (fs *RootMappingFs) doLstat(name string) ([]FileMetaInfo, error) { return []FileMetaInfo{newDirNameOnlyFileInfo(name, nil, fs.virtualDirOpener(name))}, nil } - // Find any real files or directories with this key. + // Find any real directories with this key. _, roots := fs.getRoots(key) if roots == nil { return nil, &os.PathError{Op: "LStat", Path: name, Err: os.ErrNotExist} @@ -515,7 +651,7 @@ func (fs *RootMappingFs) doLstat(name string) ([]FileMetaInfo, error) { for _, rm := range roots { var fi FileMetaInfo - fi, _, err = fs.statRoot(rm, name) + fi, err = fs.statRoot(rm, name) if err == nil { fis = append(fis, fi) } @@ -565,33 +701,52 @@ func (fs *RootMappingFs) doLstat(name string) ([]FileMetaInfo, error) { return []FileMetaInfo{roots[0].fi}, nil } -func (fs *RootMappingFs) statRoot(root RootMapping, name string) (FileMetaInfo, bool, error) { - if !root.Meta.InclusionFilter.Match(root.trimFrom(name), root.fi.IsDir()) { - return nil, false, os.ErrNotExist +func (fs *RootMappingFs) statRoot(root RootMapping, filename string) (FileMetaInfo, error) { + dir, name := filepath.Split(filename) + if root.Meta.Rename != nil { + if n := root.Meta.Rename(name, false); n != name { + filename = filepath.Join(dir, n) + } + } + + if !root.Meta.InclusionFilter.Match(root.trimFrom(filename), root.fi.IsDir()) { + return nil, os.ErrNotExist } - filename := root.filename(name) - fi, b, err := lstatIfPossible(fs.Fs, filename) + filename = root.filename(filename) + fi, err := fs.Fs.Stat(filename) if err != nil { - return nil, b, err + return nil, err } var opener func() (afero.File, error) if fi.IsDir() { - // Make sure metadata gets applied in Readdir. + // Make sure metadata gets applied in ReadDir. opener = fs.realDirOpener(filename, root.Meta) } else { + if root.Meta.Rename != nil { + if n := root.Meta.Rename(fi.Name(), true); n != fi.Name() { + meta := fi.(MetaProvider).Meta() + + meta.Name = n + + } + } + // Opens the real file directly. opener = func() (afero.File, error) { return fs.Fs.Open(filename) } + } - return decorateFileInfo(fi, fs.Fs, opener, "", "", root.Meta), b, nil + fim := decorateFileInfo(fi, opener, "", root.Meta) + + return fim, nil } func (fs *RootMappingFs) virtualDirOpener(name string) func() (afero.File, error) { - return func() (afero.File, error) { return &rootMappingFile{name: name, fs: fs}, nil } + return func() (afero.File, error) { return &rootMappingDir{name: name, fs: fs}, nil } } func (fs *RootMappingFs) realDirOpener(name string, meta *FileMeta) func() (afero.File, error) { @@ -600,39 +755,41 @@ func (fs *RootMappingFs) realDirOpener(name string, meta *FileMeta) func() (afer if err != nil { return nil, err } - return &rootMappingFile{name: name, meta: meta, fs: fs, File: f}, nil + return &rootMappingDir{name: name, meta: meta, fs: fs, DirOnlyOps: f}, nil } } -type rootMappingFile struct { - afero.File +var _ iofs.ReadDirFile = (*rootMappingDir)(nil) + +type rootMappingDir struct { + *noOpRegularFileOps + DirOnlyOps fs *RootMappingFs name string meta *FileMeta } -func (f *rootMappingFile) Close() error { - if f.File == nil { +func (f *rootMappingDir) Close() error { + if f.DirOnlyOps == nil { return nil } - return f.File.Close() + return f.DirOnlyOps.Close() } -func (f *rootMappingFile) Name() string { +func (f *rootMappingDir) Name() string { return f.name } -func (f *rootMappingFile) Readdir(count int) ([]os.FileInfo, error) { - if f.File != nil { - - fis, err := f.File.Readdir(count) +func (f *rootMappingDir) ReadDir(count int) ([]iofs.DirEntry, error) { + if f.DirOnlyOps != nil { + fis, err := f.DirOnlyOps.(iofs.ReadDirFile).ReadDir(count) if err != nil { return nil, err } - var result []os.FileInfo + var result []iofs.DirEntry for _, fi := range fis { - fim := decorateFileInfo(fi, f.fs, nil, "", "", f.meta) + fim := decorateFileInfo(fi, nil, "", f.meta) meta := fim.Meta() if f.meta.InclusionFilter.Match(strings.TrimPrefix(meta.Filename, meta.SourceRoot), fim.IsDir()) { result = append(result, fim) @@ -644,10 +801,31 @@ func (f *rootMappingFile) Readdir(count int) ([]os.FileInfo, error) { return f.fs.collectDirEntries(f.name) } -func (f *rootMappingFile) Readdirnames(count int) ([]string, error) { - dirs, err := f.Readdir(count) +// Sentinal error to signal that a file is a directory. +var errIsDir = errors.New("isDir") + +func (f *rootMappingDir) Stat() (iofs.FileInfo, error) { + return nil, errIsDir +} + +func (f *rootMappingDir) Readdir(count int) ([]os.FileInfo, error) { + panic("not supported: use ReadDir") +} + +// Note that Readdirnames preserves the order of the underlying filesystem(s), +// which is usually directory order. +func (f *rootMappingDir) Readdirnames(count int) ([]string, error) { + dirs, err := f.ReadDir(count) if err != nil { return nil, err } - return fileInfosToNames(dirs), nil + return dirEntriesToNames(dirs), nil +} + +func dirEntriesToNames(fis []iofs.DirEntry) []string { + names := make([]string, len(fis)) + for i, d := range fis { + names[i] = d.Name() + } + return names } diff --git a/hugofs/rootmapping_fs_test.go b/hugofs/rootmapping_fs_test.go index b71462a8d..982e6dfaf 100644 --- a/hugofs/rootmapping_fs_test.go +++ b/hugofs/rootmapping_fs_test.go @@ -15,11 +15,12 @@ package hugofs import ( "fmt" - "io" "path/filepath" "sort" "testing" + iofs "io/fs" + "github.com/gohugoio/hugo/config" "github.com/gohugoio/hugo/hugofs/glob" @@ -35,16 +36,16 @@ func TestLanguageRootMapping(t *testing.T) { fs := NewBaseFileDecorator(afero.NewMemMapFs()) - c.Assert(afero.WriteFile(fs, filepath.Join("content/sv/svdir", "main.txt"), []byte("main sv"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("content/sv/svdir", "main.txt"), []byte("main sv"), 0o755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", "sv-f.txt"), []byte("some sv blog content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent", "en-f.txt"), []byte("some en blog content in a"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent/d1", "sv-d1-f.txt"), []byte("some sv blog content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent/d1", "en-d1-f.txt"), []byte("some en blog content in a"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", "sv-f.txt"), []byte("some sv blog content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent", "en-f.txt"), []byte("some en blog content in a"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent/d1", "sv-d1-f.txt"), []byte("some sv blog content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent/d1", "en-d1-f.txt"), []byte("some en blog content in a"), 0o755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myotherenblogcontent", "en-f2.txt"), []byte("some en content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvdocs", "sv-docs.txt"), []byte("some sv docs content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/b/myenblogcontent", "en-b-f.txt"), []byte("some en content"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myotherenblogcontent", "en-f2.txt"), []byte("some en content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvdocs", "sv-docs.txt"), []byte("some sv docs content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/b/myenblogcontent", "en-b-f.txt"), []byte("some en content"), 0o755), qt.IsNil) rfs, err := NewRootMappingFs(fs, RootMapping{ @@ -76,12 +77,12 @@ func TestLanguageRootMapping(t *testing.T) { c.Assert(err, qt.IsNil) - collected, err := collectFilenames(rfs, "content", "content") + collected, err := collectPaths(rfs, "content") c.Assert(err, qt.IsNil) c.Assert(collected, qt.DeepEquals, - []string{"blog/d1/en-d1-f.txt", "blog/d1/sv-d1-f.txt", "blog/en-f.txt", "blog/en-f2.txt", "blog/sv-f.txt", "blog/svdir/main.txt", "docs/sv-docs.txt"}, qt.Commentf("%#v", collected)) + []string{"/blog/d1/en-d1-f.txt", "/blog/d1/sv-d1-f.txt", "/blog/en-f.txt", "/blog/en-f2.txt", "/blog/sv-f.txt", "/blog/svdir/main.txt", "/docs/sv-docs.txt"}, qt.Commentf("%#v", collected)) - dirs, err := rfs.Dirs(filepath.FromSlash("content/blog")) + dirs, err := rfs.Mounts(filepath.FromSlash("content/blog")) c.Assert(err, qt.IsNil) c.Assert(len(dirs), qt.Equals, 4) for _, dir := range dirs { @@ -92,7 +93,8 @@ func TestLanguageRootMapping(t *testing.T) { blog, err := rfs.Open(filepath.FromSlash("content/blog")) c.Assert(err, qt.IsNil) - fis, err := blog.Readdir(-1) + fis, err := blog.(iofs.ReadDirFile).ReadDir(-1) + c.Assert(err, qt.IsNil) for _, fi := range fis { f, err := fi.(FileMetaInfo).Meta().Open() c.Assert(err, qt.IsNil) @@ -146,10 +148,10 @@ func TestRootMappingFsDirnames(t *testing.T) { fs := NewBaseFileDecorator(afero.NewMemMapFs()) testfile := "myfile.txt" - c.Assert(fs.Mkdir("f1t", 0755), qt.IsNil) - c.Assert(fs.Mkdir("f2t", 0755), qt.IsNil) - c.Assert(fs.Mkdir("f3t", 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("f2t", testfile), []byte("some content"), 0755), qt.IsNil) + c.Assert(fs.Mkdir("f1t", 0o755), qt.IsNil) + c.Assert(fs.Mkdir("f2t", 0o755), qt.IsNil) + c.Assert(fs.Mkdir("f3t", 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("f2t", testfile), []byte("some content"), 0o755), qt.IsNil) rfs, err := newRootMappingFsFromFromTo("", fs, "static/bf1", "f1t", "static/cf2", "f2t", "static/af3", "f3t") c.Assert(err, qt.IsNil) @@ -177,8 +179,8 @@ func TestRootMappingFsFilename(t *testing.T) { testfilename := filepath.Join(workDir, "f1t/foo/file.txt") - c.Assert(fs.MkdirAll(filepath.Join(workDir, "f1t/foo"), 0777), qt.IsNil) - c.Assert(afero.WriteFile(fs, testfilename, []byte("content"), 0666), qt.IsNil) + c.Assert(fs.MkdirAll(filepath.Join(workDir, "f1t/foo"), 0o777), qt.IsNil) + c.Assert(afero.WriteFile(fs, testfilename, []byte("content"), 0o666), qt.IsNil) rfs, err := newRootMappingFsFromFromTo(workDir, fs, "static/f1", filepath.Join(workDir, "f1t"), "static/f2", filepath.Join(workDir, "f2t")) c.Assert(err, qt.IsNil) @@ -197,14 +199,14 @@ func TestRootMappingFsMount(t *testing.T) { testfile := "test.txt" - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mynoblogcontent", testfile), []byte("some no content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent", testfile), []byte("some en content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", testfile), []byte("some sv content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", "other.txt"), []byte("some sv content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/singlefiles", "no.txt"), []byte("no text"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/singlefiles", "sv.txt"), []byte("sv text"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mynoblogcontent", testfile), []byte("some no content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/myenblogcontent", testfile), []byte("some en content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", testfile), []byte("some sv content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/mysvblogcontent", "other.txt"), []byte("some sv content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/singlefiles", "no.txt"), []byte("no text"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("themes/a/singlefiles", "sv.txt"), []byte("sv text"), 0o755), qt.IsNil) - bfs := afero.NewBasePathFs(fs, "themes/a").(*afero.BasePathFs) + bfs := NewBasePathFs(fs, "themes/a") rm := []RootMapping{ // Directories { @@ -224,16 +226,16 @@ func TestRootMappingFsMount(t *testing.T) { }, // Files { - From: "content/singles/p1.md", - To: "singlefiles/no.txt", - ToBasedir: "singlefiles", - Meta: &FileMeta{Lang: "no"}, + From: "content/singles/p1.md", + To: "singlefiles/no.txt", + ToBase: "singlefiles", + Meta: &FileMeta{Lang: "no"}, }, { - From: "content/singles/p1.md", - To: "singlefiles/sv.txt", - ToBasedir: "singlefiles", - Meta: &FileMeta{Lang: "sv"}, + From: "content/singles/p1.md", + To: "singlefiles/sv.txt", + ToBase: "singlefiles", + Meta: &FileMeta{Lang: "sv"}, }, } @@ -254,49 +256,49 @@ func TestRootMappingFsMount(t *testing.T) { // Union with duplicate dir names filtered. c.Assert(dirs1, qt.DeepEquals, []string{"test.txt", "test.txt", "other.txt", "test.txt"}) - files, err := afero.ReadDir(rfs, filepath.FromSlash("content/blog")) - c.Assert(err, qt.IsNil) - c.Assert(len(files), qt.Equals, 4) - - testfilefi := files[1] - c.Assert(testfilefi.Name(), qt.Equals, testfile) - - testfilem := testfilefi.(FileMetaInfo).Meta() - c.Assert(testfilem.Filename, qt.Equals, filepath.FromSlash("themes/a/mynoblogcontent/test.txt")) - - tf, err := testfilem.Open() + d, err := rfs.Open(filepath.FromSlash("content/blog")) c.Assert(err, qt.IsNil) - defer tf.Close() - b, err := io.ReadAll(tf) + files, err := d.(iofs.ReadDirFile).ReadDir(-1) c.Assert(err, qt.IsNil) - c.Assert(string(b), qt.Equals, "some no content") - - // Ambiguous - _, err = rfs.Stat(filepath.FromSlash("content/singles/p1.md")) - c.Assert(err, qt.Not(qt.IsNil)) + c.Assert(len(files), qt.Equals, 4) singlesDir, err := rfs.Open(filepath.FromSlash("content/singles")) c.Assert(err, qt.IsNil) defer singlesDir.Close() - singles, err := singlesDir.Readdir(-1) + singles, err := singlesDir.(iofs.ReadDirFile).ReadDir(-1) c.Assert(err, qt.IsNil) c.Assert(singles, qt.HasLen, 2) for i, lang := range []string{"no", "sv"} { fi := singles[i].(FileMetaInfo) - c.Assert(fi.Meta().PathFile(), qt.Equals, filepath.FromSlash("themes/a/singlefiles/"+lang+".txt")) c.Assert(fi.Meta().Lang, qt.Equals, lang) c.Assert(fi.Name(), qt.Equals, "p1.md") } + + // Test ReverseLookup. + // Single file mounts. + cps, err := rfs.ReverseLookup(filepath.FromSlash("singlefiles/no.txt"), true) + c.Assert(err, qt.IsNil) + c.Assert(cps, qt.DeepEquals, []ComponentPath{ + {Component: "content", Path: "singles/p1.md", Lang: "no"}, + {Component: "content", Path: "singles/p1.md", Lang: "sv"}, + }) + + // File inside directory mount. + cps, err = rfs.ReverseLookup(filepath.FromSlash("mynoblogcontent/test.txt"), true) + c.Assert(err, qt.IsNil) + c.Assert(cps, qt.DeepEquals, []ComponentPath{ + {Component: "content", Path: "blog/test.txt", Lang: "no"}, + }) } func TestRootMappingFsMountOverlap(t *testing.T) { c := qt.New(t) fs := NewBaseFileDecorator(afero.NewMemMapFs()) - c.Assert(afero.WriteFile(fs, filepath.FromSlash("da/a.txt"), []byte("some no content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.FromSlash("db/b.txt"), []byte("some no content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.FromSlash("dc/c.txt"), []byte("some no content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.FromSlash("de/e.txt"), []byte("some no content"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.FromSlash("da/a.txt"), []byte("some no content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.FromSlash("db/b.txt"), []byte("some no content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.FromSlash("dc/c.txt"), []byte("some no content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.FromSlash("de/e.txt"), []byte("some no content"), 0o755), qt.IsNil) rm := []RootMapping{ { @@ -349,24 +351,24 @@ func TestRootMappingFsOs(t *testing.T) { defer clean() testfile := "myfile.txt" - c.Assert(fs.Mkdir(filepath.Join(d, "f1t"), 0755), qt.IsNil) - c.Assert(fs.Mkdir(filepath.Join(d, "f2t"), 0755), qt.IsNil) - c.Assert(fs.Mkdir(filepath.Join(d, "f3t"), 0755), qt.IsNil) + c.Assert(fs.Mkdir(filepath.Join(d, "f1t"), 0o755), qt.IsNil) + c.Assert(fs.Mkdir(filepath.Join(d, "f2t"), 0o755), qt.IsNil) + c.Assert(fs.Mkdir(filepath.Join(d, "f3t"), 0o755), qt.IsNil) // Deep structure deepDir := filepath.Join(d, "d1", "d2", "d3", "d4", "d5") - c.Assert(fs.MkdirAll(deepDir, 0755), qt.IsNil) + c.Assert(fs.MkdirAll(deepDir, 0o755), qt.IsNil) for i := 1; i <= 3; i++ { - c.Assert(fs.MkdirAll(filepath.Join(d, "d1", "d2", "d3", "d4", fmt.Sprintf("d4-%d", i)), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join(d, "d1", "d2", "d3", fmt.Sprintf("f-%d.txt", i)), []byte("some content"), 0755), qt.IsNil) + c.Assert(fs.MkdirAll(filepath.Join(d, "d1", "d2", "d3", "d4", fmt.Sprintf("d4-%d", i)), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(d, "d1", "d2", "d3", fmt.Sprintf("f-%d.txt", i)), []byte("some content"), 0o755), qt.IsNil) } - c.Assert(afero.WriteFile(fs, filepath.Join(d, "f2t", testfile), []byte("some content"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(d, "f2t", testfile), []byte("some content"), 0o755), qt.IsNil) // https://github.com/gohugoio/hugo/issues/6854 mystaticDir := filepath.Join(d, "mystatic", "a", "b", "c") - c.Assert(fs.MkdirAll(mystaticDir, 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join(mystaticDir, "ms-1.txt"), []byte("some content"), 0755), qt.IsNil) + c.Assert(fs.MkdirAll(mystaticDir, 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(mystaticDir, "ms-1.txt"), []byte("some content"), 0o755), qt.IsNil) rfs, err := newRootMappingFsFromFromTo( d, @@ -407,33 +409,30 @@ func TestRootMappingFsOs(t *testing.T) { c.Assert(getDirnames("static/a/b/c"), qt.DeepEquals, []string{"d4", "f-1.txt", "f-2.txt", "f-3.txt", "ms-1.txt"}) c.Assert(getDirnames("static/a/b/c/d4"), qt.DeepEquals, []string{"d4-1", "d4-2", "d4-3", "d5"}) - all, err := collectFilenames(rfs, "static", "static") + all, err := collectPaths(rfs, "static") c.Assert(err, qt.IsNil) - c.Assert(all, qt.DeepEquals, []string{"a/b/c/f-1.txt", "a/b/c/f-2.txt", "a/b/c/f-3.txt", "a/b/c/ms-1.txt", "cf2/myfile.txt"}) + c.Assert(all, qt.DeepEquals, []string{"/a/b/c/f-1.txt", "/a/b/c/f-2.txt", "/a/b/c/f-3.txt", "/a/b/c/ms-1.txt", "/cf2/myfile.txt"}) - fis, err := collectFileinfos(rfs, "static", "static") + fis, err := collectFileinfos(rfs, "static") c.Assert(err, qt.IsNil) - c.Assert(fis[9].Meta().PathFile(), qt.Equals, filepath.FromSlash("d1/d2/d3/f-1.txt")) - dirc := fis[3].Meta() f, err := dirc.Open() c.Assert(err, qt.IsNil) defer f.Close() - fileInfos, err := f.Readdir(-1) + dirEntries, err := f.(iofs.ReadDirFile).ReadDir(-1) c.Assert(err, qt.IsNil) - sortFileInfos(fileInfos) + sortDirEntries(dirEntries) i := 0 - for _, fi := range fileInfos { + for _, fi := range dirEntries { if fi.IsDir() || fi.Name() == "ms-1.txt" { continue } i++ meta := fi.(FileMetaInfo).Meta() c.Assert(meta.Filename, qt.Equals, filepath.Join(d, fmt.Sprintf("/d1/d2/d3/f-%d.txt", i))) - c.Assert(meta.PathFile(), qt.Equals, filepath.FromSlash(fmt.Sprintf("d1/d2/d3/f-%d.txt", i))) } _, err = rfs.Stat(filepath.FromSlash("layouts/d2/d3/f-1.txt")) @@ -452,17 +451,17 @@ func TestRootMappingFsOsBase(t *testing.T) { // Deep structure deepDir := filepath.Join(d, "d1", "d2", "d3", "d4", "d5") - c.Assert(fs.MkdirAll(deepDir, 0755), qt.IsNil) + c.Assert(fs.MkdirAll(deepDir, 0o755), qt.IsNil) for i := 1; i <= 3; i++ { - c.Assert(fs.MkdirAll(filepath.Join(d, "d1", "d2", "d3", "d4", fmt.Sprintf("d4-%d", i)), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join(d, "d1", "d2", "d3", fmt.Sprintf("f-%d.txt", i)), []byte("some content"), 0755), qt.IsNil) + c.Assert(fs.MkdirAll(filepath.Join(d, "d1", "d2", "d3", "d4", fmt.Sprintf("d4-%d", i)), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(d, "d1", "d2", "d3", fmt.Sprintf("f-%d.txt", i)), []byte("some content"), 0o755), qt.IsNil) } mystaticDir := filepath.Join(d, "mystatic", "a", "b", "c") - c.Assert(fs.MkdirAll(mystaticDir, 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join(mystaticDir, "ms-1.txt"), []byte("some content"), 0755), qt.IsNil) + c.Assert(fs.MkdirAll(mystaticDir, 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(mystaticDir, "ms-1.txt"), []byte("some content"), 0o755), qt.IsNil) - bfs := afero.NewBasePathFs(fs, d) + bfs := NewBasePathFs(fs, d) rfs, err := newRootMappingFsFromFromTo( "", @@ -470,6 +469,7 @@ func TestRootMappingFsOsBase(t *testing.T) { "static", "mystatic", "static/a/b/c", filepath.Join("d1", "d2", "d3"), ) + c.Assert(err, qt.IsNil) getDirnames := func(dirname string) []string { dirname = filepath.FromSlash(dirname) @@ -491,13 +491,13 @@ func TestRootMappingFileFilter(t *testing.T) { for _, lang := range []string{"no", "en", "fr"} { for i := 1; i <= 3; i++ { - c.Assert(afero.WriteFile(fs, filepath.Join(lang, fmt.Sprintf("my%s%d.txt", lang, i)), []byte("some text file for"+lang), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(lang, fmt.Sprintf("my%s%d.txt", lang, i)), []byte("some text file for"+lang), 0o755), qt.IsNil) } } for _, lang := range []string{"no", "en", "fr"} { for i := 1; i <= 3; i++ { - c.Assert(afero.WriteFile(fs, filepath.Join(lang, "sub", fmt.Sprintf("mysub%s%d.txt", lang, i)), []byte("some text file for"+lang), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join(lang, "sub", fmt.Sprintf("mysub%s%d.txt", lang, i)), []byte("some text file for"+lang), 0o755), qt.IsNil) } } @@ -545,9 +545,11 @@ func TestRootMappingFileFilter(t *testing.T) { c.Assert(err, qt.IsNil) c.Assert(len(dirEntriesSub), qt.Equals, 3) - dirEntries, err := afero.ReadDir(rfs, "content") + f, err := rfs.Open("content") + c.Assert(err, qt.IsNil) + defer f.Close() + dirEntries, err := f.(iofs.ReadDirFile).ReadDir(-1) c.Assert(err, qt.IsNil) c.Assert(len(dirEntries), qt.Equals, 4) - } diff --git a/hugofs/slice_fs.go b/hugofs/slice_fs.go deleted file mode 100644 index 574a5cb5f..000000000 --- a/hugofs/slice_fs.go +++ /dev/null @@ -1,303 +0,0 @@ -// Copyright 2019 The Hugo Authors. All rights reserved. -// -// Licensed under the Apache License, Version 2.0 (the "License"); -// you may not use this file except in compliance with the License. -// You may obtain a copy of the License at -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, software -// distributed under the License is distributed on an "AS IS" BASIS, -// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -// See the License for the specific language governing permissions and -// limitations under the License. - -package hugofs - -import ( - "fmt" - "os" - "syscall" - "time" - - "errors" - - "github.com/gohugoio/hugo/common/herrors" - "github.com/spf13/afero" -) - -var ( - _ afero.Fs = (*SliceFs)(nil) - _ afero.Lstater = (*SliceFs)(nil) - _ FilesystemsUnwrapper = (*SliceFs)(nil) - _ afero.File = (*sliceDir)(nil) -) - -func NewSliceFs(dirs ...FileMetaInfo) (afero.Fs, error) { - if len(dirs) == 0 { - return NoOpFs, nil - } - - for _, dir := range dirs { - if !dir.IsDir() { - return nil, errors.New("this fs supports directories only") - } - } - - fs := &SliceFs{ - dirs: dirs, - } - - return fs, nil -} - -// SliceFs is an ordered composite filesystem. -type SliceFs struct { - dirs []FileMetaInfo -} - -func (fs *SliceFs) UnwrapFilesystems() []afero.Fs { - var fss []afero.Fs - for _, dir := range fs.dirs { - fss = append(fss, dir.Meta().Fs) - } - return fss -} - -func (fs *SliceFs) Chmod(n string, m os.FileMode) error { - return syscall.EPERM -} - -func (fs *SliceFs) Chtimes(n string, a, m time.Time) error { - return syscall.EPERM -} - -func (fs *SliceFs) Chown(n string, uid, gid int) error { - return syscall.EPERM -} - -func (fs *SliceFs) LstatIfPossible(name string) (os.FileInfo, bool, error) { - fi, _, err := fs.pickFirst(name) - if err != nil { - return nil, false, err - } - - if fi.IsDir() { - return decorateFileInfo(fi, fs, fs.getOpener(name), "", "", nil), false, nil - } - - return nil, false, fmt.Errorf("lstat: files not supported: %q", name) -} - -func (fs *SliceFs) Mkdir(n string, p os.FileMode) error { - return syscall.EPERM -} - -func (fs *SliceFs) MkdirAll(n string, p os.FileMode) error { - return syscall.EPERM -} - -func (fs *SliceFs) Name() string { - return "SliceFs" -} - -func (fs *SliceFs) Open(name string) (afero.File, error) { - fi, idx, err := fs.pickFirst(name) - if err != nil { - return nil, err - } - - if !fi.IsDir() { - panic("currently only dirs in here") - } - - return &sliceDir{ - lfs: fs, - idx: idx, - dirname: name, - }, nil -} - -func (fs *SliceFs) OpenFile(name string, flag int, perm os.FileMode) (afero.File, error) { - panic("not implemented") -} - -func (fs *SliceFs) ReadDir(name string) ([]os.FileInfo, error) { - panic("not implemented") -} - -func (fs *SliceFs) Remove(n string) error { - return syscall.EPERM -} - -func (fs *SliceFs) RemoveAll(p string) error { - return syscall.EPERM -} - -func (fs *SliceFs) Rename(o, n string) error { - return syscall.EPERM -} - -func (fs *SliceFs) Stat(name string) (os.FileInfo, error) { - fi, _, err := fs.LstatIfPossible(name) - return fi, err -} - -func (fs *SliceFs) Create(n string) (afero.File, error) { - return nil, syscall.EPERM -} - -func (fs *SliceFs) getOpener(name string) func() (afero.File, error) { - return func() (afero.File, error) { - return fs.Open(name) - } -} - -func (fs *SliceFs) pickFirst(name string) (os.FileInfo, int, error) { - for i, mfs := range fs.dirs { - meta := mfs.Meta() - fs := meta.Fs - fi, _, err := lstatIfPossible(fs, name) - if err == nil { - // Gotta match! - return fi, i, nil - } - - if !herrors.IsNotExist(err) { - // Real error - return nil, -1, err - } - } - - // Not found - return nil, -1, os.ErrNotExist -} - -func (fs *SliceFs) readDirs(name string, startIdx, count int) ([]os.FileInfo, error) { - collect := func(lfs *FileMeta) ([]os.FileInfo, error) { - d, err := lfs.Fs.Open(name) - if err != nil { - if !herrors.IsNotExist(err) { - return nil, err - } - return nil, nil - } else { - defer d.Close() - dirs, err := d.Readdir(-1) - if err != nil { - return nil, err - } - return dirs, nil - } - } - - var dirs []os.FileInfo - - for i := startIdx; i < len(fs.dirs); i++ { - mfs := fs.dirs[i] - - fis, err := collect(mfs.Meta()) - if err != nil { - return nil, err - } - - dirs = append(dirs, fis...) - - } - - seen := make(map[string]bool) - var duplicates []int - for i, fi := range dirs { - if !fi.IsDir() { - continue - } - - if seen[fi.Name()] { - duplicates = append(duplicates, i) - } else { - // Make sure it's opened by this filesystem. - dirs[i] = decorateFileInfo(fi, fs, fs.getOpener(fi.(FileMetaInfo).Meta().Filename), "", "", nil) - seen[fi.Name()] = true - } - } - - // Remove duplicate directories, keep first. - if len(duplicates) > 0 { - for i := len(duplicates) - 1; i >= 0; i-- { - idx := duplicates[i] - dirs = append(dirs[:idx], dirs[idx+1:]...) - } - } - - if count > 0 && len(dirs) >= count { - return dirs[:count], nil - } - - return dirs, nil -} - -type sliceDir struct { - lfs *SliceFs - idx int - dirname string -} - -func (f *sliceDir) Close() error { - return nil -} - -func (f *sliceDir) Name() string { - return f.dirname -} - -func (f *sliceDir) Read(p []byte) (n int, err error) { - panic("not implemented") -} - -func (f *sliceDir) ReadAt(p []byte, off int64) (n int, err error) { - panic("not implemented") -} - -func (f *sliceDir) Readdir(count int) ([]os.FileInfo, error) { - return f.lfs.readDirs(f.dirname, f.idx, count) -} - -func (f *sliceDir) Readdirnames(count int) ([]string, error) { - dirsi, err := f.Readdir(count) - if err != nil { - return nil, err - } - - dirs := make([]string, len(dirsi)) - for i, d := range dirsi { - dirs[i] = d.Name() - } - return dirs, nil -} - -func (f *sliceDir) Seek(offset int64, whence int) (int64, error) { - panic("not implemented") -} - -func (f *sliceDir) Stat() (os.FileInfo, error) { - panic("not implemented") -} - -func (f *sliceDir) Sync() error { - panic("not implemented") -} - -func (f *sliceDir) Truncate(size int64) error { - panic("not implemented") -} - -func (f *sliceDir) Write(p []byte) (n int, err error) { - panic("not implemented") -} - -func (f *sliceDir) WriteAt(p []byte, off int64) (n int, err error) { - panic("not implemented") -} - -func (f *sliceDir) WriteString(s string) (ret int, err error) { - panic("not implemented") -} diff --git a/hugofs/walk.go b/hugofs/walk.go index e883f892e..18667a5fc 100644 --- a/hugofs/walk.go +++ b/hugofs/walk.go @@ -15,73 +15,60 @@ package hugofs import ( "fmt" - "os" + "io/fs" "path/filepath" "sort" "strings" "github.com/gohugoio/hugo/common/herrors" "github.com/gohugoio/hugo/common/loggers" - - "errors" + "github.com/gohugoio/hugo/common/paths" "github.com/spf13/afero" ) type ( - WalkFunc func(path string, info FileMetaInfo, err error) error + WalkFunc func(path string, info FileMetaInfo) error WalkHook func(dir FileMetaInfo, path string, readdir []FileMetaInfo) ([]FileMetaInfo, error) ) type Walkway struct { - fs afero.Fs - root string - basePath string - logger loggers.Logger - // May be pre-set - fi FileMetaInfo - dirEntries []FileMetaInfo - - walkFn WalkFunc + // Prevent a walkway to be walked more than once. walked bool - // We may traverse symbolic links and bite ourself. - seen map[string]bool - - // Optional hooks - hookPre WalkHook - hookPost WalkHook + // Config from client. + cfg WalkwayConfig } type WalkwayConfig struct { - Fs afero.Fs - Root string - BasePath string + // The filesystem to walk. + Fs afero.Fs + // The root to start from in Fs. + Root string + + // The logger to use. Logger loggers.Logger // One or both of these may be pre-set. - Info FileMetaInfo - DirEntries []FileMetaInfo + Info FileMetaInfo // The start info. + DirEntries []FileMetaInfo // The start info's dir entries. + // Will be called in order. + HookPre WalkHook // Optional. WalkFn WalkFunc - HookPre WalkHook - HookPost WalkHook + HookPost WalkHook // Optional. + + // Some optional flags. + FailOnNotExist bool // If set, return an error if a directory is not found. + SortDirEntries bool // If set, sort the dir entries by Name before calling the WalkFn, default is ReaDir order. } func NewWalkway(cfg WalkwayConfig) *Walkway { - var fs afero.Fs - if cfg.Info != nil { - fs = cfg.Info.Meta().Fs - } else { - fs = cfg.Fs - } - - basePath := cfg.BasePath - if basePath != "" && !strings.HasSuffix(basePath, filepathSeparator) { - basePath += filepathSeparator + if cfg.Fs == nil { + panic("fs must be set") } logger := cfg.Logger @@ -90,16 +77,8 @@ func NewWalkway(cfg WalkwayConfig) *Walkway { } return &Walkway{ - fs: fs, - root: cfg.Root, - basePath: basePath, - fi: cfg.Info, - dirEntries: cfg.DirEntries, - walkFn: cfg.WalkFn, - hookPre: cfg.HookPre, - hookPost: cfg.HookPost, - logger: logger, - seen: make(map[string]bool), + cfg: cfg, + logger: logger, } } @@ -109,53 +88,16 @@ func (w *Walkway) Walk() error { } w.walked = true - if w.fs == NoOpFs { + if w.cfg.Fs == NoOpFs { return nil } - var fi FileMetaInfo - if w.fi != nil { - fi = w.fi - } else { - info, _, err := lstatIfPossible(w.fs, w.root) - if err != nil { - if herrors.IsNotExist(err) { - return nil - } - - if w.checkErr(w.root, err) { - return nil - } - return w.walkFn(w.root, nil, fmt.Errorf("walk: %q: %w", w.root, err)) - } - fi = info.(FileMetaInfo) - } - - if !fi.IsDir() { - return w.walkFn(w.root, nil, errors.New("file to walk must be a directory")) - } - - return w.walk(w.root, fi, w.dirEntries, w.walkFn) -} - -// if the filesystem supports it, use Lstat, else use fs.Stat -func lstatIfPossible(fs afero.Fs, path string) (os.FileInfo, bool, error) { - if lfs, ok := fs.(afero.Lstater); ok { - fi, b, err := lfs.LstatIfPossible(path) - return fi, b, err - } - fi, err := fs.Stat(path) - return fi, false, err + return w.walk(w.cfg.Root, w.cfg.Info, w.cfg.DirEntries) } // checkErr returns true if the error is handled. func (w *Walkway) checkErr(filename string, err error) bool { - if err == ErrPermissionSymlink { - logUnsupportedSymlink(filename, w.logger) - return true - } - - if herrors.IsNotExist(err) { + if herrors.IsNotExist(err) && !w.cfg.FailOnNotExist { // The file may be removed in process. // This may be a ERROR situation, but it is not possible // to determine as a general case. @@ -166,115 +108,73 @@ func (w *Walkway) checkErr(filename string, err error) bool { return false } -func logUnsupportedSymlink(filename string, logger loggers.Logger) { - logger.Warnf("Unsupported symlink found in %q, skipping.", filename) -} - // walk recursively descends path, calling walkFn. -// It follow symlinks if supported by the filesystem, but only the same path once. -func (w *Walkway) walk(path string, info FileMetaInfo, dirEntries []FileMetaInfo, walkFn WalkFunc) error { - err := walkFn(path, info, nil) +func (w *Walkway) walk(path string, info FileMetaInfo, dirEntries []FileMetaInfo) error { + pathRel := strings.TrimPrefix(path, w.cfg.Root) + + if info == nil { + var err error + fi, err := w.cfg.Fs.Stat(path) + if err != nil { + if path == w.cfg.Root && herrors.IsNotExist(err) { + return nil + } + if w.checkErr(path, err) { + return nil + } + return fmt.Errorf("walk: stat: %s", err) + } + info = fi.(FileMetaInfo) + } + + err := w.cfg.WalkFn(path, info) if err != nil { if info.IsDir() && err == filepath.SkipDir { return nil } return err } + if !info.IsDir() { return nil } - meta := info.Meta() - filename := meta.Filename - if dirEntries == nil { - f, err := w.fs.Open(path) + f, err := w.cfg.Fs.Open(path) if err != nil { if w.checkErr(path, err) { return nil } - return walkFn(path, info, fmt.Errorf("walk: open %q (%q): %w", path, w.root, err)) + return fmt.Errorf("walk: open: path: %q filename: %q: %s", path, info.Meta().Filename, err) } + fis, err := f.(fs.ReadDirFile).ReadDir(-1) - fis, err := f.Readdir(-1) f.Close() if err != nil { - if w.checkErr(filename, err) { + if w.checkErr(path, err) { return nil } - return walkFn(path, info, fmt.Errorf("walk: Readdir: %w", err)) + return fmt.Errorf("walk: Readdir: %w", err) } - dirEntries = fileInfosToFileMetaInfos(fis) + dirEntries = DirEntriesToFileMetaInfos(fis) + for _, fi := range dirEntries { + if fi.Meta().PathInfo == nil { + fi.Meta().PathInfo = paths.Parse("", filepath.Join(pathRel, fi.Name())) + } + } - if !meta.IsOrdered { + if w.cfg.SortDirEntries { sort.Slice(dirEntries, func(i, j int) bool { - fii := dirEntries[i] - fij := dirEntries[j] - - fim, fjm := fii.Meta(), fij.Meta() - - // Pull bundle headers to the top. - ficlass, fjclass := fim.Classifier, fjm.Classifier - if ficlass != fjclass { - return ficlass < fjclass - } - - // With multiple content dirs with different languages, - // there can be duplicate files, and a weight will be added - // to the closest one. - fiw, fjw := fim.Weight, fjm.Weight - if fiw != fjw { - - return fiw > fjw - } - - // When we walk into a symlink, we keep the reference to - // the original name. - fin, fjn := fim.Name, fjm.Name - if fin != "" && fjn != "" { - return fin < fjn - } - - return fii.Name() < fij.Name() + return dirEntries[i].Name() < dirEntries[j].Name() }) } - } - - // First add some metadata to the dir entries - for _, fi := range dirEntries { - fim := fi.(FileMetaInfo) - - meta := fim.Meta() - - // Note that we use the original Name even if it's a symlink. - name := meta.Name - if name == "" { - name = fim.Name() - } - - if name == "" { - panic(fmt.Sprintf("[%s] no name set in %v", path, meta)) - } - pathn := filepath.Join(path, name) - - pathMeta := pathn - if w.basePath != "" { - pathMeta = strings.TrimPrefix(pathn, w.basePath) - } - - meta.Path = normalizeFilename(pathMeta) - meta.PathWalk = pathn - if fim.IsDir() && meta.IsSymlink && w.isSeen(meta.Filename) { - // Prevent infinite recursion - // Possible cyclic reference - meta.SkipDir = true - } } - if w.hookPre != nil { - dirEntries, err = w.hookPre(info, path, dirEntries) + if w.cfg.HookPre != nil { + var err error + dirEntries, err = w.cfg.HookPre(info, path, dirEntries) if err != nil { if err == filepath.SkipDir { return nil @@ -283,24 +183,19 @@ func (w *Walkway) walk(path string, info FileMetaInfo, dirEntries []FileMetaInfo } } - for _, fi := range dirEntries { - fim := fi.(FileMetaInfo) - meta := fim.Meta() - - if meta.SkipDir { - continue - } - - err := w.walk(meta.PathWalk, fim, nil, walkFn) + for _, fim := range dirEntries { + nextPath := filepath.Join(path, fim.Name()) + err := w.walk(nextPath, fim, nil) if err != nil { - if !fi.IsDir() || err != filepath.SkipDir { + if !fim.IsDir() || err != filepath.SkipDir { return err } } } - if w.hookPost != nil { - dirEntries, err = w.hookPost(info, path, dirEntries) + if w.cfg.HookPost != nil { + var err error + dirEntries, err = w.cfg.HookPost(info, path, dirEntries) if err != nil { if err == filepath.SkipDir { return nil @@ -310,16 +205,3 @@ func (w *Walkway) walk(path string, info FileMetaInfo, dirEntries []FileMetaInfo } return nil } - -func (w *Walkway) isSeen(filename string) bool { - if filename == "" { - return false - } - - if w.seen[filename] { - return true - } - - w.seen[filename] = true - return false -} diff --git a/hugofs/walk_test.go b/hugofs/walk_test.go index 2e162fa72..7366d008d 100644 --- a/hugofs/walk_test.go +++ b/hugofs/walk_test.go @@ -15,17 +15,13 @@ package hugofs import ( "context" + "errors" "fmt" - "os" "path/filepath" - "runtime" "strings" "testing" - "errors" - "github.com/gohugoio/hugo/common/para" - "github.com/gohugoio/hugo/htesting" "github.com/spf13/afero" @@ -37,14 +33,14 @@ func TestWalk(t *testing.T) { fs := NewBaseFileDecorator(afero.NewMemMapFs()) - afero.WriteFile(fs, "b.txt", []byte("content"), 0777) - afero.WriteFile(fs, "c.txt", []byte("content"), 0777) - afero.WriteFile(fs, "a.txt", []byte("content"), 0777) + afero.WriteFile(fs, "b.txt", []byte("content"), 0o777) + afero.WriteFile(fs, "c.txt", []byte("content"), 0o777) + afero.WriteFile(fs, "a.txt", []byte("content"), 0o777) - names, err := collectFilenames(fs, "", "") + names, err := collectPaths(fs, "") c.Assert(err, qt.IsNil) - c.Assert(names, qt.DeepEquals, []string{"a.txt", "b.txt", "c.txt"}) + c.Assert(names, qt.DeepEquals, []string{"/a.txt", "/b.txt", "/c.txt"}) } func TestWalkRootMappingFs(t *testing.T) { @@ -55,9 +51,9 @@ func TestWalkRootMappingFs(t *testing.T) { testfile := "test.txt" - c.Assert(afero.WriteFile(fs, filepath.Join("a/b", testfile), []byte("some content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("c/d", testfile), []byte("some content"), 0755), qt.IsNil) - c.Assert(afero.WriteFile(fs, filepath.Join("e/f", testfile), []byte("some content"), 0755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("a/b", testfile), []byte("some content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("c/d", testfile), []byte("some content"), 0o755), qt.IsNil) + c.Assert(afero.WriteFile(fs, filepath.Join("e/f", testfile), []byte("some content"), 0o755), qt.IsNil) rm := []RootMapping{ { @@ -77,16 +73,16 @@ func TestWalkRootMappingFs(t *testing.T) { rfs, err := NewRootMappingFs(fs, rm...) c.Assert(err, qt.IsNil) - return afero.NewBasePathFs(rfs, "static") + return NewBasePathFs(rfs, "static") } c.Run("Basic", func(c *qt.C) { bfs := prepare(c) - names, err := collectFilenames(bfs, "", "") + names, err := collectPaths(bfs, "") c.Assert(err, qt.IsNil) - c.Assert(names, qt.DeepEquals, []string{"a/test.txt", "b/test.txt", "c/test.txt"}) + c.Assert(names, qt.DeepEquals, []string{"/a/test.txt", "/b/test.txt", "/c/test.txt"}) }) c.Run("Para", func(c *qt.C) { @@ -97,7 +93,7 @@ func TestWalkRootMappingFs(t *testing.T) { for i := 0; i < 8; i++ { r.Run(func() error { - _, err := collectFilenames(bfs, "", "") + _, err := collectPaths(bfs, "") if err != nil { return err } @@ -117,111 +113,35 @@ func TestWalkRootMappingFs(t *testing.T) { }) } -func skipSymlink() bool { - if runtime.GOOS != "windows" { - return false - } - if os.Getenv("GITHUB_ACTION") != "" { - // TODO(bep) figure out why this fails on GitHub Actions. - return true - } - return os.Getenv("CI") == "" -} - -func TestWalkSymbolicLink(t *testing.T) { - if skipSymlink() { - t.Skip("Skip; os.Symlink needs administrator rights on Windows") - } - c := qt.New(t) - workDir, clean, err := htesting.CreateTempDir(Os, "hugo-walk-sym") - c.Assert(err, qt.IsNil) - defer clean() - wd, _ := os.Getwd() - defer func() { - os.Chdir(wd) - }() - - fs := NewBaseFileDecorator(Os) - - blogDir := filepath.Join(workDir, "blog") - docsDir := filepath.Join(workDir, "docs") - blogReal := filepath.Join(blogDir, "real") - blogRealSub := filepath.Join(blogReal, "sub") - c.Assert(os.MkdirAll(blogRealSub, 0777), qt.IsNil) - c.Assert(os.MkdirAll(docsDir, 0777), qt.IsNil) - afero.WriteFile(fs, filepath.Join(blogRealSub, "a.txt"), []byte("content"), 0777) - afero.WriteFile(fs, filepath.Join(docsDir, "b.txt"), []byte("content"), 0777) - - os.Chdir(blogDir) - c.Assert(os.Symlink("real", "symlinked"), qt.IsNil) - os.Chdir(blogReal) - c.Assert(os.Symlink("../real", "cyclic"), qt.IsNil) - os.Chdir(docsDir) - c.Assert(os.Symlink("../blog/real/cyclic", "docsreal"), qt.IsNil) - - t.Run("OS Fs", func(t *testing.T) { - c := qt.New(t) - - names, err := collectFilenames(fs, workDir, workDir) - c.Assert(err, qt.IsNil) - - c.Assert(names, qt.DeepEquals, []string{"blog/real/sub/a.txt", "blog/symlinked/sub/a.txt", "docs/b.txt"}) - }) - - t.Run("BasePath Fs", func(t *testing.T) { - c := qt.New(t) - - docsFs := afero.NewBasePathFs(fs, docsDir) - - names, err := collectFilenames(docsFs, "", "") - c.Assert(err, qt.IsNil) - - // Note: the docsreal folder is considered cyclic when walking from the root, but this works. - c.Assert(names, qt.DeepEquals, []string{"b.txt", "docsreal/sub/a.txt"}) - }) -} - -func collectFilenames(fs afero.Fs, base, root string) ([]string, error) { +func collectPaths(fs afero.Fs, root string) ([]string, error) { var names []string - walkFn := func(path string, info FileMetaInfo, err error) error { - if err != nil { - return err - } - + walkFn := func(path string, info FileMetaInfo) error { if info.IsDir() { return nil } - - filename := info.Meta().Path - filename = filepath.ToSlash(filename) - - names = append(names, filename) + names = append(names, info.Meta().PathInfo.Path()) return nil } - w := NewWalkway(WalkwayConfig{Fs: fs, BasePath: base, Root: root, WalkFn: walkFn}) + w := NewWalkway(WalkwayConfig{Fs: fs, Root: root, WalkFn: walkFn, SortDirEntries: true, FailOnNotExist: true}) err := w.Walk() return names, err } -func collectFileinfos(fs afero.Fs, base, root string) ([]FileMetaInfo, error) { +func collectFileinfos(fs afero.Fs, root string) ([]FileMetaInfo, error) { var fis []FileMetaInfo - walkFn := func(path string, info FileMetaInfo, err error) error { - if err != nil { - return err - } - + walkFn := func(path string, info FileMetaInfo) error { fis = append(fis, info) return nil } - w := NewWalkway(WalkwayConfig{Fs: fs, BasePath: base, Root: root, WalkFn: walkFn}) + w := NewWalkway(WalkwayConfig{Fs: fs, Root: root, WalkFn: walkFn, SortDirEntries: true, FailOnNotExist: true}) err := w.Walk() @@ -235,7 +155,7 @@ func BenchmarkWalk(b *testing.B) { writeFiles := func(dir string, numfiles int) { for i := 0; i < numfiles; i++ { filename := filepath.Join(dir, fmt.Sprintf("file%d.txt", i)) - c.Assert(afero.WriteFile(fs, filename, []byte("content"), 0777), qt.IsNil) + c.Assert(afero.WriteFile(fs, filename, []byte("content"), 0o777), qt.IsNil) } } @@ -249,10 +169,7 @@ func BenchmarkWalk(b *testing.B) { writeFiles("root/l1_2/l2_1", numFilesPerDir) writeFiles("root/l1_3", numFilesPerDir) - walkFn := func(path string, info FileMetaInfo, err error) error { - if err != nil { - return err - } + walkFn := func(path string, info FileMetaInfo) error { if info.IsDir() { return nil } |