Build enters an infinite loop

[aleab]

• Operating System: Windows 10
• Node Version: v15.12.0
• NPM Version: 7.6.3
• webpack Version: 5.31.2
• copy-webpack-plugin Version: 8.1.1

Expected Behavior

• npm run build -- --watch should build the project only once.
• When a file is changed, the project should be rebuilt only once.

Actual Behavior

Multiple odd things happen.

• npm run build -- --watch builds the project twice.
  
  This does not happen with npm run build.
  This does not happen if I remove copy-webpack-plugin from webpack's plugins.

• The build enters an infinite loop if you do either one of these two things:

  1. 　
     • npm run build -- --watch
     • Modify any ./some-file*.txt or ./index.js
  2. 　
     • Uncomment the first and only line of code inside ./index.js
     • npm run build -- --watch

  Everything works as expected if I remove copy-webpack-plugin from webpack's plugins.

Code

webpack.config.js

const path = require('path');
const CopyWebpackPlugin = require('copy-webpack-plugin');

module.exports = function getWebpackConfig(env, argv) {
    const mode = argv.nodeEnv || argv.mode || 'production';
    const copyPlugin = new CopyWebpackPlugin({
        patterns: [
            { from: './some-file*.txt', },
        ]
    });

    return {
        mode,
        entry: './index.js',
        output: {
            filename: 'main.js',
            path: path.resolve(__dirname, 'dist'),
        },
        plugins: [ copyPlugin ],
    };
}

How Do We Reproduce?

Test repo: https://github.com/aleab/copy-webpack-plugin_build-loop

[alexander-akait]

Here problems with glob and watching (to he honesty no problems), when you use glob, we add base directory for watching (in your example you use './some-file*.txt', after as glob do own job you have ['/absolute/path/to/some-file.0.txt', '/absolute/path/to/some-file.1.txt', '/absolute/path/to/some-file.2.txt'] paths, so we add /absolute/path/to/ directory for watching, because we need watch base directory, we can't know future files, you can add more files or delete them in future), otherwise watching with glob will be broken. Even more ./some-file*.txt can be directory on some file systems.

You can fix it:

• moving files in nested directory
• copy them by files

it is limitation for using glob

[alexander-akait]

In theory we add watchFilter: () => {} option, so you can filter what you need watching, but adding new files or deleting them will not work

[aleab]

So the problem is that the base directory of the globbed files is the root of the project; and since that is being added to the watch list, basically any change to any file or folder inside the root and its subdirectories will trigger a rebuild (in this case I assume ./dist). Am I understanding this correctly?

In that case shouldn't watchOptions.ignored in webpack.config.js allow me to ignore at least some files/directories?

EDIT: watchOptions.ignored works. I was having issues with Windows paths when I first tried that.

In any case, a watchFilter: () => {} option would probably be a better solution, at least for my use case.

[alexander-akait]

So the problem is that the base directory of the globbed files is the root of the project; and since that is being added to the watch list, basically any change to any file or folder inside the root and its subdirectories will trigger a rebuild (in this case I assume ./dist). Am I understanding this correctly?

Yes

EDIT: watchOptions.ignored works. I was having issues with Windows paths when I first tried that.
In any case, a watchFilter: () => {} option would probably be a better solution, at least for my use case.

Why do not use watchOptions.ignored if it is working?

[aleab]

I mean, it is an acceptable working solution, but I have many files in the root directory that I would have to manually add to the ignored list and I don't really like that, it just seems unclean.
The best solution for my use case is to either have a way to tell this plugin what to add to the watch list (i.e. to not add the context directory), or to manually resolve the glob pattern beforehand inside webpack.config.js (which is what I'll be doing).

new CopyWebpackPlugin({
    patterns: [
        ...require('glob').sync('./some-file*.txt')
    ],
});

---

This is definitely outside the scope of this plugin, but the cleanest solution would be a way of telling the watcher to only report changes to files that match the original glob pattern within the context directory.

That is also, in my opinion, what the user would expect.
If I tell copy-wepback-plugin to copy ./some-file*.txt I would expect the plugin to tell the compiler that those files are meaningful and should be watched for changes. What I wouldn't expect is the side effect of the compiler thinking that everything else inside ./ is also meaningful.

I understand that watching the directory is necessary, otherwise you'd miss potential new files/directories matching the glob pattern, but I think that there needs to be a way of telling the watcher/compiler that "Yes, you do need to watch the directory, but don't forget why you are watching it: within that directory the only meaningful files that should trigger a rebuild are those that match the pattern(s)" without having to explicitly add the files that do not match the pattern(s) to the ignore list.

[alexander-akait]

I understand that watching the directory is necessary, otherwise you'd miss potential new files/directories matching the glob pattern, but I think that there needs to be a way of telling the watcher/compiler that "Yes, you do need to watch the directory, but don't forget why you are watching it: within that directory the only meaningful files that should trigger a rebuild are those that match the pattern(s)" without having to explicitly add the files that do not match the pattern(s) to the ignore list.

Yes, but it is impossible/hard to implement, we need evaluate glob before running, but it is very reduce performance, even glob doesn't know which files you will get at the end, so any potential changes in directory can be resolved by glob or not, I don't have idea(s) how it is possible to solve without perf problems

[aleab]

perf problems

Yeah, that's what I thought.
I'm not familiar enough with the code of this plugin, webpack or watchpack to know if and how something can be implemented.

However, the first thing that came to my mind after quickly reading some of the code, was to have something similar to watchOptions.ignored that's handled internally at the level of the compiler/compilation. (This is obviously a change that would involve at the very least both webpack and watchpack repos, so it's non trivial and it may be a terrible idea, I don't know...)

You'd have a Map<string, Set<string>> somewhere, which is basically a map of context directory -> Set of glob patterns (or even Set of regexs by using glob-to-regex, which is already used in watchpack to filter out the ignored files); each plugin can do something like compilation.addContextDependency(directory, globPattern) that would add that glob pattern to the appropriate set (instead of doing compilation.contextDependencies.add(directory)).

Then you would pass the appropriate Set<string> | RegExp down to watchpack's DirectoryWatcher (here for example) and the watcher would take care of filtering out anything that doesn't match the globs/regex the same way it is currently filtering out anything that matches watchOptions.ignored (here for example).

As far as performance goes, I think there are currently two scenarios:

1. The user decides to use watchOptions.ignored to ignore all the files/directories that would otherwise trigger a number of unnecessary re-compilations.
2. The user decides to not use watchOptions.ignored and suffer the unnecessary re-compilations.

• In the first case, the performance loss would be the same, because you'd use the Set<string> | RegExp instead of watchOptions.ignored to filter out the same files.
• In the second case, it's a matter of comparing the performance hit of an unnecessary re-compilation (which is what happens currently) with the performance hit of testing a bunch of file names/paths against a regex.
  If no files are changed inside any certain watched directory, then there is obviously no performance loss.

[alexander-akait]

glob-to-regex supported only limited numbers of globs (popular) and here is problem, maybe we will improve in future, also fast-glob have more options https://github.com/mrmlnc/fast-glob#options-3 we should take this into account, that's why I said that it is rather difficult, yes it is possible, but honestly it seems to me that nobody want to help us with this

[corsik]

I faced the same problem.

  watchOptions: {
    ignored: path.resolve("dist"),
  },

Helps solve the problem, but then the copy only happens on the first build

[k4mr4n]

I've recently upgraded my webpack v4 to v5 and in consequence I had to upgrade this plugin too and after that Im getting infinite loop on dev environment after a file save.
this is my configuration:

new CopyWebpackPlugin({
        patterns: [
            { from: 'src/assets' },
        ]
    });

which tries to copy all the image and svg files from the assets folder.
I tried using glob but no change. also tried using ignored in the watchOptions but again no chance.

Can you please help me figure out what needs to be done in-order to fix this issue?

[alexander-akait]

@k4mr4n Can you provide webpack configuration?

[k4mr4n]

@alexander-akait Finally, I managed to fix it by adding the assets in the ignored path.
At first I was trying to add the build directory to the ignored and it wasn't working.

so at the end:

watchOptions: {
       ...,
      ignored: /node_modules|assets/,
 },

webpack.config:

plugins: [
...,
new CopyWebpackPlugin({
      patterns: [{ from: 'src/assets' }],
 }),
]

[kboshold]

We also get an endless loop here when CopyWebPack is used.

[Ponynjaa]

I've recently upgraded my webpack v4 to v5 and in consequence I had to upgrade this plugin too and after that Im getting infinite loop on dev environment after a file save. this is my configuration:

new CopyWebpackPlugin({
        patterns: [
            { from: 'src/assets' },
        ]
    });

which tries to copy all the image and svg files from the assets folder. I tried using glob but no change. also tried using ignored in the watchOptions but again no chance.

Can you please help me figure out what needs to be done in-order to fix this issue?

We have the same issue with

new CopyWebpackPlugin({
        patterns: [
            {
                from: 'static',
                to: '.' ,
                force: true,
                transform: (content, absoluteFrom) => {
			const relPath = path.relative(config.from, absoluteFrom);
			const outPath = path.normalize(path.join(config.to, relPath));

			return getContent(outPath);
		}
            },
        ]
    });

and

watchOptions: {
    ...,
    ignored: /node_modules/
}

When adding static to ignored of watchOptions like this

watchOptions: {
    ...,
    ignored: /node_modules|static/
}

it doesnt rebuild infinitely anymore but the static folder isnt watched anymore. So whenever we change something within the static folder (e.g. we have css files inside the static folder) we would have to restart the whole watch process for it to be copied over into the build directory again (which in our case is really annoying given the fact we have css files in that directory).
So I dont think this is "Nice to have" but a bug that should be resolved asap.

[alexander-akait]

@Ponynjaa Why do you consider this as a bug, if you want static directory, webpack rebuilds then you changed something

[Ponynjaa]

@alexander-akait it does, but it enters an infinite loop when you change something in a static directory. In the meantime we found out, that the problem seems to be, that when you use poll with a too small number (smaller than the actual time it takes to run one build cycle) it just keeps triggering a new build. When we changed poll to a higher number, that is USUALLY longer than the actual build time this problem doesn't occur anymore. So yes I think this is a bug and should be fixed ASAP.

Our config that triggers the infinite loop looks like this:

watchOptions: {
	aggregateTimeout: 500,
	poll: 500,
	ignored: /node_modules/
}

Copy-Webpack-Plugin config:

new CopyWebpackPlugin({
    patterns: [
        {
            from: 'static',
            to: '.' ,
            force: true,
            transform: (content, absoluteFrom) => {
                const relPath = path.relative(config.from, absoluteFrom);
                const outPath = path.normalize(path.join(config.to, relPath));

                return getContent(outPath);
            }
        },
    ]
});

If you change something in static/ it triggers the infinite rebuild loop.

Thanks :)

[josecarlosrx]

I was having the same problem. For me the solution was to add this to webpack.config.js:

watchOptions: { ignored: path.resolve(__dirname, "dist") },

I guess watchOptions.ignored has to be the same as output.path.

[alexander-akait]

@josecarlosrx Do you want to improve our docs and add this note there?

[josecarlosrx]

@alexander-akait I would like to but I'm experiencing the problem noted by @corsik, only the first build is copied.

Edit:

Not trying to discourage this project but a simple solution, until this is patched, could be:

const path = require("node:path");
const fsp = require("node:fs/promises");

class CopyPlugin {
  apply(compiler) {
    compiler.hooks.afterEmit.tapAsync(
      "CopyPlugin",
      async (compilation, callback) => {
        const dir = compilation.options.output.path;
        const filename = "index.js";

        const src = path.resolve(dir, filename);

        const destPaths = [
          path.resolve(__dirname, "dir1", filename),
          path.resolve(
            __dirname,
            "dir2",
            filename
          ),
        ];

        const promises = destPaths.map(async (dest) => {
          const dir = path.dirname(dest);

          await fsp.mkdir(dir, { recursive: true });

          await fsp.copyFile(src, dest);
        });

        await Promise.all(promises);

        callback();
      }
    );
  }
}

webpack.config.js:

module.exports = {
  entry: "./src/index.js",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "index.js",
  },
  plugins: [new CopyPlugin()],
};