Proposal to allow files to pass through dest without writing them to disk.

[doowb]

@jonschlinkert and I have been trying to figure out a way to pipe files through dest but not actually write the files to disk or create the directories.

The goal is to ensure the file properties are updated using the same logic from prepare-write, then use the properties in another stream plugin. Our specific use case is to create a directory tree from the destination file paths.

We've come up with this solution, which uses a file.writeFile property to skip creating the destination directory and return early in write-contents when set to false.

Will you accept a pull request with this change, or suggest another way we could accomplish the same thing?

Thanks.

[jonschlinkert]

fwiw we searched around quite a bit for other solutions/ideas before coming to this solution.

I can't think of another way of being able to emulate the file tree - exactly as it would be if it were processed by gulp.dest(), but without actually writing the files. Every other way we came up with had some kind of drawback.

Edit: to save you some clicking, here are the proposed commits mentioned by @doowb.

[phated]

3 ideas come to mind.

• An option for dest called write to mirror read on src.
• A separate method that just prepares the file.
• (my favorite currently) Split the prepare logic into a module that exposes a stream and use that internally in vinyl-fs.

The reason I like the last one is that other vinyl adapters could use it to emulate our behavior.

[jonschlinkert]

The reason I like the last one is that other vinyl adapters could use it to emulate our behavior.

I love that idea. It might resolve/allow other things we've wanted to do.

[doowb]

Split the prepare logic into a module that exposes a stream and use that internally in vinyl-fs.

I think this is my favorite also. I was thinking vinyl-prepare or vinyl-prewrite for the name.

Also, I think that the mkdirp call should be moved to write-contents. I'll set the dirMode on the file so it can be used like this:

var writeFolder = path.dirname(file.path);
fo.mkdirp(writeFolder, file.dirMode, onMkdirp);

If this sounds good, I'll put a repo and PR together.

[yocontra]

Why don't you write your own stream that uses the prepare function on the file and not use dest? Would only be 3 lines of code - I don't think this is common enough of a case to warrant adding an option to dest for it, but we can make sure that the prepare logic is accessible so you can use it in your own streams.

[jonschlinkert]

Why don't you write your own stream that uses the prepare function

I think that's what @phated is suggesting. I agree, this would be the way to go.

I don't think this is common enough of a case to warrant adding an option to dest for it

I also agree with this. plus, adding an option to dest wouldn't be useful for this unless it was a function, and that seems heavy for this.

I don't think this is common enough of a case

Lol yeah I suspect it's pretty rare considering it's not possible atm. jk

we can make sure that the prepare logic is accessible so you can use it in your own streams.

That would be awesome. I'd love to avoid having to duplicate the code and keep parity.

The last thing we want to do is add feature bloat, but I know we'll get a lot of use out of this. I think other devs will find it useful too, once it's exposed. With only a few lines of code it opens up the possibility to do other interesting things

[phated]

Interesting side note that I didn't want to type on my phone last night: in refactoring the tests, I've found a lot of inconsistencies with the prepareWrite module and would like to have it fully unit tested apart from gulp.dest. So this issue comes at an opportune time. We also want to reach parity with the gulp.src method (ref #191) so a module could expose a src/dest prepare stream. As soon as I finish this test refactor, I'll look into splitting it apart.

[erikkemperman]

Kind of related:

I looked into the two skipped tests in test/dest.js. The error they expect should come from prepareWrite (https://github.com/gulpjs/vinyl-fs/blob/master/lib/prepare-write.js#L32) but currently we don't get that far: through2 never gets around to calling the saveFile function (https://github.com/gulpjs/vinyl-fs/blob/master/lib/dest/index.js#L20) because nothing is ever piped in to dest by these tests.

But actually prepareWrite seems like the wrong place to handle arguments/options anyway, so maybe this is part of a bigger issue.

I like the proposed refactor of prepareWrite, if I understand it correctly; change it to a wrapping stream, much like is done for wrap-with-vinyl-file in src, and then make that (those) available for external use.

Maybe make a point of cleaning up options processing as part of the same effort? I have some half-baked work in this direction, but got distracted by diving into the value-or-function helper. Let me know if there's anything you think I might contribute.

(edited to fix typo)

[phated]

Package scaffolded at https://github.com/gulpjs/vinyl-prepare

[doowb]

@phated I have some code I started on and can send a PR. I'll update the tests with what you just merged in and add specific prepare tests. I can also include prepare.src which I think will handle the options and wrap-with-vinyl-file functionality.

[phated]

@doowb awesome, I didn't see where you were working on stuff so I just pushed the logic split from vinyl-fs (no tests). I tested it against vinyl-fs but needed to add pumpify (see https://github.com/gulpjs/vinyl-fs/tree/prepare-stream)

[doowb]

I didn't see where you were working on stuff

np... i didn't push anything up, but basically made the same changes as you.

but needed to add pumpify

I did something similar, but used duplexify to match what the sourcemaps part was doing.

When I was adding tests (for high watermark and slow streams), I needed to use sink around prepare. Do you think sink should be externalized so it can be used in vinyl-prepare?

[phated]

Due to this always being a Transform stream, the tests will have to always use from and to or concat. The reason vinyl-fs sinks is due to a bad pattern we support (sometimes a Transform stream/sometimes a writeable)

[doowb]

That makes sense.
Let me know if there's anything I can do.

[phated]

I'd like to see the tests you have, since i don't have any yet

[doowb]

PR sent to vinyl-prepare. We can discuss there and I can make changes and additions as needed.

[phated]

Basic support for this has landed in master but I still need to some more work on this and get the libraries to 1.0 before the vinyl-fs@3 release.