Document the lockfile

[zkochan]

close #188

[ghost]

@zkochan suggested I might comment here regarding some questions on using hash based version ids, lock files, and machine level (global) installs.

Fwiw, I've been made aware I'm not that good at paraphrasing things for easy consumption on this type of forum. My apologies in advance if any of this seems overly "wordy".

In Summary (TL;DR;)

• Was reusing yarn.lock's format considered?
• Potential issues, between two machines, caused be versions of dependencies used, when running the install lifecycle, while installing modules to machine level stores.

Reinventing the wheel?

I was wondering if simply using yarn's lock format had been considered? I cursorily looked at both specs and on the surface it appears they're both intending to solve the same fundamental problem in the same fundamental way, by atomically storing: a flat version dependency graph using content hashes as version identifiers; the dependency version-set specifiers used when obtaining the content and their mapping's to the semantic version and hash version; and the content's source registry? Please correct me if I'm in error.

The pro of this, and the impetus for the question, is it would ease developer workflows where the graph has previously been computed and stored with the source in a SCCS, for both yarn and pnpm/ied users.

The con is that pnpm/ied is then coupled to what yarn is doing. For example, I was dismayed that they chose a proprietary serialization encoding rather than yaml or even json; it may have been related to trying to keep the file human-readable, but still machine parse-able and small; they get quite large and I wouldn't be surprised if the proposed lock files here wouldn't be similarly sized. However this to me seems like another opportunity for collaboration, and it's hard to imagine a great deal of backwards breaking "innovation" occurring to the file structure and encoding given its purpose.

I do see the value of being able to simply open one in an editor and find what I'm looking for, and it would be nice to that end if it used some common encoding, but part of me would prefer doing that through the command line tool (as the files are usually large), where I could just type something like (btw what would the merged effort's tool be named?) pnpmied list somemod and it would nicely format outputting all the version info for it, maybe along with just a double check that's actually what's installed, along with the timestamp of when it was locked (which I think would be an enhancement to whats currently captured and stored), and it would both fuzzy match if no exact name found, and allow for regex's in specifying the name; I'd love that! Just a thought.

Timing of computing hash version of globally installed modules

A question I have, not directly related to the lock format itself, is exactly when the version hash of a module installed at the machine level is computed. I believe certain activities that can occur while running install lifecycle steps rely on other modules, and are therefore potentially deterministic in behavior and output by the versions of the modules being depended upon during installation, and today those dependency versions can change from local install to local install. I also wonder if certain installs alter their output as a function of what's depending on them (a bad thing to be doing in my opinion). I honestly have no direct measure of how frequently altered output as a consequence of either of those actually occurs in practice, or if at all, I just know it's technically possible, and if the desire is to "nail it" with respect to version locking to ensure determinism, it might need to be considered.

To me, this is just one of those 'sucky' problems that might happen once in a blue moon, and if it did would 'break' things in a very subtle, hard to initially locate kind of way, and requires a fair amount of effort to completely deal with. For example, in this case, to mitigate problems as a consequence of output coupled to versions of dependencies used during installation, we'd want to then store for potential re-use on other machines, and include as part of the hash input, what hashed dependencies where used for installation, and then integrate all of that when rendering the physical tree; i.e. if the module hash in the lock file didn't exist, we'd have to install at the machine level, using the dependency versions originally used to generate that hash, and NOT those from the lock file been rendered.

For the other issue, output coupled to the dependent of what's being installed?... uhm.... punt to documentation that it's not supported, and discourage making modules behave that way, possible with a recommended alternate pattern to address how to achieve the same ends; although I'd wait to do any of that until it's actually encountered and can be analyzed.

[ghost]

I deleted a comment unnecessarily. Regarding the effect of dependencies used for a machine install, I think I over complicated a potential way to handle. The hash-versions of the dependencies used just need to be part of the input to hash(); i.e. h-ver-mod = hash(h-ver-deps[] + mod-content). The dependencies themselves are just the ones already in the lock file.

[billiegoose]

Responding (just) to the TL;DR:

Was reusing yarn.lock's format considered?

Yes! See #188 (comment)