Article
Technical
Depp: A fast unused and duplicate dependency checker
11/3/2021
··
profile
Recently we ran into a couple of bugs on our main fairly large monorepo which were caused my the same package existing twice but having different versions. This made me search for some dependency checking tools, and while there are existing ones none of them really had good monorepo support.
 
So I decided to make one, this is a brief about how I made it and also how to use it.
 
If you want to just try the tool without hearing ramble about it, feel free to checkout the repo

Basic Usage and Features

Install

npm install -g depp-installer depp --help # Will should all avaliable options

Usage

Default Config

depp
 
Will check only typescript (.ts, .tsx) files in your root folder and all its children. It will also read the packages from then root package.json but also any package.json inside child folder (It supports mono repositories by default)
 
It will show unused packages, unused @type packages and duplicate packages with different versions
 
It will also generate a temporary html report file and open it in your browser. This file will look something like the following

Some Major Flags

  • --js Will enable checking js files
  • --dev Will enable checking dev dependencies (this is not very accurate)
  • --report Will save the report to .depp/report.md
  • --externals Can use this to external certain packages, useful if the build fails by default
  • --ignore-namespace Can ignore namspaced internal packages using thing, good for ignore @monorepo packages
  • --show-versions Will explicitly print the versions of duplicate packages in console
 
This is not an exhaustive list of all flags, for that run depp --help
 
Example advance usage is
depp -e mobx -e magic-sdk -e domain -e @daybrush/utils -e yjs -e constants -e ws -v -in @editor -in @server -j -e perf_hooks --report
 

How it was built

Just briefly wanted to talk about how this built and how it works, of course the entire codebase is open-source so feel free to poke around yourself
 
So this package works by wrapping around the esbuild go api to add this extra functionality, so yes technically we built your whole code base to parse what dependencies you are using. This might sound incredibly stupid, but considering esbuild speed and flexibility, it is still significantly quicker than most solutions written in javascript
 
I realize this is not a great benchmark but considering most other tools don't have good monorepo support it was the best I can do. I compared depp (this tool) vs depcheck (a popular javascript tool with almost 3k stars) on a previous project https://github.com/wei/socialify
 
depp 0.42s user 0.36s system 135% cpu 0.576 total depcheck 1.69s user 0.13s system 116% cpu 1.567 total
 
This tool was about 2.7x faster than depcheck while practically building the entire codebase
 
Using esbuild like this also allows me to build this tool, without writing an entire javascript/typescript parser

Setup

First we need to get the paths of all the package.json files and then all the source files,
I did this using something I built for an older project which is a modification of go standard library's Glob This modification allow me to easily find all nested package.json and source files
 
Now we can read the package.json files and get all the packages in the repository, we can store these as a hash map or dictionary to compare against in the future.
 
Finally, we can pass the source files to esbuild as entry points and have it do its magic

Callbacks

First we use Resolve and Load callbacks from the esbuild api to check for packages as they are being imported.
 
There are some caveats here tho, we don't want to go down a rabbit hole of every import. Any import from node_module we don't care about, these are generally nested imports of modules importing other modules. So we can mark these files as external
 
Another optimization we can add is, any non .ts, .tsx, .js,.jsx file can be completely ignored. This is a bit painful in go because it does not support regex lookaheads (?)
 
Now with any actual import, we can check the kind of resolve it is, whether it is a import-statement , require-call or something else we don't care about. Finally we can check if the imported path contains any of our packages

Metafile

Additionally we use the esbuild meta file to do a second pass on any import we might have missed or that slip through or resolve callbacks. The esbuild meta file contains a bunch of information about inputs and their imports which we can use to see if a given package has been used or not
 

Duplicates

While the process of finding unused packages is quite involved, finding duplicates is quite easy on the other hand. It is just about correctly inputting the parsed package.jsons and checking for duplicates and storing their versions. There is a bit more work for check unused @types packages but it mostly similar to duplicates
 
Not going to go into it much here, that said it probably constitutes its own post of how to bundle go binaries with an npm package and how to release them
 
All that said this was a fairly fun project to make in the 8 or so hours it took, and it has already been fairly useful internally. Hope you find some use in the tool itself or maybe how I built it