feat(go): report project .go sources as project-graph inputs

[jonpoole-fluidstack]

Apologies, bit of a bug from my last PR that slipped by me. I didn't realise this was being cached based solely on the go.mod. This PR attempts to find all .go files that belong in that project and add these as inputs to the project-graph, as that's what we're basing the project inference on.

[milesj]

@jonpoole-fluidstack Wait so this adds every .go file as an input? This will basically constantly wipe out the workspace graph cache, resulting in go list basically running on every invocation right?

[jonpoole-fluidstack]

@jonpoole-fluidstack Wait so this adds every .go file as an input? This will basically constantly wipe out the workspace graph cache, resulting in go list basically running on every invocation right?

😢 What I was going for is to invalidate the cache per-project so if you changed a source file under there it would list the deps, but I realise this cached as a single call per workspace. I'm not sure what a good option here is.

I think to represent the Go semantics we need to re-run when the imports change, which means busting the cache when go sources change. Shelling out to Go per project is pretty slow though. You can do one list upfront and get the full project graph pretty quickly. In the gRPC go project:

➜  grpc-go git:(master) time go list -f "{{.ImportPath}} -> {{.Imports}}" ./... > /dev/null
go list -f "{{.ImportPath}} -> {{.Imports}}" ./... > /dev/null  0.26s user 0.27s system 124% cpu 0.429 total

So around 200ms - 300ms wall time for a fresh re-run on each moon call. If we don't care about build tags etc. I'm sure doing this in process in with a rust Go parser would be even faster. What do you think? I think where I've left you is not great because it's caching things it shouldn't (sorry D:) so maybe you'd prefer to roll back?

[milesj]

Yeah right now the graph is cached as a whole. I would like to cache it per-node (project) eventually, but that would basically require a rewrite I think.

think where I've left you is not great because it's caching things it shouldn't (sorry D:) so maybe you'd prefer to roll back?

It's just go.mod right? So that's a little bit better than nothing, otherwise the graph will always be cached. Do we include go.sum? Maybe include that also.

I'm sure doing this in process in with a rust Go parser would be even faster.

It is possible to glob + parse every file. May actually be faster than child process calls. Is an option if the parsing is actually fast and accurate.

[jonpoole-fluidstack]

Do we include go.sum? Maybe include that also.
No not at the moment but can do.

It is possible to glob + parse every file. May actually be faster than child process calls. Is an option if the parsing is actually fast and accurate.

I think go list would work in all but the biggest monorepos, but I had a bit of a look at what Go actually does. The syntax for imports is very simple, but they do have a lightweight parser that loops through the file just looking for package ... and import ... lines. It stops reading as soon as it matches some other declaration. This is invoked on every go build/test/list, but unlike moon, this also walks transitive deps. If we wanted to hand roll something, we could probably test it against the go list ground truth on some sizeable public projects to make sure it's correct.

We could probably drop the fiddly build tag semantics, but we may end up with edges that aren't technically needed based on the current build tags/platform. I don't think an extra edge once in a while is likely to be a real problem. You can't dynamically change the project graph in .yml anyway.

What would be nice down the line would be to infer the project sources as well. We're currently just globbing for **/*.go which isn't technically correct. You can have .cgo and .c sources but also just arbitrary files embedded into the build. I think re-creating all the go directive parsing to achieve this in a hand rolled parser would be beyond what you'd want to maintain, but go list can output this per package for you no problem. Not sure what you think about that.

I would like to cache it per-node (project) eventually, but that would basically require a rewrite I think.

Yeah, ideally, we'd start from the task we're asking of moon, and lazily populate the project edges as we need them. The PDK would need to expose a new function to allow us to finalise a project, and call that when it needs the edge info.

Maybe the roadmap for this is:

1. One repo wide go list with cache busting on .go files
2. New PDK function to allow incrementally building the project graph lazily
3. go list per project now cached

And then we can figure out if we want to hand roll a parser to pull out the Go imports. If we wanted to get fancy with it, we could look into some wasm/tranpilation here to get access to bits of the Go SDK but this is all private internal code that you're not really meant to access. Even Go tool shell out to go list.