Add a custom hook/extension mechanism for user defined passes

[Tobski]

We've had requests from some ISVs to enable a way to hook new passes for validation and optimization without having to fork the codebase or add patches to the main codebase. Having a way to do this would allow some ISVs to more frequently update their versions of spirv-tools, by not requiring patch modifications, and instead making it more "plug'n'play". This is not a small effort, but recording it here to track its viability, and to focus discussion.

[s-perron]

Can you be move specific about the ask?

1. Users can write their own out-of-tree passes, but it does requires them to access interfaces that spriv-tools does not officially export. We have generally not exported them because we wanted to reserve the right to change them. However, they are pretty stable now.
2. User are able to register their own passes, if we do 1. However, they do not have the ability to modify the sets of passes like the legalization passes. Is the ask to have a way to insert their own pass in the middle of the optimizations passes or something simpler?

[gmitrano-activision]

To add some context here, we currently maintain our own fork of the SPIR-V tools repo because we have a few internal custom passes that we implemented in the optimizer for various reasons (primarily to address platform portability issues and add warnings relating to performance). The only reason we have to fork the upstream code and not use an upstream read-only dependency, is because the only way to add passes to the optimizer is to modify the source directly. Ideally, we'd be able to inject our own passes without needing to modify the source files from the SPIRV-Tools project.

One possible approach to enable this would be to do something similar to LLVM pass plugins (-fpass-plugin in clang) where developers can author a separate plugin DLL that's loaded by the compiler dynamically via a command line option. This allows them to isolate their logic inside the DLL and use an existing prebuilt SPIRV-Tools library without needing to build or modify the source. Of course you'd still need to include relevant headers to implement the DLL, but that could achieved by including them from an unmodified upstream copy. As far as ABI compatibility, I think it'd be fine to use the same approach as LLVM which just requires the underlying versions to be compatible between the compiled dll and the SPIRV-Tools binary loading it (meaning no heavy public interface stability requirements). This does mean that the dll technically should be recompiled every time the SPIRV-Tools binaries are updated, but at least this approach allows us to avoid forking and maintaining a modified SPIRV-Tools project.

The goal of this github issue is not to specifically request the exact implementation above. The intention here is to raise that this workflow is a pain point so we can get discussions started about how to best address it long-term especially considering that DirectX/HLSL will eventually be using SPIR-V as well.