Support to Run VSCode unit tests in a CI setup

[egamma]

[bpasero]

Since this is VSCode release independent, I can still finish this one during the week. I have the necessary code ready for running tests in Travis CI and just need to make sure it is documented properly.

[bpasero]

I have documented this here now: https://github.com/Microsoft/vscode-docs/blob/vnext/docs/extensions/testing-extensions.md#running-tests-automatically-on-travis-ci-build-machines

@gregvanl fyi. I hope its ok to make the example with Travis CI.

@egamma @lukehoban I think it would be cool if we enable this for Go so that people have a live example. I have setup https://github.com/bpasero/ci-testing as showcase (see https://travis-ci.org/bpasero/ci-testing). There are some gotchas:

• since we have not released 0.10.4 (December) yet, I am hardcoding the VS Code download location in the .travis.yml file: https://github.com/bpasero/ci-testing/blob/master/.travis.yml#L10 and https://github.com/bpasero/ci-testing/blob/master/.travis.yml#L14
• since we have not released the associated vscode npm module for 0.10.4 yet, I have pushed it to the npm tag next: https://github.com/bpasero/ci-testing/blob/master/package.json#L30
• once the December release is out, you don't have to set environment variables, and 0.10.4 will be picked by default

@dbaeumer also fyi since you have asked for this. there is still a gap that we currently do not have published stable links for nightly releases of VS Code. But I think once we change our update server to produce something like a stable link that always points to latest we could enable CI tests also for extensions that want to test pre-release versions of VS Code.

[lukehoban]

@bpasero I've tried this out for the Go extension. It seems to be working.

• Travis: https://travis-ci.org/Microsoft/vscode-go/builds
• PR for vscode-go with the Travis configuration: Setup Travis CI vscode-go#161

Any feedback on whether these looks correct? Also - is it safe to adopt vscode: next for an extension that is still being used on VS Code 0.10.3? I assume there is no problem with that?

One thing I did noticed while setting this up - running these tests locally is a little rough right now - three issues:

• Always downloads VSCode - there is no caching.
• Files are downloaded into a local .vscode-test folder which must be added to all ignore files
• The inability to run tests from the command line while other instances of Code are open

[bpasero]

@lukehoban very cool looks perfect!

Any feedback on whether these looks correct? Also - is it safe to adopt vscode: next for an extension > that is still being used on VS Code 0.10.3? I assume there is no problem with that?

We are planning on decoupling the vscode node module so that the API part moves out and the tooling part remains. Once we do that, the only thing that you will need to set in the package.json to identify the target VSCode version is via the engine field (similar to how a node module does it for node). What remains in the vscode node module is the tooling for compile and test and I would eagerly publish releases there as long as I am not breaking existing extensions in a version form that is compatible (e.g. minor version increments) so that you can just keep 0.10.x to receive updates properly.

For now, please do not publish the extension with next. I suggest to wait with the PR until we have released end of week and then change it back to 0.10.x. Once we have released you also don't need to set the custom download URL.

One thing I did noticed while setting this up - running these tests locally is a little rough right now - three issues

Yes, we are not in a good state to run the tests locally from the command line. There are still issues and you listed them all. The test script is optimised for running in a build machine that just dismisses any storage on disk after the run.