Add test suite

[SleeplessByte]

I prefer using something like ava because it's nice for our needs (runs everything randomly and concurrently by default, and requires us to compile it first, which is the first step of the code).

We should:

• test inputs (finding input files) and their flags
• test outputs (that the output files are correctly created)
• for each exercise:
  • test at least 1 optimal solution
  • try to test hitting each path

Tracking progress:

• two-fer
  • smoke (inline each status once)
  • snapshot (0-20 fixture snapshot per status)
  • specific path tests
• resistor-color
  • smoke (inline each status once)
  • snapshot (0-20 fixture snapshot per status)
  • specific path tests
• resistor-color-duo
  • smoke (inline each status once)
  • snapshot (0-20 fixture snapshot per status)
  • specific path tests
• gigasecond
  • smoke (inline each status once)
  • snapshot (0-20 fixture snapshot per status)
  • specific path tests

[tejasbubane]

I have not used Typescript much, so excuse me if this is incorrect. But I feel we should stick to Jest for following reasons:

• Jest also runs everything randomly and concurrently by default.
• Having same tooling with the JS track repo (apart from typescript) should be better. All decisions (test, eslint rules, code-conventions, etc.) apply to both repos - less bikeshedding.
• I hope compiling can be integrated in Jest as well (typescript-jest integration).

[SleeplessByte]

@tejasbubane yes, I'm fine with jest for the same reasons as I'm fine with ava (hence me saying "something like ava"). I think your reasoning makes sense. We should also use jest in the typescript and typescript-analyzer repositories.

I hope compiling can be integrated in Jest as well (typescript-jest integration).

As long as we don't use ts-node I'm good with that. I'm also okay with stripping types via babel, but that has to be a decision we make (to use babel instead of the tsc to to transpilation).

[tejasbubane]

Are fixtures used in any way right now? If yes where? If not whats the plan with them?

[SleeplessByte]

Are fixtures used in any way right now? If yes where? If not whats the plan with them?

Yes, these are the first 500 submissions to the exercise on the track. They can be used locally to run:

• batch-runner
• statistics

I use it to manually check if the right response is generated for an analyzer:

1. write the analyzer according to the optimal and minimal solution
2. inspect a few random solutions
3. make sure the analyzer works with that
4. run batch (all of them)
5. inspect at least 20 random exercises
6. repeat

The plan is to incorporate them in the tests to automatically do that. Basically, as per above, a test could tag a sub-set with "exercise x, y and z should have output a, b, c"

[SleeplessByte]

@depsir I see you've removed your comment. If you don't mind I'd still like to respond to it because it might help other wondering the same thing:

[...] any reason why not using ts-jest? [...]

ts-jest was only a necessity when ts-node was still a real solution. It provided direct integration with compilation of typescript when running jest (and indeed also provide --watch support).

Since babel 7 and TypeScript 3+, Microsoft has been working to help people integrate their TypeScript tooling with their JavaScript tooling, among other things enabling you to "compile" typescript via Babel (7+), essentially by stripping away the type information nodes. This allows us to rely on babel for transpilation ánd typescript for type tooling instead of:

Source TypeScript > compilation (tsc/ts-node) > JavaScript > transpilation (babel) >  Output

Facebook no longer recommends ts-jest and I wouldn't be surprised if it will be deprecated in the (near) future (even though the author says it will prevail)

[SleeplessByte]

There is now a document on 🌫 smoke tests and 📸 snapshot tests.