Package Testing

[ikeough]

Extend the package schema to include a /test folder. In that folder will be a series of .dyn files that use the nodes in your package. The automated system for package testing will do the following:

• Download your package.
• Open each .dyn file in the /tests folder and do the following:
  • Execute
  • Check for dummy nodes.
  • Check for errors.
• Send an email to the package maintainer with the status of the package.

[mjkkirschner]

what about an assert node?

[ikeough]

@mjkkirschner Yes, we talked about this a while back. It would be useful here. BTW, this conversation is coming up because no one is paying attention to the packages that we administer, like Solar Analysis. Whenever it breaks because of an API change, we should know.

[kronz]

@riteshchandawar @monikaprabhu @jnealb

[riteshchandawar]

@ikeough @mjkkirschner Just 2 days back I discussed this with @aparajit-pratap. We have automated the Mesh Tool Kit and Dynamo Text. The next step is to Automate the remaining internal Packages and also the most used Public Packages.

I explained the below workflow to Aparajit.

Many Packages are having their Sample DYN files in Extra folder, so we can make use of them for automating their package.

The Framework will do the following stuff:

1. Download the latest version of Packages listed in one xml file. (so that we can add/remove more packages based on need )
2. It will search for DYN files in Extra folder.
3. It will execute the graph and then dump output data from all the nodes to xml.
   a. If this Package running first time then it will generate new xml for that DYN file.
   b. If this Package running second time then it will compare the newly generated xml with base xml.
4. If it fails then it will report to Package Owner.

Limitation to the above workflow is for Revit nodes in the DYN, if we want to cover everything then we have to run these DYn on their associated Revit files and the rest of the workflow will be same.

Thoughts?

-Ritesh

[jnealb]

@ikeough @riteshchandawar @mjkkirschner @kronz @monikaprabhu I am going to develop regressions that drive the Package Manager ui.

[ikeough]

@riteshchandawar We should be very clear to users about this. By adding a /test folder we're saying that "the .dyn files in this folder are to be used for testing." We have a lot of "magic" in our systems, like "if there are .dyn files in the /extra folder, we'll run them." We tend to be bad at documenting this magic anywhere and it causes confusion.

Also, the testing system should not be some Electric Commander thing unless doing so gives us huge benefits. The CI/CD team is working on Jenkins builds from any branch. That's great, except that both EC and Jenkins as administered internally will not work with github public repos. We should be able to hook when a package's public repo is changed and run the tests automatically.

[aparajit-pratap]

@ikeough for this automated system for package testing (specifically 3rd party packages), are you suggesting, downloading the packages from the package manager and testing them OR downloading the source from their repos, building and then testing?

For packages maintained by us, we are setting up each one on EC to be built from source and tested. For those that don't have unit tests we need to implement a simple test fx to do what you suggested:

Open each .dyn file in the /tests folder and do the following:
Execute
Check for dummy nodes.
Check for errors.

[ikeough]

@aparajit-pratap The first version of this would just download packages from the package manager.

In the future, I'd like to offer github support for package repositories so that our system could download and build the packages. That way we could verify whether the packages would work with Mono. I'm thinking about this because we've enabled package white-listing on Fractal, that is, the ability for users' packages to be run on our Reach instances. And those packages have to be ones that are mono-compatible.