validation: create: don't skip errors on state

[alban]

If the runtime fails to give us the state, don't just ignore this and
skip the test but ensure we print "not ok" in the TAP output.

Signed-off-by: Alban Crequy alban@kinvolk.io

[wking]

This effectively rolls back part of d8d2396 (#564), but I didn't motivate the skip then, and can't remember why I'd felt it was a good idea. I'm fine returning to to stricter handling like you've done here.

[wking]

While we're touching this line, we may want to use the appropriate spec error. In this case, a failed state call would be violating the combination of EntityOperSameContainer, QueryStateImplement, and, from the CLI spec, “The runtime MUST support the entire API defined in this specification” and “Exit code: Zero if the state was successfully written to stdout and non-zero on errors” (which should really have a MUST, I'll file a PR for that). From among those, the most important seems like QueryStateImplement. Following this pattern would give us something like:

err = specerror.NewError(specerror.QueryStateImplement, errors.New("'state' returns the state of the container"), g.Config.Version)
runtimeError, ok := err.(*specerror.Error)
if !ok {
  return util.Fatal(fmt.Errorf("cannot convert %v to a runtime-spec error", err))
}
state, err := r.State()
t.Ok(err == nil && state.ID == c.id, runtimeError.Err.Err.Error())
diagnostic = map[string]string{
  "level": runtimeError.Err.Level.String(),
  "reference": runtimeError.Err.Reference,
  "container ID" = c.id,
}
if err == nil {
  diagnostic["state ID"] = state.ID
} else {
  diagnostic["error"] = err
  if e, ok := err.(*exec.ExitError); ok && len(e.Stderr) > 0 {
    diagnostic["stderr"] = string(e.Stderr)
  }
}
t.YAML(diagnostic)

Although we could make that a bit easier by adding a helper like func NewRFCError(code Code, err error, version string) rfc2119.Error to specerror. I've filed #627 to add NewRFCError.

[liangchenye]

LGTM