ethtool error: "panic: "node_ethtool_ err" is not a valid metric name"

[FreeZzzz93]

Node_exporter falls into error at start when adding collector ethtool. Other optional collectors by flag are working fine.
I see this problem only on CentOS and RHEL, in Ubuntu everything works as it should.

panic: "node_ethtool_     err" is not a valid metric name
goroutine 60 [running]:
github.com/prometheus/client_golang/prometheus.MustNewConstMetric(...)
/go/pkg/mod/github.com/prometheus/client_golang@v1.11.0/prometheus/value.go:107
github.com/prometheus/node_exporter/collector.(*ethtoolCollector).Update(0xc0001ea180, 0xc0002107e0, 0x10d9580, 0x0)
         /app/collector/ethtool_linux.go:186 +0x1266
 github.com/prometheus/node_exporter/collector.execute(0xc0f535, 0x7, 0xcf1e00, 0xc0001ea180, 0xc0002107e0, 0xcf1980, 0xc00006f900)
         /app/collector/collector.go:161 +0x84
 github.com/prometheus/node_exporter/collector.NodeCollector.Collect.func1(0xc0002107e0, 0xc000178ba0, 0xcf1980, 0xc00006f900, 0xc0002049c0, 0xc0f535, 0x7, 0xcf1e00, 0xc0001ea180)
         /app/collector/collector.go:152 +0x71
 created by github.com/prometheus/node_exporter/collector.NodeCollector.Collect
         /app/collector/collector.go:151 +0x13c
 node_exporter.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
 node_exporter.service: Failed with result 'exit-code'.

Host operating system: output of uname -a

CentOS-8.4.2105
5.14.14-1.el8.elrepo.x86_64 #1 SMP Tue Oct 19 07:38:35 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
and RHEL8.3
4.18.0-240.el8.x86_64 #1 SMP Wed Sep 23 05:13:10 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

  node_exporter, version 1.2.2 (branch: HEAD, revision: 26645363b486e12be40af7ce4fc91e731a33104e)
  build user:       root@b9cb4aa2eb17
  build date:       20210806-13:44:18
  go version:       go1.16.7
  platform:         linux/amd64

node_exporter command line flags

/usr/local/bin/node_exporter --collector.ethtool

Are you running node_exporter in Docker?

no

ethtool -S output:

NIC statistics:
     Tx Queue#: 0
       TSO pkts tx: 0
       TSO bytes tx: 0
       ucast pkts tx: 20487
       ucast bytes tx: 1908107
       mcast pkts tx: 83
       mcast bytes tx: 5906
       bcast pkts tx: 4
       bcast bytes tx: 168
       pkts tx err: 0
       pkts tx discard: 0
       drv dropped tx total: 0
          too many frags: 0
          giant hdr: 0
          hdr err: 0
          tso: 0
       ring full: 0
       pkts linearized: 0
       hdr cloned: 0
       giant hdr: 0
     Rx Queue#: 0
       LRO pkts rx: 0
       LRO byte rx: 0
       ucast pkts rx: 25086
       ucast bytes rx: 2404103
       mcast pkts rx: 0
       mcast bytes rx: 0
       bcast pkts rx: 0
       bcast bytes rx: 0
       pkts rx OOB: 0
       pkts rx err: 0
       drv dropped rx total: 0
          err: 0
          fcs: 0
       rx buf alloc fail: 0
     tx timeout count: 0

In ubuntu, the same ethtool -S output looks like this:

NIC statistics:
     rx_packets: 5765
     tx_packets: 3521
     rx_bytes: 7960038
     tx_bytes: 264187
     rx_broadcast: 0
     tx_broadcast: 0
     rx_multicast: 0
     tx_multicast: 0
     rx_errors: 0
     tx_errors: 0
     tx_dropped: 0
     multicast: 0
     collisions: 0
     rx_length_errors: 0
     rx_over_errors: 0
     rx_crc_errors: 0
     rx_frame_errors: 0
     rx_no_buffer_count: 0
     rx_missed_errors: 0
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0
     tx_window_errors: 0
     tx_abort_late_coll: 0
     tx_deferred_ok: 0
     tx_single_coll_ok: 0
     tx_multi_coll_ok: 0
     tx_timeout_count: 0
     tx_restart_queue: 0
     rx_long_length_errors: 0
     rx_short_length_errors: 0
     rx_align_errors: 0
     tx_tcp_seg_good: 0
     tx_tcp_seg_failed: 0
     rx_flow_control_xon: 0
     rx_flow_control_xoff: 0
     tx_flow_control_xon: 0
     tx_flow_control_xoff: 0
     rx_long_byte_count: 7960038
     rx_csum_offload_good: 5723
     rx_csum_offload_errors: 0
     alloc_rx_buff_failed: 0
     tx_smbus: 0
     rx_smbus: 0
     dropped_smbus: 0

[SuperQ]

I think this may have been recently fixed with #2181.

[discordianfish]

@FreeZzzz93 Can you verify?
@SuperQ We should make sure this is fixed before releasing 1.3.0

[bdrung]

I doubt that #2181 fixes this issue. My gut feeling says that it is related to the indentation of the ethtool output. Can you check what the ethtool library produces for your interface?

git clone https://github.com/safchain/ethtool.git
cd ethtool/example
go build
./example -interface eth0

[ventifus]

I'm certain the kernel metric here has leading spaces, therefore the TrimLeft in #2093 ought to resolve this. (I suspect the difference in ethtool output between Ubuntu and RHEL is due to a different driver being used for that device).

@FreeZzzz93 are you able to build current master and check to see if we've already fixed this issue?

[ventifus]

There's another problem, this driver has both a "     giant hdr" (5 leading spaces) and a "  giant hdr" (2 leading spaces) metric, leading to duplicate node_ethtool_giant_hdr metrics. We don't currently have a way of ignoring ethtool metrics, so I'm afraid you'll have to configure the command-line flag --collector.ethtool.metrics-include to select which metrics you do want. Maybe you can select all the metrics that don't start with a "g": --collector.ethtool.metrics-include="^\s*[^g]"

[SuperQ]

@ventifus I added an exclude flag in #2165. 🎊

Sorry, I misread, that's for devices, not metrics. ☹️

[FreeZzzz93]

@bdrung the output turned out this

features: map[esp-hw-offload:false esp-tx-csum-hw-offload:false fcoe-mtu:false highdma:true hsr-dup-offload:false hsr-fwd-offload:false hsr-tag-ins-offload:false hsr-tag-rm-offload:false hw-tc-offload:false l2-fwd-offload:false loopback:false macsec-hw-offload:false netns-local:false rx-all:false rx-checksum:true rx-fcs:false rx-gro:true rx-gro-hw:false rx-gro-list:false rx-hashing:false rx-lro:false rx-ntuple-filter:false rx-udp-gro-forwarding:false rx-udp_tunnel-port-offload:false rx-vlan-filter:true rx-vlan-hw-parse:true rx-vlan-stag-filter:false rx-vlan-stag-hw-parse:false tls-hw-record:false tls-hw-rx-offload:false tls-hw-tx-offload:false tx-checksum-fcoe-crc:false tx-checksum-ip-generic:true tx-checksum-ipv4:false tx-checksum-ipv6:false tx-checksum-sctp:false tx-esp-segmentation:false tx-fcoe-segmentation:false tx-generic-segmentation:true tx-gre-csum-segmentation:false tx-gre-segmentation:false tx-gso-list:false tx-gso-partial:false tx-gso-robust:false tx-ipxip4-segmentation:false tx-ipxip6-segmentation:false tx-lockless:false tx-nocache-copy:false tx-scatter-gather:true tx-scatter-gather-fraglist:false tx-sctp-segmentation:false tx-tcp-ecn-segmentation:false tx-tcp-mangleid-segmentation:false tx-tcp-segmentation:true tx-tcp6-segmentation:true tx-tunnel-remcsum-segmentation:false tx-udp-segmentation:false tx-udp_tnl-csum-segmentation:false tx-udp_tnl-segmentation:false tx-vlan-hw-insert:true tx-vlan-stag-hw-insert:false vlan-challenged:false]
stats: map[     err:0      fcs:0      giant hdr:0      hdr err:0      too many frags:0      tso:0   LRO byte rx:0   LRO pkts rx:0   TSO bytes tx:4694   TSO pkts tx:1   bcast bytes rx:0   bcast bytes tx:528   bcast pkts rx:0   bcast pkts tx:9   drv dropped rx total:0   drv dropped tx total:0   giant hdr:0   hdr cloned:0   mcast bytes rx:0   mcast bytes tx:26056   mcast pkts rx:0   mcast pkts tx:195   pkts linearized:0   pkts rx OOB:0   pkts rx err:0   pkts tx discard:0   pkts tx err:0   ring full:0   rx buf alloc fail:0   ucast bytes rx:116054347   ucast bytes tx:9576827   ucast pkts rx:160390   ucast pkts tx:118390 Rx Queue#:0 Tx Queue#:0 tx timeout count:0]
bus info: 0000:0b:00.0
driver name: vmxnet3
cmd get: map[Advertising:128 Autoneg:0 Cmd:1 Duplex:1 Eth_tp_mdix:0 Lp_advertising:0 Maxrxpkt:0 Maxtxpkt:0 Mdio_support:0 Phy_address:0 Port:0 Reserved2:0 Reserved_unknown_type:0 Speed:10000 Speed_hi:0 Supported:4256 Transceiver:0 speed:10000]
panic: operation not supported

goroutine 1 [running]:
main.main()
	/home/hostuser/ethtool/example/main.go:57 +0x6e5

[ventifus]

Ah, vmware vmxnet3. I can test that.

Master fails with the error

ts=2021-10-28T18:08:53.057Z caller=level.go:63 level=error caller="error gathering metrics: [from Gatherer #2] collected metric \"node_ethtool_giant_hdr\" { label:<name:\"device\" value:\"ens192\" > untyped:<value:0" msg=" > } was collected before with the same name and label values"

Please use master (or the forthcoming 1.3.0 release) with the command line flags --collector.ethtool --collector.ethtool.metrics-include="^\s*[^g]" in order to gather all ethtool metrics except those that start with the letter g. I've verified this works as intended, so we don't need to hold back releasing 1.3.0 #2168 for this.

[FreeZzzz93]

@ventifus maybe I misunderstood how to use this key,
I add to node_exporter.service a line like this:
ExecStart=/usr/local/bin/node_exporter --collector.ethtool --collector.ethtool.metrics-include="^\s*[^g]"

but I get an error:

node_exporter: error: unknown long flag '--collector.ethtool.metrics-include', try --help
node_exporter.service: Main process exited, code=exited, status=1/FAILURE
node_exporter.service: Failed with result 'exit-code'

[ventifus]

Hi @FreeZzzz93, you have the right command-line flag, but it only applies to the next version of node_exporter that hasn't been released yet. Your options right now are to either not use the ethtool collector, or build node_exporter yourself from source code.

[bdrung]

I am working on fix for the duplicate metric names.