Mod.score inconsistency between LogisticRegressionOutput and SoftmaxOutput

[yifeim]

Description

LogisticRegressionOutput (with 1 fc) and SoftmaxOutput (with 2 fcs) are mostly interchangeable in training codes. However, mod.score output fails on LogisticRegressionOutput. Original problem discovered while following the example from https://xiandong79.github.io/MXNet-Logistic-Regression-Example

Environment info (Required)

----------Python Info----------
Version : 3.6.4
Compiler : GCC 7.2.0
Build : ('default', 'Jan 16 2018 18:10:19')
Arch : ('64bit', '')
------------Pip Info-----------
Version : 9.0.1
Directory : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/pip
----------MXNet Info-----------
Version : 1.0.0
Directory : /home/ubuntu/anaconda3/envs/mxnet_p36/lib/python3.6/site-packages/mxnet
Commit Hash : 9ef1969
----------System Info----------
Platform : Linux-4.4.0-1055-aws-x86_64-with-debian-stretch-sid
system : Linux
node : ip-172-31-0-77
release : 4.4.0-1055-aws
version : #64-Ubuntu SMP Thu Apr 5 17:06:36 UTC 2018
----------Hardware Info----------
machine : x86_64
processor : x86_64
----------Network Test----------
Setting timeout: 10
Timing for MXNet: https://github.com/apache/incubator-mxnet, DNS: 0.0030 sec, LOAD: 0.5252 sec.
Timing for Gluon Tutorial(en): http://gluon.mxnet.io, DNS: 0.0688 sec, LOAD: 0.0552 sec.
Timing for Gluon Tutorial(cn): https://zh.gluon.ai, DNS: 0.4129 sec, LOAD: 0.1303 sec.
Timing for FashionMNIST: https://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/dataset/fashion-mnist/train-labels-idx1-ubyte.gz, DNS: 0.0087 sec, LOAD: 0.1347 sec.
Timing for PYPI: https://pypi.python.org/pypi/pip, DNS: 0.0034 sec, LOAD: 0.3965 sec.
Timing for Conda: https://repo.continuum.io/pkgs/free/, DNS: 0.0089 sec, LOAD: 0.0761 sec.

Package used (Python/R/Scala/Julia):
I'm using Python (jupyter notebook in Amazon Deep Learning AMI).

Minimum reproducible example

import numpy as np
import mxnet as mx
import logging, imp

imp.reload(logging)
logging.basicConfig(level=logging.INFO)

np.random.seed(0)
mx.random.seed(0)
data = np.random.randn(100,2).astype('float32')
label = (data.dot([1,-1])>0.).astype('float32')

sym = mx.sym.SoftmaxOutput(
    mx.sym.FullyConnected(data=mx.sym.Variable('data'), num_hidden=2, name ='fc'),
    label=mx.sym.Variable('softmax_label')
)
train_iter = mx.io.NDArrayIter(data, label, 100,)

mod = mx.mod.Module(symbol=sym)
mod.fit(train_iter, eval_metric='acc', num_epoch=5,
        optimizer='adam', optimizer_params={'learning_rate':1.})
print(mod.score(train_iter, 'acc'))
print(np.mean((mod.predict(train_iter).asnumpy()[:,-1]>.5)==label))

# INFO:root:Epoch[0] Train-accuracy=0.450000
# INFO:root:Epoch[0] Time cost=0.005
# INFO:root:Epoch[1] Train-accuracy=0.770000
# INFO:root:Epoch[1] Time cost=0.004
# INFO:root:Epoch[2] Train-accuracy=0.960000
# INFO:root:Epoch[2] Time cost=0.003
# INFO:root:Epoch[3] Train-accuracy=0.990000
# INFO:root:Epoch[3] Time cost=0.004
# INFO:root:Epoch[4] Train-accuracy=0.960000
# INFO:root:Epoch[4] Time cost=0.003
# [('accuracy', 0.95999999999999996)]
# 0.96

sym = mx.sym.LogisticRegressionOutput(
    mx.sym.FullyConnected(data=mx.sym.Variable('data'), num_hidden=1, name ='fc'),
    label=mx.sym.Variable('softmax_label')
)
train_iter = mx.io.NDArrayIter(data, label, 100,)

mod = mx.mod.Module(symbol=sym)
mod.fit(train_iter, eval_metric='acc', num_epoch=5,
        optimizer='adam', optimizer_params={'learning_rate':1.})
print(mod.score(train_iter, 'acc'))
print(np.mean((mod.predict(train_iter).asnumpy()[:,-1]>.5)==label))

# INFO:root:Epoch[0] Train-accuracy=0.540000
# INFO:root:Epoch[0] Time cost=0.004
# INFO:root:Epoch[1] Train-accuracy=0.540000
# INFO:root:Epoch[1] Time cost=0.001
# INFO:root:Epoch[2] Train-accuracy=0.540000
# INFO:root:Epoch[2] Time cost=0.001
# INFO:root:Epoch[3] Train-accuracy=0.540000
# INFO:root:Epoch[3] Time cost=0.001
# INFO:root:Epoch[4] Train-accuracy=0.540000
# INFO:root:Epoch[4] Time cost=0.001
# [('accuracy', 0.54000000000000004)]
# 0.97

Steps to reproduce

1. run the above codes in jupyter conda_mxnet_p36 environment.

What have you tried to solve it?

1. Same errors if I use label[:,None] with LogisticRegressionOutput.
2. Write my own training codes.
3. Just a side question, what to do if I want to capture my training errors not in log messages but as a list so I can plot if afterwards, in mod.fit?

[nswamy]

@yifeim Please ask the question https://discuss.mxnet.io, its the best place to get answers from maintainers and other users with similar experience.