Skip to content

PR for: #189 pre and post coefficients excluding omitted ones#190

Merged
jorpppp merged 3 commits into
mainfrom
189-difference-of-averages-doesnt-include-all-coefficients
May 9, 2024
Merged

PR for: #189 pre and post coefficients excluding omitted ones#190
jorpppp merged 3 commits into
mainfrom
189-difference-of-averages-doesnt-include-all-coefficients

Conversation

@Constantino-Carreto-Romero
Copy link
Copy Markdown
Contributor

Closes #189

@jorpppp

This comment was marked as resolved.

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 8, 2024

Note that diffavg will return different results with trend(, method(ols)) and trend, method(gmm) because the coefficients for negative event time will differ between the two methods. I think this is fine but it is not properly explained in the helpfile and could lead to confusion. I opened #192 to address this later.

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 8, 2024

Note that diffavg will return different results with trend(, method(ols)) and trend, method(gmm) because the coefficients for negative event time will differ between the two methods. I think this is fine but it is not properly explained in the helpfile and could lead to confusion. I opened #192 to address this later.

Actually now I am thinking that we should exclude the endpoints from the diffavg calculation when we use trend. We omit them from the plot anyway. The endpoint is responsible for discrepancies in diffavg when using the two trend methods:

xtevent y, pol(z) window(5) trend(-2, method(ols)) diffavg
xtevent y, pol(z) window(5) trend(-2, method(gmm)) diffavg

Excluding the endpoints for this calculation would be consistent with #166

What do you think @Constantino-Carreto-Romero ?

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 9, 2024

@Constantino-Carreto-Romero With the example data:

xtevent y , panelvar(i) timevar(t) policyvar(z) window(5) impute(stag) diffavg proxy(x)

returns a "option diffavg not allowed" error message.

In 8d1f449 (#190) I enabled diffavg in _eventiv, so now diffavg can work with proxy()

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 9, 2024

I figure there's some weird issue going on with the flow of the code because

xtevent y , panelvar(i) timevar(t) policyvar(z) window(5) norm(-1 -2) diffavg

returns a "norm not allowed" error message, when it should probably return a syntax error instead.

@Constantino-Carreto-Romero let's wait until I can see if I find more bugs here to tackle this. Maybe I'll tackle it while you work on other tasks.

I actually found a deeper issue here. We use * in the syntax to pass options to the regression commands without actually spelling them out. This has a negative unintended consequence of skipping syntax checks. Consider the following example:

cap program drop mytest

program define mytest

syntax, [norm(integer -1 ) *]

di "`norm'"
di "`options'"

end

mytest, norm(-1)
mytest, norm(abc)

This returns:

image

which is unexpected. Because of the * in the syntax, Stata is not returning an invalid syntax error for mytest, norm(abc) even though it is not the right syntax because norm should be a string. Instead, it is using the default value for norm and passing the incorrectly-syntaxed norm(abc) option as an extra option in options.

This means that Stata is not properly checking for syntax errors in some places. This has not generated problems for us so far, but it might. I'll open a new issue for this.

Edit: We'll follow up on this in #193

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 9, 2024

Note that diffavg will return different results with trend(, method(ols)) and trend, method(gmm) because the coefficients for negative event time will differ between the two methods. I think this is fine but it is not properly explained in the helpfile and could lead to confusion. I opened #192 to address this later.

Actually now I am thinking that we should exclude the endpoints from the diffavg calculation when we use trend. We omit them from the plot anyway. The endpoint is responsible for discrepancies in diffavg when using the two trend methods:

xtevent y, pol(z) window(5) trend(-2, method(ols)) diffavg
xtevent y, pol(z) window(5) trend(-2, method(gmm)) diffavg

Excluding the endpoints for this calculation would be consistent with #166

What do you think @Constantino-Carreto-Romero ?

@Constantino-Carreto-Romero I'm going to go ahead and exclude the endpoints from the diffavg calculation when trend is active.

@jorpppp
Copy link
Copy Markdown
Collaborator

jorpppp commented May 9, 2024

All looks good here, merging the PR now @Constantino-Carreto-Romero

@jorpppp jorpppp merged commit fa0611b into main May 9, 2024
@jorpppp jorpppp deleted the 189-difference-of-averages-doesnt-include-all-coefficients branch May 9, 2024 15:39
jorpppp added a commit that referenced this pull request May 9, 2024
…195)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>
jorpppp added a commit that referenced this pull request May 9, 2024
* #188 ordering when there are not pre-event dummies

* #188 ordering the left endpoint

* #188 Do not test for pretrends for plot if left window is zero

* #188 #194 Disable smoothest path if left window

* PR for: #189 pre and post coefficients excluding omitted ones (#190) (#195)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

* #188 #194 Fix error messages for diffavg when windows are zero

* #188 #194 Fix lead selection and normalization when left window is zero or when many coefs are normalized

* #188 #194 Recheck that lead instrument is in estimation window when it is changed because normalizations coincide

* #188 #194 Do not test linear pretrend if too few pre-event coefs

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>
jorpppp added a commit that referenced this pull request May 14, 2024
* #181 Fix check for constant treatment value and add returns (#182)

* 186 error when normalizing 0 coefficient (#187)

* #186 Fix bug when normalizing to 0 or positive

* #186 Propagate the fix to _eventiv

* PR for #170: let the window option choose a window range (#178)

* #170 bring changes from main

* #170 implement finding max or balanced window

* #170 examples to test implementation

* #170 add examples to test file

* #170 change window description and add example in help file

* #170 wording of calculated-window message

* #170 modify description of left and right window

* #170 wording of window limits

* #170 wording of calculated-window message

* #170 check for zero words in window

* #170 rearrange parsewindow checks

* #170 wording of window-parser error messges

* #170 add to help file example about maximum window

* #170 conflict with static

* "#170 revert previous commit"

This reverts commit c6d98aa.

* #170 parse window only when it is non-empty

* #170 window in cmdline parser

* #170 Delete issue170 special directory

* #170 wording of string window message

* #170 let max or balanced only if impute is specified

* #170 wording of error message

* #170 restructure checks for staggered adoption

* #170 restructure for IV case

* #170 informative window message and error message for balanced

* "#170 revert previous commit"

This reverts commit fc197a9.

* #170 informative window message and error message for balanced

* #170 paragraph for max balanced

* #170 check trend outside window for max and balanced

* #170 remove comment

* #170 trend specification if window is max or balanced

* #170 clear message about treatment time

* #170 check leads in proxyiv are in estimation window

* #170 wording of message about calculated window

* #178 Minor change to calculated window message

* #170 mark non-missing obs in varlist

* #170 examples to test implementation

* #170 pass marker and revert syntax of eventgenvars

* #170 Add marked sample to variables to keep when repeatedcs

* #170 Delete issue170 directory

* #178 Edits to helpfile

---------

Co-authored-by: jorpppp <jorpppp@gmail.com>

* PR for #179 Check behavior of the cohort and control_cohort options and allow automatic generation (#185)

* #179 Create cohort_consistency.do

* #179 Add consistency checks between cohort, control_cohort, and policyvar, add basic cohort variable creation

* #179 Change cohort, control_cohort to string in xtevent

* #179 Add bin and no reversion returns to _eventgenvars plus changes from #181

* #179 Enable saving created cohort variable

* #179 Automatic generation of control_cohort

* #179 Update help file

* #179 Modify test file

* #179 Delete issue folder

* #179 #185 Generate cohort in sample that ignores missings in control vars

* #179 #185 Add checks for existing cohort variables

* #179 #185 SA and proxy not allowed

* #179 #185 Fix bug in naming saved interaction variables

* #179 #185 Fix sorting bug when creating cohorts in repeatedcs setting

* PR for: #189 pre and post coefficients excluding omitted ones (#190)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: jorpppp <jorpppp@gmail.com>

* 188 left window of zero returns an error (#194)

* #188 ordering when there are not pre-event dummies

* #188 ordering the left endpoint

* #188 Do not test for pretrends for plot if left window is zero

* #188 #194 Disable smoothest path if left window

* PR for: #189 pre and post coefficients excluding omitted ones (#190) (#195)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

* #188 #194 Fix error messages for diffavg when windows are zero

* #188 #194 Fix lead selection and normalization when left window is zero or when many coefs are normalized

* #188 #194 Recheck that lead instrument is in estimation window when it is changed because normalizations coincide

* #188 #194 Do not test linear pretrend if too few pre-event coefs

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>
This was referenced May 14, 2024
jorpppp added a commit that referenced this pull request May 14, 2024
* Bring #177 up to date again (#198)

* #181 Fix check for constant treatment value and add returns (#182)

* 186 error when normalizing 0 coefficient (#187)

* #186 Fix bug when normalizing to 0 or positive

* #186 Propagate the fix to _eventiv

* PR for #170: let the window option choose a window range (#178)

* #170 bring changes from main

* #170 implement finding max or balanced window

* #170 examples to test implementation

* #170 add examples to test file

* #170 change window description and add example in help file

* #170 wording of calculated-window message

* #170 modify description of left and right window

* #170 wording of window limits

* #170 wording of calculated-window message

* #170 check for zero words in window

* #170 rearrange parsewindow checks

* #170 wording of window-parser error messges

* #170 add to help file example about maximum window

* #170 conflict with static

* "#170 revert previous commit"

This reverts commit c6d98aa.

* #170 parse window only when it is non-empty

* #170 window in cmdline parser

* #170 Delete issue170 special directory

* #170 wording of string window message

* #170 let max or balanced only if impute is specified

* #170 wording of error message

* #170 restructure checks for staggered adoption

* #170 restructure for IV case

* #170 informative window message and error message for balanced

* "#170 revert previous commit"

This reverts commit fc197a9.

* #170 informative window message and error message for balanced

* #170 paragraph for max balanced

* #170 check trend outside window for max and balanced

* #170 remove comment

* #170 trend specification if window is max or balanced

* #170 clear message about treatment time

* #170 check leads in proxyiv are in estimation window

* #170 wording of message about calculated window

* #178 Minor change to calculated window message

* #170 mark non-missing obs in varlist

* #170 examples to test implementation

* #170 pass marker and revert syntax of eventgenvars

* #170 Add marked sample to variables to keep when repeatedcs

* #170 Delete issue170 directory

* #178 Edits to helpfile

---------

Co-authored-by: jorpppp <jorpppp@gmail.com>

* PR for #179 Check behavior of the cohort and control_cohort options and allow automatic generation (#185)

* #179 Create cohort_consistency.do

* #179 Add consistency checks between cohort, control_cohort, and policyvar, add basic cohort variable creation

* #179 Change cohort, control_cohort to string in xtevent

* #179 Add bin and no reversion returns to _eventgenvars plus changes from #181

* #179 Enable saving created cohort variable

* #179 Automatic generation of control_cohort

* #179 Update help file

* #179 Modify test file

* #179 Delete issue folder

* #179 #185 Generate cohort in sample that ignores missings in control vars

* #179 #185 Add checks for existing cohort variables

* #179 #185 SA and proxy not allowed

* #179 #185 Fix bug in naming saved interaction variables

* #179 #185 Fix sorting bug when creating cohorts in repeatedcs setting

* PR for: #189 pre and post coefficients excluding omitted ones (#190)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: jorpppp <jorpppp@gmail.com>

* 188 left window of zero returns an error (#194)

* #188 ordering when there are not pre-event dummies

* #188 ordering the left endpoint

* #188 Do not test for pretrends for plot if left window is zero

* #188 #194 Disable smoothest path if left window

* PR for: #189 pre and post coefficients excluding omitted ones (#190) (#195)

* #189 pre and post coefficients excluding omitted ones

* #189 #190 enable diff avg in _eventiv

* #189 #190 exclude endpoints from diffavg when trend is active

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

* #188 #194 Fix error messages for diffavg when windows are zero

* #188 #194 Fix lead selection and normalization when left window is zero or when many coefs are normalized

* #188 #194 Recheck that lead instrument is in estimation window when it is changed because normalizations coincide

* #188 #194 Do not test linear pretrend if too few pre-event coefs

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>

* Delete issue177 directory

* Revert "Delete issue177 directory"

This reverts commit 5c18688.

* Revert "Bring #177 up to date again (#198)"

This reverts commit 50b9efc.

---------

Co-authored-by: Constantino Carreto Romero <carretoromeroc@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Difference of averages doesn't include all coefficients

2 participants