"ValueError: Information ratio must be monotonically increasing" What does it mean? #102
Comments
|
Hi,
”Information ratio” is basically just a fancy way to say “current sample
size divided by final expected sample size”. So it would seem that, in your
data, your sample size decreases one day compared to the previous. Could
that be the case?
Cheers,
Pelle (who unfortunately doesn’t maintain confidence anymore since I left
Spotify in Dec. -23)
tis 16 juli 2024 kl. 23:29 skrev Eric Scott ***@***.***>:
… I'm trying to use the lib for the first time, for a sequential test. I'm
getting an exception "ValueError: Information ratio must be monotonically
increasing", but I don't know what it means. I've tried digesting the code,
but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:135](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py#line=134), in compute_sequential_adjusted_alpha.<locals>.adjusted_alphas_for_group(grp)
133 def adjusted_alphas_for_group(grp: DataFrame) -> Series:
134 return (
--> 135 sequential_bounds(
136 t=grp["sample_size_proportions"].values,
137 alpha=grp[ALPHA].values[0] [/](https://escott-dsc3099.dev.de.gcp.rokulabs.net/) n_comparisons,
138 sides=2 if (grp[PREFERENCE_TEST] == TWO_SIDED).all() else 1,
139 )
140 .df.set_index(grp.index)
141 .assign(
142 **{
143 ADJUSTED_ALPHA: lambda df: df.apply(
144 lambda row: 2 * (1 - st.norm.cdf(row["zb"]))
145 if (grp[PREFERENCE_TEST] == TWO_SIDED).all()
146 else 1 - st.norm.cdf(row["zb"]),
147 axis=1,
148 )
149 }
150 )
151 )[["zb", ADJUSTED_ALPHA]]
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py:53](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/confidence_computers/z_test_computer.py#line=52), in sequential_bounds(t, alpha, sides, state)
52 def sequential_bounds(t: np.array, alpha: float, sides: int, state: DataFrame = None):
---> 53 return bounds(t, alpha, rho=2, ztrun=8, sides=sides, max_nints=1000, state=state)
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:343](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py#line=342), in bounds(t, alpha, rho, ztrun, sides, state, max_nints)
341 raise ValueError(f"Information ratio must must not be zero, {get_input_str()}")
342 if any(t[i] > t[i + 1] for i in range(len(t) - 1)):
--> 343 raise ValueError(f"Information ratio must be monotonically increasing, {get_input_str()}")
344 if not (sides == 1 or sides == 2):
345 raise ValueError(f"sides must either be one a zero, {get_input_str()}")
File [~/deepsea/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py:337](https://escott-dsc3099.dev.de.gcp.rokulabs.net/lab/tree/notebooks/nba/nco/notebooks/nba/nco/confidence-master/spotify_confidence/analysis/frequentist/sequential_bound_solver.py#line=336), in bounds.<locals>.get_input_str()
334 def get_input_str():
335 return (
336 f"input params: t={t}, alpha={alpha}, sides={sides}, rho={rho}, ztrun={ztrun},"
--> 337 f"state_df={state.df.to_json()}, state_fcab={state.last_fcab}, max_nints={max_nints}"
338 )
AttributeError: 'NoneType' object has no attribute 'df'
And here's the setup I did, trying to follow the example notebook:
test = conf.ZTest(
metrics_df,
numerator_column='metric1',
numerator_sum_squares_column='metric1_sumsq',
denominator_column='user_cnt',
categorical_group_columns=['bucket'],
ordinal_group_column='date'
)
# This call goes boom!
result = test.multiple_difference(
level='control',
groupby=['date'],
level_as_reference=True,
final_expected_sample_size_column='expected_users',
non_inferiority_margins = (None, 'increase'),
absolute=False
)
—
Reply to this email directly, view it on GitHub
<#102>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEHJDXQWCMFVUCI6GFNENNDZMWGE7AVCNFSM6AAAAABK7MIQDCVHI2DSMVQWIX3LMV43ASLTON2WKOZSGQYTEMBXGQ2TINY>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Hi Pelle,
Thanks for the quick response.
My data set captures the users and their results for each date. The rows
are independent of each other. From your explanation I'm guessing that's
not what is expected. What is the assumed relationship between one row and
the next?
Here are some example rows to illustrate my data:
```
bucket | date | user_cnt | metric1 | expected_users
control | 8/30/23 | 4349 | 102.764 | 179800
treated | 8/30/23 | 11327 | 108.553 | 179800
control | 8/31/23 | 4299 | 100.326 | 179800
treated | 8/31/23 | 11117 | 106.565 | 179800
control | 9/1/23 | 5278 | 98.143 | 179800
treated | 9/1/23 | 10456 | 100.402 | 179800
```
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to use the lib for the first time, for a sequential test. I'm getting an exception "ValueError: Information ratio must be monotonically increasing", but I don't know what it means. I've tried digesting the code, but this is buried more levels deep than I understand. Can you educate me?
Here's the snipped exception:
And here's the setup I did, trying to follow the example notebook:
The text was updated successfully, but these errors were encountered: