Fix bug in ohlc code and reenable parallelism #644
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
WireBaron
requested review from
jerryxwu,
rtwalker,
syvb,
thatzopoulos,
davidkohn88 and
epgts
epgts
reviewed
Dec 1, 2022
There was a problem hiding this comment.
Fix looks OK, but the lack of an automated test failing without it and passing with it, even if it's too expensive to run under github actions, is concerning.
Also, for future reference, if you revert the commit that deleted this first and then add the fix in a separate commit, the fix is reviewable in itself.
I did this locally so I could see:
git revert 408b498539f568273f4e71d87d5b5a433693ca0b
git cherry-pick origin/br/ohlc_fix
# resolve the trivial conflict (which is also just the bug-fix :)
git show
[...]
- (None, Some(only)) | (Some(only), None) => Some(only),
+ (None, Some(only)) | (Some(only), None) => Some((*only).into()),
rtwalker
approved these changes
Dec 1, 2022
There was a problem hiding this comment.
I've been using the below query to reliably detect bad results:
SELECT
symbol,
time_bucket('1 hour'::INTERVAL, "time") AS ts,
first(price, "time"),
max(price),
min(price),
last(price, "time"),
toolkit_experimental.open(toolkit_experimental.ohlc("time", price)),
toolkit_experimental.high(toolkit_experimental.ohlc("time", price)),
toolkit_experimental.low(toolkit_experimental.ohlc("time", price)),
toolkit_experimental.close(toolkit_experimental.ohlc("time", price)),
abs(first(price, "time") - toolkit_experimental.open(toolkit_experimental.ohlc("time", price))) AS o_diff,
abs(max(price) - toolkit_experimental.high(toolkit_experimental.ohlc("time", price))) AS h_diff,
abs(min(price) - toolkit_experimental.low(toolkit_experimental.ohlc("time", price))) AS l_diff,
abs(last(price, "time") - toolkit_experimental.close(toolkit_experimental.ohlc("time", price))) AS c_diff
FROM stocks_real_time
GROUP BY ts, symbol
HAVING abs(max(price) - toolkit_experimental.high(toolkit_experimental.ohlc("time", price))) > 0.0001
OR abs(min(price) - toolkit_experimental.low(toolkit_experimental.ohlc("time", price))) > 0.0001
OR abs(last(price, "time") - toolkit_experimental.close(toolkit_experimental.ohlc("time", price))) > 0.0001
OR abs(first(price, "time") - toolkit_experimental.open(toolkit_experimental.ohlc("time", price))) > 0.0001
;with this branch installed:
symbol │ ts │ first │ max │ min │ last │ open │ high │ low │ close │ o_diff │ h_diff │ l_diff │ c_diff
────────┼────┼───────┼─────┼─────┼──────┼──────┼──────┼─────┼───────┼────────┼────────┼────────┼────────
(0 rows)
1.12.0 behavior:
symbol │ ts │ first │ max │ min │ last │ open │ high │ low │ close │ o_diff │ h_diff │ l_diff │ c_diff
────────┼────────────────────────┼─────────────┼──────────┼──────────┼──────────┼──────────┼──────────┼──────────┼──────────┼───────────────────────┼───────────────────────┼───────────────────────┼───────────────────────
AAPL │ 2022-10-26 12:00:00-04 │ 151.5599 │ 151.985 │ 150.5 │ 150.59 │ 151.0915 │ 151.12 │ 150.57 │ 150.57 │ 0.4684000000000026 │ 0.8650000000000091 │ 0.06999999999999318 │ 0.020000000000010232
AMAT │ 2022-10-26 12:00:00-04 │ 89.69 │ 90.25 │ 88.56 │ 88.78 │ 89 │ 89.02 │ 88.56 │ 88.705 │ 0.6899999999999977 │ 1.230000000000004 │ 0 │ 0.07500000000000284
AMZN │ 2022-10-26 12:00:00-04 │ 118.64 │ 118.8 │ 116.3632 │ 116.56 │ 116.85 │ 116.96 │ 116.3632 │ 116.5511 │ 1.7900000000000063 │ 1.8400000000000034 │ 0 │ 0.008899999999997021
BA │ 2022-10-26 12:00:00-04 │ 142.45 │ 142.54 │ 139.94 │ 140.04 │ 142.45 │ 142.54 │ 139.99 │ 140.04 │ 0 │ 0 │ 0.05000000000001137 │ 0
BAC │ 2022-10-26 12:00:00-04 │ 35.95 │ 36.02 │ 35.66 │ 35.67 │ 35.73 │ 35.73 │ 35.66 │ 35.66 │ 0.22000000000000597 │ 0.29000000000000625 │ 0 │ 0.010000000000005116
C │ 2022-10-26 12:00:00-04 │ 45.83 │ 45.87 │ 45.41 │ 45.45 │ 45.52 │ 45.52 │ 45.42 │ 45.42 │ 0.30999999999999517 │ 0.3499999999999943 │ 0.010000000000005116 │ 0.030000000000001137
CMCSA │ 2022-10-26 12:00:00-04 │ 31.82 │ 31.855 │ 31.6 │ 31.615 │ 31.705 │ 31.705 │ 31.6 │ 31.605 │ 0.11500000000000199 │ 0.15000000000000213 │ 0 │ 0.00999999999999801
CRM │ 2022-10-26 12:00:00-04 │ 163.47 │ 163.81 │ 160.73 │ 160.79 │ 163.47 │ 163.81 │ 160.74 │ 160.79 │ 0 │ 0 │ 0.010000000000019327 │ 0
CSCO │ 2022-10-26 12:00:00-04 │ 44.96 │ 45.03 │ 44.435 │ 44.485 │ 44.55 │ 44.56 │ 44.435 │ 44.47 │ 0.4100000000000037 │ 0.46999999999999886 │ 0 │ 0.015000000000000568
CVX │ 2022-10-26 12:00:00-04 │ 177.7 │ 178.48 │ 177.58 │ 177.97 │ 178.16 │ 178.16 │ 177.81 │ 177.81 │ 0.46000000000000796 │ 0.3199999999999932 │ 0.22999999999998977 │ 0.1599999999999966
GE │ 2022-10-26 12:00:00-04 │ 76.11 │ 76.62 │ 76.11 │ 76.3 │ 76.45 │ 76.45 │ 76.28 │ 76.31 │ 0.3400000000000034 │ 0.1700000000000017 │ 0.1700000000000017 │ 0.010000000000005116
GM │ 2022-10-26 12:00:00-04 │ 37.91 │ 38.19 │ 37.91 │ 37.95 │ 38 │ 38.01 │ 37.92 │ 37.92 │ 0.09000000000000341 │ 0.17999999999999972 │ 0.010000000000005116 │ 0.030000000000001137
GOOG │ 2022-10-26 12:00:00-04 │ 98.1199 │ 98.19 │ 97.03 │ 97.12 │ 98.1199 │ 98.19 │ 97.04 │ 97.12 │ 0 │ 0 │ 0.010000000000005116 │ 0
HD │ 2022-10-26 12:00:00-04 │ 291.31 │ 291.85 │ 290.19 │ 290.33 │ 291.14 │ 291.14 │ 290.19 │ 290.45 │ 0.17000000000001592 │ 0.7100000000000364 │ 0 │ 0.12000000000000455
IBM │ 2022-10-26 12:00:00-04 │ 135.74 │ 135.86 │ 135.24 │ 135.32 │ 135.74 │ 135.86 │ 135.25 │ 135.32 │ 0 │ 0 │ 0.009999999999990905 │ 0
INTC │ 2022-10-26 12:00:00-04 │ 27.595 │ 27.605 │ 27.32 │ 27.33 │ 27.43 │ 27.435 │ 27.325 │ 27.335 │ 0.16499999999999915 │ 0.1700000000000017 │ 0.004999999999999005 │ 0.005000000000002558
JPM │ 2022-10-26 12:00:00-04 │ 124.69 │ 124.75 │ 123.63 │ 123.74 │ 123.99 │ 123.99 │ 123.71 │ 123.71 │ 0.7000000000000028 │ 0.7600000000000051 │ 0.0799999999999983 │ 0.030000000000001137
KO │ 2022-10-26 12:00:00-04 │ 59.66 │ 59.77 │ 59.25 │ 59.33 │ 59.66 │ 59.77 │ 59.26 │ 59.33 │ 0 │ 0 │ 0.00999999999999801 │ 0
MMM │ 2022-10-26 12:00:00-04 │ 122.1 │ 123.27 │ 122.07 │ 122.31 │ 122.44 │ 122.52 │ 122.22 │ 122.31 │ 0.3400000000000034 │ 0.75 │ 0.15000000000000568 │ 0
MRK │ 2022-10-26 12:00:00-04 │ 98.98 │ 99.27 │ 98.54 │ 98.72 │ 98.76 │ 98.78 │ 98.69 │ 98.72 │ 0.21999999999999886 │ 0.4899999999999949 │ 0.14999999999999147 │ 0
MSFT │ 2022-10-26 12:00:00-04 │ 236.52 │ 237.1697 │ 234.85 │ 234.96 │ 235.33 │ 235.3624 │ 234.955 │ 235.05 │ 1.1899999999999977 │ 1.807299999999998 │ 0.10500000000001819 │ 0.09000000000000341
NFLX │ 2022-10-26 12:00:00-04 │ 303.37 │ 304.38 │ 299.656 │ 301.02 │ 300.75 │ 300.83 │ 299.656 │ 300.09 │ 2.6200000000000045 │ 3.5500000000000114 │ 0 │ 0.9300000000000068
NVDA │ 2022-10-26 12:00:00-04 │ 132.99 │ 133.33 │ 130.8 │ 130.965 │ 131.4808 │ 131.5899 │ 130.8 │ 130.945 │ 1.5092000000000212 │ 1.7401000000000124 │ 0 │ 0.020000000000010232
PFE │ 2022-10-26 12:00:00-04 │ 46.59 │ 46.65 │ 46.4 │ 46.44 │ 46.48 │ 46.49 │ 46.4 │ 46.41 │ 0.11000000000000654 │ 0.1599999999999966 │ 0 │ 0.030000000000001137
PYPL │ 2022-10-26 12:00:00-04 │ 90.49 │ 90.65 │ 88.92 │ 88.98 │ 89.28 │ 89.28 │ 88.92 │ 89 │ 1.2099999999999937 │ 1.3700000000000045 │ 0 │ 0.01999999999999602
QCOM │ 2022-10-26 12:00:00-04 │ 120.16 │ 120.5 │ 118.85 │ 118.8999 │ 120.16 │ 120.5 │ 118.88 │ 118.8999 │ 0 │ 0 │ 0.030000000000001137 │ 0
SNAP │ 2022-10-26 12:00:00-04 │ 9.66 │ 9.78 │ 9.5 │ 9.54 │ 9.55 │ 9.56 │ 9.5 │ 9.53 │ 0.10999999999999943 │ 0.21999999999999886 │ 0 │ 0.009999999999999787
SQ │ 2022-10-26 12:00:00-04 │ 61.84 │ 62.31 │ 60.3 │ 60.44 │ 61.84 │ 62.31 │ 60.34 │ 60.44 │ 0 │ 0 │ 0.04000000000000625 │ 0
T │ 2022-10-26 12:00:00-04 │ 18.11 │ 18.15 │ 18.02 │ 18.02 │ 18.05 │ 18.06 │ 18.02 │ 18.02 │ 0.05999999999999872 │ 0.08999999999999986 │ 0 │ 0
TSLA │ 2022-10-26 12:00:00-04 │ 228.9 │ 230 │ 225.347 │ 226.11 │ 228.9 │ 230 │ 225.56 │ 226.11 │ 0 │ 0 │ 0.21299999999999386 │ 0
UBER │ 2022-10-26 12:00:00-04 │ 28.83 │ 28.95 │ 28.06 │ 28.09 │ 28.23 │ 28.23 │ 28.09 │ 28.11 │ 0.5999999999999979 │ 0.7199999999999989 │ 0.030000000000001137 │ 0.019999999999999574
V │ 2022-10-26 12:00:00-04 │ 204.1 │ 204.87 │ 202.82 │ 203.06 │ 203.37 │ 203.37 │ 202.87 │ 202.99 │ 0.7299999999999898 │ 1.5 │ 0.05000000000001137 │ 0.06999999999999318
WFC │ 2022-10-26 12:00:00-04 │ 46.11 │ 46.14 │ 45.67 │ 45.71 │ 45.76 │ 45.76 │ 45.68 │ 45.69 │ 0.3500000000000014 │ 0.38000000000000256 │ 0.00999999999999801 │ 0.020000000000003126
ZM │ 2022-10-26 12:00:00-04 │ 84.425 │ 84.73 │ 83.2 │ 83.3 │ 83.54 │ 83.54 │ 83.2 │ 83.355 │ 0.8849999999999909 │ 1.1899999999999977 │ 0 │ 0.05500000000000682
ABNB │ 2022-10-26 13:00:00-04 │ 116.29 │ 116.71 │ 115.28 │ 116.37 │ 116.29 │ 116.63 │ 115.28 │ 116.31 │ 0 │ 0.0799999999999983 │ 0 │ 0.060000000000002274
AMAT │ 2022-10-26 13:00:00-04 │ 88.735 │ 88.93 │ 88.01 │ 88.625 │ 88.735 │ 88.93 │ 88.12 │ 88.625 │ 0 │ 0 │ 0.10999999999999943 │ 0
AMD │ 2022-10-26 13:00:00-04 │ 60.4 │ 60.405 │ 59.295 │ 59.9625 │ 60.3999 │ 60.405 │ 59.295 │ 59.965 │ 9.999999999621423e-05 │ 0 │ 0 │ 0.0025000000000048317
AMZN │ 2022-10-26 13:00:00-04 │ 116.515 │ 116.65 │ 115.51 │ 116.23 │ 116.515 │ 116.65 │ 115.585 │ 116.23 │ 0 │ 0 │ 0.07499999999998863 │ 0
BA │ 2022-10-26 13:00:00-04 │ 140.08 │ 140.08 │ 135.48 │ 135.61 │ 140.08 │ 140.08 │ 135.5 │ 135.77 │ 0 │ 0 │ 0.020000000000010232 │ 0.1599999999999966
BAC │ 2022-10-26 13:00:00-04 │ 35.66 │ 35.76 │ 35.62 │ 35.73 │ 35.63 │ 35.75 │ 35.62 │ 35.73 │ 0.02999999999999403 │ 0.00999999999999801 │ 0 │ 0
BMY │ 2022-10-26 13:00:00-04 │ 74.24 │ 74.58 │ 74.15 │ 74.56 │ 74.24 │ 74.57 │ 74.15 │ 74.57 │ 0 │ 0.010000000000005116 │ 0 │ 0.009999999999990905
C │ 2022-10-26 13:00:00-04 │ 45.44 │ 45.55 │ 45.37 │ 45.53 │ 45.37 │ 45.55 │ 45.37 │ 45.53 │ 0.07000000000000028 │ 0 │ 0 │ 0
CAT │ 2022-10-26 13:00:00-04 │ 196.92 │ 197.98 │ 196.62 │ 197.74 │ 196.92 │ 197.93 │ 196.72 │ 197.72 │ 0 │ 0.04999999999998295 │ 0.09999999999999432 │ 0.020000000000010232
CMCSA │ 2022-10-26 13:00:00-04 │ 31.61 │ 31.815 │ 31.5423 │ 31.805 │ 31.61 │ 31.81 │ 31.5423 │ 31.805 │ 0 │ 0.005000000000002558 │ 0 │ 0
CRM │ 2022-10-26 13:00:00-04 │ 160.75 │ 160.9 │ 158.71 │ 160.79 │ 160.71 │ 160.9 │ 158.71 │ 160.8 │ 0.03999999999999204 │ 0 │ 0 │ 0.010000000000019327
CSCO │ 2022-10-26 13:00:00-04 │ 44.46 │ 44.55 │ 44.34 │ 44.505 │ 44.43 │ 44.55 │ 44.37 │ 44.505 │ 0.030000000000001137 │ 0 │ 0.02999999999999403 │ 0
CVS │ 2022-10-26 13:00:00-04 │ 93.1 │ 93.5 │ 93.01 │ 93.34 │ 93.1 │ 93.49 │ 93.05 │ 93.36 │ 0 │ 0.010000000000005116 │ 0.03999999999999204 │ 0.01999999999999602
CVX │ 2022-10-26 13:00:00-04 │ 177.94 │ 178.25 │ 176.83 │ 177.37 │ 177.94 │ 178.22 │ 176.83 │ 177.37 │ 0 │ 0.030000000000001137 │ 0 │ 0
DIS │ 2022-10-26 13:00:00-04 │ 105.87 │ 105.93 │ 105.35 │ 105.69 │ 105.83 │ 105.91 │ 105.35 │ 105.68 │ 0.04000000000000625 │ 0.020000000000010232 │ 0 │ 0.009999999999990905
GE │ 2022-10-26 13:00:00-04 │ 76.3 │ 76.5 │ 75.74 │ 76.05 │ 76.3 │ 76.5 │ 75.85 │ 76.05 │ 0 │ 0 │ 0.10999999999999943 │ 0
GILD │ 2022-10-26 13:00:00-04 │ 71.07 │ 71.09 │ 70.81 │ 70.82 │ 71.06 │ 71.085 │ 70.81 │ 70.835 │ 0.009999999999990905 │ 0.005000000000009663 │ 0 │ 0.015000000000000568
GM │ 2022-10-26 13:00:00-04 │ 37.95 │ 38.13 │ 37.87 │ 38.12 │ 37.92 │ 38.12 │ 37.9 │ 38.12 │ 0.030000000000001137 │ 0.010000000000005116 │ 0.030000000000001137 │ 0
GOOG │ 2022-10-26 13:00:00-04 │ 97.1175 │ 97.365 │ 96.4 │ 96.44 │ 97.11 │ 97.3201 │ 96.4 │ 96.485 │ 0.00750000000000739 │ 0.044899999999998386 │ 0 │ 0.045000000000001705
HD │ 2022-10-26 13:00:00-04 │ 290.5 │ 291.16 │ 289.96 │ 291.11 │ 290.15 │ 291.16 │ 290.15 │ 290.98 │ 0.35000000000002274 │ 0 │ 0.18999999999999773 │ 0.12999999999999545
IBM │ 2022-10-26 13:00:00-04 │ 135.35 │ 135.74 │ 135.17 │ 135.66 │ 135.35 │ 135.7 │ 135.17 │ 135.66 │ 0 │ 0.040000000000020464 │ 0 │ 0
INTC │ 2022-10-26 13:00:00-04 │ 27.325 │ 27.3661 │ 27.1832 │ 27.33 │ 27.325 │ 27.3661 │ 27.2101 │ 27.33 │ 0 │ 0 │ 0.026900000000001256 │ 0
JNJ │ 2022-10-26 13:00:00-04 │ 172.27 │ 172.39 │ 172.04 │ 172.21 │ 172.27 │ 172.37 │ 172.09 │ 172.25 │ 0 │ 0.01999999999998181 │ 0.05000000000001137 │ 0.03999999999999204
JPM │ 2022-10-26 13:00:00-04 │ 123.75 │ 124.09 │ 123.58 │ 124.01 │ 123.75 │ 124.09 │ 123.59 │ 124.01 │ 0 │ 0 │ 0.010000000000005116 │ 0
KO │ 2022-10-26 13:00:00-04 │ 59.33 │ 59.47 │ 59.27 │ 59.4 │ 59.33 │ 59.45 │ 59.3 │ 59.43 │ 0 │ 0.01999999999999602 │ 0.02999999999999403 │ 0.030000000000001137
MMM │ 2022-10-26 13:00:00-04 │ 122.32 │ 122.89 │ 121.97 │ 122.85 │ 122.07 │ 122.87 │ 121.97 │ 122.85 │ 0.25 │ 0.01999999999999602 │ 0 │ 0
MO │ 2022-10-26 13:00:00-04 │ 46.45 │ 46.56 │ 46.41 │ 46.49 │ 46.44 │ 46.56 │ 46.41 │ 46.49 │ 0.010000000000005116 │ 0 │ 0 │ 0
MRK │ 2022-10-26 13:00:00-04 │ 98.69 │ 98.69 │ 98.45 │ 98.56 │ 98.63 │ 98.64 │ 98.45 │ 98.56 │ 0.060000000000002274 │ 0.04999999999999716 │ 0 │ 0
MS │ 2022-10-26 13:00:00-04 │ 81.03 │ 81.19 │ 80.79 │ 81.14 │ 81.03 │ 81.17 │ 80.79 │ 81.15 │ 0 │ 0.01999999999999602 │ 0 │ 0.010000000000005116
MSFT │ 2022-10-26 13:00:00-04 │ 234.91 │ 235.189 │ 233.285 │ 235.1 │ 234.91 │ 235.189 │ 233.805 │ 235.1 │ 0 │ 0 │ 0.5200000000000102 │ 0
MU │ 2022-10-26 13:00:00-04 │ 55.445 │ 55.48 │ 54.96 │ 55.465 │ 55.42 │ 55.48 │ 54.96 │ 55.47 │ 0.02499999999999858 │ 0 │ 0 │ 0.0049999999999954525
NFLX │ 2022-10-26 13:00:00-04 │ 300.7 │ 302.64 │ 299.4793 │ 302.03 │ 300.7 │ 302.64 │ 299.5779 │ 302.03 │ 0 │ 0 │ 0.09859999999997626 │ 0
NKE │ 2022-10-26 13:00:00-04 │ 93.63 │ 93.81 │ 93.04 │ 93.27 │ 93.63 │ 93.8 │ 93.04 │ 93.27 │ 0 │ 0.010000000000005116 │ 0 │ 0
NVDA │ 2022-10-26 13:00:00-04 │ 130.9001 │ 130.93 │ 128.66 │ 130.1 │ 130.9001 │ 130.9001 │ 129.07 │ 130.1 │ 0 │ 0.029899999999997817 │ 0.4099999999999966 │ 0
ORCL │ 2022-10-26 13:00:00-04 │ 75.14 │ 75.36 │ 75.04 │ 75.35 │ 75.14 │ 75.36 │ 75.04 │ 75.36 │ 0 │ 0 │ 0 │ 0.010000000000005116
PFE │ 2022-10-26 13:00:00-04 │ 46.44 │ 46.5 │ 46.34 │ 46.47 │ 46.38 │ 46.5 │ 46.34 │ 46.47 │ 0.05999999999999517 │ 0 │ 0 │ 0
...skipping...
AMZN │ 2022-11-09 17:00:00-05 │ 86.21 │ 86.25 │ 86.07 │ 86.14 │ 86.25 │ 86.25 │ 86.07 │ 86.14 │ 0.04000000000000625 │ 0 │ 0 │ 0
BA │ 2022-11-09 17:00:00-05 │ 168.7 │ 168.85 │ 168.5 │ 168.5 │ 168.7 │ 168.85 │ 168.5 │ 168.52 │ 0 │ 0 │ 0 │ 0.020000000000010232
BAC │ 2022-11-09 17:00:00-05 │ 36.54 │ 36.56 │ 36.47 │ 36.48 │ 36.56 │ 36.56 │ 36.47 │ 36.48 │ 0.020000000000003126 │ 0 │ 0 │ 0
BMY │ 2022-11-09 17:00:00-05 │ 79.6 │ 79.78 │ 79.44 │ 79.44 │ 79.59 │ 79.78 │ 79.44 │ 79.44 │ 0.009999999999990905 │ 0 │ 0 │ 0
C │ 2022-11-09 17:00:00-05 │ 45.33 │ 45.38 │ 45.3 │ 45.38 │ 45.37 │ 45.37 │ 45.3 │ 45.3 │ 0.03999999999999915 │ 0.010000000000005116 │ 0 │ 0.0800000000000054
CAT │ 2022-11-09 17:00:00-05 │ 225.54 │ 226.7 │ 225.2 │ 226 │ 225.54 │ 226.7 │ 225.25 │ 226 │ 0 │ 0 │ 0.05000000000001137 │ 0
CMCSA │ 2022-11-09 17:00:00-05 │ 31.38 │ 31.39 │ 31.38 │ 31.39 │ 31.39 │ 31.39 │ 31.39 │ 31.39 │ 0.010000000000001563 │ 0 │ 0.010000000000001563 │ 0
CSCO │ 2022-11-09 17:00:00-05 │ 43.94 │ 44 │ 43.88 │ 43.91 │ 43.88 │ 44 │ 43.88 │ 43.99 │ 0.05999999999999517 │ 0 │ 0 │ 0.0800000000000054
CVX │ 2022-11-09 17:00:00-05 │ 178.15 │ 178.25 │ 177.93 │ 177.98 │ 178 │ 178.25 │ 177.98 │ 177.98 │ 0.15000000000000568 │ 0 │ 0.04999999999998295 │ 0
DIS │ 2022-11-09 17:00:00-05 │ 86.53 │ 134.72 │ 86.41 │ 86.43 │ 86.53 │ 86.53 │ 86.43 │ 86.43 │ 0 │ 48.19 │ 0.020000000000010232 │ 0
GE │ 2022-11-09 17:00:00-05 │ 83.1 │ 83.56 │ 83.1 │ 83.15 │ 83.1 │ 83.35 │ 83.1 │ 83.15 │ 0 │ 0.21000000000000796 │ 0 │ 0
GILD │ 2022-11-09 17:00:00-05 │ 82.28 │ 82.79 │ 82.27 │ 82.5 │ 82.28 │ 82.79 │ 82.27 │ 82.27 │ 0 │ 0 │ 0 │ 0.23000000000000398
GM │ 2022-11-09 17:00:00-05 │ 38.35 │ 38.35 │ 37.97 │ 38.03 │ 38.01 │ 38.21 │ 38.01 │ 38.03 │ 0.3400000000000034 │ 0.14000000000000057 │ 0.03999999999999915 │ 0
GOOG │ 2022-11-09 17:00:00-05 │ 87.4 │ 87.465 │ 87.25 │ 87.27 │ 87.4 │ 87.46 │ 87.25 │ 87.3088 │ 0 │ 0.005000000000009663 │ 0 │ 0.03880000000000905
HD │ 2022-11-09 17:00:00-05 │ 286.75 │ 286.75 │ 285 │ 285.17 │ 286.75 │ 286.75 │ 285.55 │ 286.11 │ 0 │ 0 │ 0.5500000000000114 │ 0.9399999999999977
IBM │ 2022-11-09 17:00:00-05 │ 136.95 │ 137.39 │ 136.95 │ 137.3 │ 136.95 │ 137.3 │ 136.95 │ 137.3 │ 0 │ 0.08999999999997499 │ 0 │ 0
INTC │ 2022-11-09 17:00:00-05 │ 27.62 │ 27.65 │ 27.52 │ 27.6 │ 27.64 │ 27.65 │ 27.56 │ 27.6 │ 0.019999999999999574 │ 0 │ 0.03999999999999915 │ 0
JNJ │ 2022-11-09 17:00:00-05 │ 172.3 │ 172.67 │ 172 │ 172.45 │ 172.3 │ 172.67 │ 172 │ 172.67 │ 0 │ 0 │ 0 │ 0.21999999999999886
MMM │ 2022-11-09 17:00:00-05 │ 123.83 │ 124.18 │ 123.83 │ 124 │ 123.83 │ 124.18 │ 123.83 │ 124.18 │ 0 │ 0 │ 0 │ 0.18000000000000682
MRK │ 2022-11-09 17:00:00-05 │ 101.55 │ 101.6 │ 88.5 │ 101.2 │ 101.55 │ 101.58 │ 88.5 │ 88.5 │ 0 │ 0.01999999999999602 │ 0 │ 12.700000000000003
MS │ 2022-11-09 17:00:00-05 │ 83.35 │ 83.48 │ 83.27 │ 83.48 │ 83.35 │ 83.35 │ 83.27 │ 83.27 │ 0 │ 0.13000000000000966 │ 0 │ 0.21000000000000796
MSFT │ 2022-11-09 17:00:00-05 │ 225 │ 225.26 │ 224.51 │ 224.75 │ 225 │ 225.26 │ 224.65 │ 224.75 │ 0 │ 0 │ 0.14000000000001478 │ 0
MU │ 2022-11-09 17:00:00-05 │ 56.02 │ 56.1 │ 55.93 │ 56 │ 56.02 │ 56.02 │ 55.98 │ 56 │ 0 │ 0.0799999999999983 │ 0.04999999999999716 │ 0
NFLX │ 2022-11-09 17:00:00-05 │ 254.58 │ 255.65 │ 254.2 │ 255.1 │ 254.25 │ 255.65 │ 254.2 │ 255.1 │ 0.3300000000000125 │ 0 │ 0 │ 0
NVDA │ 2022-11-09 17:00:00-05 │ 137.4 │ 137.76 │ 137.4 │ 137.47 │ 137.59 │ 137.76 │ 137.4 │ 137.47 │ 0.18999999999999773 │ 0 │ 0 │ 0
ORCL │ 2022-11-09 17:00:00-05 │ 75.41 │ 75.54 │ 75.4 │ 75.4 │ 75.41 │ 75.41 │ 75.4 │ 75.4 │ 0 │ 0.13000000000000966 │ 0 │ 0
PFE │ 2022-11-09 17:00:00-05 │ 46.72 │ 46.75 │ 46.68 │ 46.75 │ 46.72 │ 46.75 │ 46.68 │ 46.68 │ 0 │ 0 │ 0 │ 0.07000000000000028
PYPL │ 2022-11-09 17:00:00-05 │ 78.97 │ 78.97 │ 78.5 │ 78.52 │ 78.5956 │ 78.97 │ 78.52 │ 78.52 │ 0.3743999999999943 │ 0 │ 0.01999999999999602 │ 0
QCOM │ 2022-11-09 17:00:00-05 │ 110.55 │ 110.82 │ 110.53 │ 110.59 │ 110.54 │ 110.82 │ 110.54 │ 110.59 │ 0.009999999999990905 │ 0 │ 0.010000000000005116 │ 0
SNAP │ 2022-11-09 17:00:00-05 │ 9.51 │ 9.54 │ 9.5 │ 9.53 │ 9.5 │ 9.54 │ 9.5 │ 9.53 │ 0.009999999999999787 │ 0 │ 0 │ 0
SQ │ 2022-11-09 17:00:00-05 │ 57.11 │ 57.4 │ 57 │ 57.28 │ 57.11 │ 57.4 │ 57.01 │ 57.28 │ 0 │ 0 │ 0.00999999999999801 │ 0
T │ 2022-11-09 17:00:00-05 │ 18.39 │ 18.4 │ 18.35 │ 18.36 │ 18.39 │ 18.39 │ 18.35 │ 18.36 │ 0 │ 0.00999999999999801 │ 0 │ 0
TSLA │ 2022-11-09 17:00:00-05 │ 176.505 │ 176.99 │ 175.55 │ 176.77 │ 176.505 │ 176.85 │ 175.625 │ 176.74 │ 0 │ 0.14000000000001478 │ 0.07499999999998863 │ 0.030000000000001137
UBER │ 2022-11-09 17:00:00-05 │ 26.58 │ 26.58 │ 26.5 │ 26.55 │ 26.56 │ 26.58 │ 26.5 │ 26.55 │ 0.019999999999999574 │ 0 │ 0 │ 0
V │ 2022-11-09 17:00:00-05 │ 193.17 │ 194.25 │ 193.17 │ 194.25 │ 194.25 │ 194.25 │ 193.3 │ 194.25 │ 1.0800000000000125 │ 0 │ 0.13000000000002387 │ 0
VZ │ 2022-11-09 17:00:00-05 │ 37.64 │ 37.71 │ 37.62 │ 37.7 │ 37.65 │ 37.71 │ 37.62 │ 37.7 │ 0.00999999999999801 │ 0 │ 0 │ 0
ZM │ 2022-11-09 17:00:00-05 │ 71.96 │ 71.96 │ 71.5533 │ 71.84 │ 71.9 │ 71.9 │ 71.5533 │ 71.84 │ 0.05999999999998806 │ 0.05999999999998806 │ 0 │ 0
AAPL │ 2022-11-09 18:00:00-05 │ 134.83 │ 136.6601 │ 134.83 │ 135.05 │ 135.105 │ 135.18 │ 134.87 │ 135.05 │ 0.27499999999997726 │ 1.480099999999993 │ 0.03999999999999204 │ 0
ABNB │ 2022-11-09 18:00:00-05 │ 95.61 │ 95.78 │ 95.51 │ 95.68 │ 95.68 │ 95.68 │ 95.56 │ 95.65 │ 0.07000000000000739 │ 0.09999999999999432 │ 0.04999999999999716 │ 0.030000000000001137
AMAT │ 2022-11-09 18:00:00-05 │ 94.29 │ 94.32 │ 94.21 │ 94.21 │ 94.29 │ 94.32 │ 94.29 │ 94.32 │ 0 │ 0 │ 0.0800000000000125 │ 0.10999999999999943
AMD │ 2022-11-09 18:00:00-05 │ 59.92 │ 59.96 │ 59.76 │ 59.82 │ 59.92 │ 59.96 │ 59.76 │ 59.85 │ 0 │ 0 │ 0 │ 0.030000000000001137
AMZN │ 2022-11-09 18:00:00-05 │ 86.15 │ 86.28 │ 86.15 │ 86.17 │ 86.2 │ 86.28 │ 86.15 │ 86.17 │ 0.04999999999999716 │ 0 │ 0 │ 0
BA │ 2022-11-09 18:00:00-05 │ 168.5 │ 168.85 │ 168.5 │ 168.52 │ 168.5 │ 168.7 │ 168.5 │ 168.52 │ 0 │ 0.15000000000000568 │ 0 │ 0
BAC │ 2022-11-09 18:00:00-05 │ 36.48 │ 36.53 │ 36.45 │ 36.46 │ 36.52 │ 36.53 │ 36.46 │ 36.46 │ 0.04000000000000625 │ 0 │ 0.00999999999999801 │ 0
C │ 2022-11-09 18:00:00-05 │ 45.3 │ 45.37 │ 45.22 │ 45.22 │ 45.3 │ 45.37 │ 45.27 │ 45.37 │ 0 │ 0 │ 0.05000000000000426 │ 0.14999999999999858
CSCO │ 2022-11-09 18:00:00-05 │ 43.86 │ 44.1501 │ 43.86 │ 44.1501 │ 44.1501 │ 44.1501 │ 44.1501 │ 44.1501 │ 0.29010000000000247 │ 0 │ 0.29010000000000247 │ 0
CVS │ 2022-11-09 18:00:00-05 │ 99.2 │ 99.79 │ 99.2 │ 99.57 │ 99.2 │ 99.78 │ 99.2 │ 99.57 │ 0 │ 0.010000000000005116 │ 0 │ 0
CVX │ 2022-11-09 18:00:00-05 │ 177.98 │ 178 │ 177.55 │ 177.77 │ 177.98 │ 177.98 │ 177.7 │ 177.77 │ 0 │ 0.020000000000010232 │ 0.14999999999997726 │ 0
GE │ 2022-11-09 18:00:00-05 │ 83.15 │ 83.35 │ 83.1 │ 83.11 │ 83.1 │ 83.1 │ 83.1 │ 83.1 │ 0.05000000000001137 │ 0.25 │ 0 │ 0.010000000000005116
GM │ 2022-11-09 18:00:00-05 │ 38.03 │ 38.23 │ 38 │ 38 │ 38.01 │ 38.22 │ 38 │ 38 │ 0.020000000000003126 │ 0.00999999999999801 │ 0 │ 0
GOOG │ 2022-11-09 18:00:00-05 │ 87.33 │ 87.44 │ 87.25 │ 87.39 │ 87.33 │ 87.44 │ 87.25 │ 87.345 │ 0 │ 0 │ 0 │ 0.045000000000001705
HD │ 2022-11-09 18:00:00-05 │ 286.25 │ 287.19 │ 286.02 │ 286.41 │ 286.25 │ 286.41 │ 286.02 │ 286.41 │ 0 │ 0.7799999999999727 │ 0 │ 0
IBM │ 2022-11-09 18:00:00-05 │ 137.35 │ 138.15 │ 137.35 │ 137.5 │ 137.5 │ 138.15 │ 137.5 │ 137.5 │ 0.15000000000000568 │ 0 │ 0.15000000000000568 │ 0
INTC │ 2022-11-09 18:00:00-05 │ 27.83 │ 27.83 │ 27.55 │ 27.56 │ 27.62 │ 27.69 │ 27.57 │ 27.58 │ 0.2099999999999973 │ 0.13999999999999702 │ 0.019999999999999574 │ 0.019999999999999574
JPM │ 2022-11-09 18:00:00-05 │ 129.67 │ 129.74 │ 129.57 │ 129.58 │ 129.57 │ 129.74 │ 129.57 │ 129.74 │ 0.09999999999999432 │ 0 │ 0 │ 0.1599999999999966
KO │ 2022-11-09 18:00:00-05 │ 58.8 │ 58.8 │ 58.7 │ 58.8 │ 58.8 │ 58.8 │ 58.7 │ 58.78 │ 0 │ 0 │ 0 │ 0.01999999999999602
MRK │ 2022-11-09 18:00:00-05 │ 101.6 │ 101.6 │ 101 │ 101.6 │ 101.24 │ 101.6 │ 101.24 │ 101.6 │ 0.35999999999999943 │ 0 │ 0.23999999999999488 │ 0
MS │ 2022-11-09 18:00:00-05 │ 83.45 │ 83.45 │ 83.12 │ 83.12 │ 83.45 │ 83.45 │ 83.27 │ 83.45 │ 0 │ 0 │ 0.14999999999999147 │ 0.3299999999999983
NFLX │ 2022-11-09 18:00:00-05 │ 255.1 │ 255.49 │ 254.7 │ 254.8 │ 255.1 │ 255.49 │ 254.7 │ 254.78 │ 0 │ 0 │ 0 │ 0.020000000000010232
NKE │ 2022-11-09 18:00:00-05 │ 92.2 │ 92.5 │ 92 │ 92.5 │ 92 │ 92.5 │ 92 │ 92.5 │ 0.20000000000000284 │ 0 │ 0 │ 0
NVDA │ 2022-11-09 18:00:00-05 │ 137.46 │ 137.58 │ 137.17 │ 137.175 │ 137.46 │ 137.56 │ 137.2 │ 137.265 │ 0 │ 0.020000000000010232 │ 0.030000000000001137 │ 0.08999999999997499
PG │ 2022-11-09 18:00:00-05 │ 136.3 │ 137.44 │ 135.87 │ 136.32 │ 136.48 │ 136.48 │ 135.87 │ 135.87 │ 0.1799999999999784 │ 0.960000000000008 │ 0 │ 0.44999999999998863
SNAP │ 2022-11-09 18:00:00-05 │ 9.54 │ 9.54 │ 9.45 │ 9.46 │ 9.54 │ 9.54 │ 9.45 │ 9.45 │ 0 │ 0 │ 0 │ 0.010000000000001563
SQ │ 2022-11-09 18:00:00-05 │ 57.28 │ 57.59 │ 57.17 │ 57.3 │ 57.4 │ 57.5 │ 57.21 │ 57.3 │ 0.11999999999999744 │ 0.09000000000000341 │ 0.03999999999999915 │ 0
T │ 2022-11-09 18:00:00-05 │ 18.36 │ 18.39 │ 18.34 │ 18.36 │ 18.36 │ 18.39 │ 18.34 │ 18.37 │ 0 │ 0 │ 0 │ 0.010000000000001563
TSLA │ 2022-11-09 18:00:00-05 │ 176.75 │ 189.2101 │ 176.71 │ 176.74 │ 176.86 │ 177.19 │ 176.71 │ 176.74 │ 0.11000000000001364 │ 12.020100000000014 │ 0 │ 0
V │ 2022-11-09 18:00:00-05 │ 194.57 │ 196.52 │ 193.3 │ 193.3 │ 194 │ 194 │ 193.3 │ 193.3 │ 0.5699999999999932 │ 2.5200000000000102 │ 0 │ 0
VZ │ 2022-11-09 18:00:00-05 │ 37.66 │ 37.7 │ 37.6 │ 37.6 │ 37.7 │ 37.7 │ 37.6 │ 37.6 │ 0.04000000000000625 │ 0 │ 0 │ 0
WFC │ 2022-11-09 18:00:00-05 │ 45.98 │ 45.98 │ 45.71 │ 45.94 │ 45.98 │ 45.98 │ 45.71 │ 45.95 │ 0 │ 0 │ 0 │ 0.010000000000005116
XOM │ 2022-11-09 18:00:00-05 │ 108.97 │ 111.03 │ 108.92 │ 109.17 │ 108.97 │ 111.03 │ 108.92 │ 108.92 │ 0 │ 0 │ 0 │ 0.25
(888 rows)
This change restores the combine and (de)serialization code for ohlc. It also fixes a bug where we were returning a pointer to the object in shared memory in the combine function when the other input was null.
WireBaron
force-pushed
the
br/ohlc_fix
branch
from
4abf318 to
5d0cad0
Compare
I agree with you, but don't see an easy way to reproduce this short of running a query like Ryan's above on a dataset with millions of rows. The error here requires not only running the aggregate in parallel, but also having the system busy enough that another worker has chance to scribble over the shared memory we're returning before it gets used. |
|
bors r+ |
|
Build succeeded: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change restores the combine and (de)serialization code for ohlc. It also fixes a bug where we were returning a pointer to the object in shared memory in the combine function when the other input was null.
Fixes #611