Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........XX..........................X.X..............X......] duration: 5.709500451s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:23:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.709528925s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........XX..........................X.X..............X.....] duration: 6.031167984s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:22:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.031195206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........XX..........................X.X..............X....] duration: 5.835131941s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:21:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.835159404s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........XX..........................X.X..............X...] duration: 5.973028988s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:21:33 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.973117336s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [............XX..........................X.X..............X..] duration: 6.084679108s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:21:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 6.084775191s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.............XX..........................X.X..............X.] duration: 5.723335225s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:20:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 5.723354551s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..............XX..........................X.X..............X] duration: 30.002707164s trace: https://jaeger.dev.storj.io/trace/2dfeb311a79b59a3 FAIL (2025-12-25 10:20:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi, Piece ID: 3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q) context deadline exceeded; piecestore: (Node ID: 12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, Piece ID: R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A) context deadline exceeded ======LOGS====== s) [upload=416812]: Uploading segments... Dec 25 10:19:58.807596555 (474h50m58.67867959s) [upload=416812]: Got next segment. Inline: false Dec 25 10:19:58.807646460 (474h50m58.678729495s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:19:58.807651520 (474h50m58.678734555s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:19:59.051955352 (474h50m58.923038397s) [seg_pos={0 0}, upload=416812]: Begin upload segment... Dec 25 10:19:59.052038109 (474h50m58.923121154s) [seg_pos={0 0}, upload=416812]: Done begin upload segment. Dec 25 10:19:59.052102222 (474h50m58.923185277s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.052190549 (474h50m58.923273624s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053330851 (474h50m58.924413896s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053511314 (474h50m58.924594349s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.053925423 (474h50m58.925008468s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.054878979 (474h50m58.925962014s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.055273921 (474h50m58.926356966s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.056243658 (474h50m58.927326693s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057567889 (474h50m58.928650934s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.057732633 (474h50m58.928815668s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.058757855 (474h50m58.92984089s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.059015485 (474h50m58.93009852s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.061225053 (474h50m58.932308098s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.062213284 (474h50m58.933296329s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.063136833 (474h50m58.934219878s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:19:59.065156199 (474h50m58.936239234s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Uploading piece... Dec 25 10:20:00.764473536 (474h51m0.635556571s) [upload=416812]: Next returned nil segment Dec 25 10:20:00.764496931 (474h51m0.635579966s) [upload=416812]: Waiting for error group managing segments... Dec 25 10:20:01.053971585 (474h51m0.92505463s) [noise=true , address=145.79.227.52:4445 , piece_id=ZJPZJY7CUBUVX5N46KDDT5Y2UU6ZOQ2E4533D2N7MPUXR43NX2NQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.059821905 (474h51m0.93090494s) [noise=true , address=145.79.227.37:4566 , piece_id=UQQR5KJNI6WUOILPNQPMTCOEEG3GK7FHR4LBHXOGV5FGTGMK5AMQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.060417810 (474h51m0.931500845s) [noise=true , address=145.79.227.38:4979 , piece_id=5CB3L7XHLDTQOE45VW3BG5GFZ23VN5C7CFYUVIRX7QQPQUF7XLVQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.062429070 (474h51m0.933512115s) [noise=true , address=145.79.227.45:4229 , piece_id=JDZZEUV4TKAI53MXHX5BEWTAH5PJQCO2QS3G4EYOU4S4RNQ2GGLQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.079029619 (474h51m0.950112654s) [noise=true , address=145.79.227.49:4337 , piece_id=SSEFAQ3TFN2FPOKC3SQ4SBPX2LC6R456PJCLRUYFHXSTSZNATAZQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.080817855 (474h51m0.95190089s) [noise=true , address=145.79.227.20:4437 , piece_id=Q3WYHQP5ZNPOVJ6LCNZT7G7YQSWY5FWSDGJJ3NK2KAUMXN4HZDVA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.082831960 (474h51m0.953915005s) [noise=true , address=145.79.227.5:4221 , piece_id=2JEJEFOBVAYMXRBQ36Z36STNEJBANUU4PGXCA62GL2RFCJI63JGA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.083011462 (474h51m0.954094517s) [noise=true , address=145.79.227.17:4970 , piece_id=MVRUZOI3PXNXR2HVLYNB6CZ2HITYI2J4JIPBFTHVNTJ2JIISAC5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.084581852 (474h51m0.955664897s) [noise=true , address=145.79.227.47:4265 , piece_id=S5I3A256ZRNQ2W2ED5GYU3AD2W6MC4GLM2RTPVGX7KSWJIO4EU5Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.094076711 (474h51m0.965159756s) [noise=true , address=145.79.227.29:4941 , piece_id=7L6SHXDLPA7WE7JIGEGVT6HOZ6OUCUEHZ475B6CFDT4Q6RD7AUXA, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.391734257 (474h51m1.262817292s) [noise=true , address=45.91.116.12:4227 , piece_id=R2N5YKFWWALLMLQB2SU72ZAZDTOGPZ52FWYP6HJNPRX26DSOK64A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.392297670 (474h51m1.263380715s) [noise=true , address=45.91.116.4:4939 , piece_id=BTXYPWZF7LRSUJAXTEFCRXM3DGNWGVHUFXETJEOSXVSIGXGVRAPQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476090513 (474h51m1.347173549s) [noise=true , address=45.91.116.9:4261 , piece_id=3KKVMKGWI6ELHI3CKX64FT54NCA45TQHLX7TPYMALAN7G7K7WB3Q, seg_pos={0 0}, upload=416812]: Done uploading piece. err: <nil> Dec 25 10:20:01.476118287 (474h51m1.347201332s) [seg_pos={0 0}, upload=416812]: Segment reached optimal threshold of 13 pieces. Dec 25 10:20:01.476358354 (474h51m1.347441389s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f): protocol: expected piece hash; context canceled; EOF Dec 25 10:20:01.476367822 (474h51m1.347450867s) [noise=true , address=77.90.31.13:4629 , piece_id=EP6BDULMBQMPYSTDE7MXUWJR7VGKATKZR5YHAT2TMXD3LYD5YIQQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476553265 (474h51m1.34763631s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz): protocol: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40824->77.90.31.11:4553: use of closed network connection Dec 25 10:20:01.476572652 (474h51m1.347655697s) [noise=true , address=77.90.31.11:4553 , piece_id=PPKGIRFTN5NWDYZRFXCMLVEVX7RC2QUUUWY2EKVOQQ7X4PTE2W5A, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476747294 (474h51m1.347830339s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49006->45.91.116.14:4189: use of closed network connection Dec 25 10:20:01.476763225 (474h51m1.34784627s) [noise=true , address=45.91.116.14:4189 , piece_id=JPMX2NDK5O2EAHVCQUPCCZLPLLFBEJYTAS4G5YBHHUR2ZGMCQ3BQ, seg_pos={0 0}, upload=416812]: Long tail cancellation due to optimal threshold Dec 25 10:20:01.476805465 (474h51m1.34788852s) [seg_pos={0 0}, upload=416812]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:20:01.476917869 (474h51m1.348000914s) [upload=416812]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:20:01.476924482 (474h51m1.348007527s) [upload=416812]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:20:01.761672034 (474h51m1.632755089s) [upload=416812]: Done uploading segments... Dec 25 10:20:01.761698464 (474h51m1.632781509s) [upload=416812]: Upload finished. err: <nil> Duration: 30.002731349s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [...............XX..........................X.X..............] duration: 5.981572234s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:19:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.981602001s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X...............XX..........................X.X.............] duration: 5.977083662s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:18:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.977112857s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X...............XX..........................X.X............] duration: 6.001574564s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:18:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 6.001597226s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...............XX..........................X.X...........] duration: 6.025802096s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:17:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 6.025828595s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...............XX..........................X.X..........] duration: 5.781995798s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:16:57 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.78202294s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X...............XX..........................X.X.........] duration: 5.822628826s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:16:15 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.822659826s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X...............XX..........................X.X........] duration: 6.14528593s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:15:02 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 6.14531709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [......X...............XX..........................X.X.......] duration: 5.764942458s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:14:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.764970341s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.......X...............XX..........................X.X......] duration: 5.75887143s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:14:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.758900535s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [........X...............XX..........................X.X.....] duration: 5.797549872s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:13:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.797579848s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.........X...............XX..........................X.X....] duration: 6.063780743s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:13:04 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 6.063834966s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..........X...............XX..........................X.X...] duration: 5.670241277s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:11:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.670266504s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...........X...............XX..........................X.X..] duration: 5.860261754s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:10:45 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.860287682s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X...........X...............XX..........................X.X.] duration: 5.784729248s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:09:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 5.78474591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X...........X...............XX..........................X.X] duration: 30.003896876s trace: https://jaeger.dev.storj.io/trace/6875ff7990e6b816 FAIL (2025-12-25 10:09:29 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 6645]: Starting upload Dec 25 10:09:00.426750214 (474h40m0.297833259s) [upload=416645]: Done starting upload Dec 25 10:09:00.426936027 (474h40m0.298019072s) [upload=416645]: Uploading segments... Dec 25 10:09:00.426970222 (474h40m0.298053258s) [upload=416645]: Got next segment. Inline: false Dec 25 10:09:00.426995661 (474h40m0.298078706s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:09:00.426999649 (474h40m0.298082694s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:09:00.674148378 (474h40m0.545231423s) [seg_pos={0 0}, upload=416645]: Begin upload segment... Dec 25 10:09:00.674195809 (474h40m0.545278854s) [seg_pos={0 0}, upload=416645]: Done begin upload segment. Dec 25 10:09:00.674226217 (474h40m0.545309252s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674304005 (474h40m0.54538705s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.674850506 (474h40m0.545933551s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675167188 (474h40m0.546250254s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675553113 (474h40m0.546636148s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.675894283 (474h40m0.546977318s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676189194 (474h40m0.547272229s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676552266 (474h40m0.547635311s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.676963468 (474h40m0.548046513s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.678438107 (474h40m0.549521152s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.679900231 (474h40m0.550983276s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.681148398 (474h40m0.552231443s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.682492958 (474h40m0.553576003s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.683868697 (474h40m0.554951742s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.685251841 (474h40m0.556334886s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:00.686492023 (474h40m0.557575058s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Uploading piece... Dec 25 10:09:02.310368671 (474h40m2.181451716s) [upload=416645]: Next returned nil segment Dec 25 10:09:02.310387937 (474h40m2.181470983s) [upload=416645]: Waiting for error group managing segments... Dec 25 10:09:02.717525551 (474h40m2.588608596s) [noise=true , address=145.79.227.14:4838 , piece_id=KDE7CB3DKPUA62ZPJUSFDMXZN5CFK7Y77YRO74MXQQZZGNHV4D7A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.718054288 (474h40m2.589137333s) [noise=true , address=145.79.227.5:4221 , piece_id=AMWPPNDETISESMNPQZDDIAOCGBHB2BJMZK4OCVDBURWZZIW4XGDQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.720448978 (474h40m2.591532033s) [noise=true , address=145.79.227.51:4411 , piece_id=RZPR26GZIIZURI2EQKTDV7CYKBC3YD7N2ZZDHA5PCMNSU2APM45A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732284444 (474h40m2.603367489s) [noise=true , address=145.79.227.25:4618 , piece_id=U7Y6YNCSZPRCHJXDA64R2AYMQN3SAZIS76CAGSN4D6PA7CDS4W5A, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732404182 (474h40m2.603487227s) [noise=true , address=145.79.227.7:4295 , piece_id=W77JNI3XYCUD65H3SMBRQY4JKH6WH3TU2N7LFQGHDKW77OAGXVEA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.732983595 (474h40m2.60406664s) [noise=true , address=145.79.227.16:4910 , piece_id=22IAGWI5IZLB7I2B4LL3QCDQZX4DV54NQC44YHLYKWD7A2HHAMOQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733053638 (474h40m2.604136683s) [noise=true , address=145.79.227.43:4157 , piece_id=NDI4SEPD2OMR23UZJY425AIICGL3AEMUIZIZ3VQ3GUAJ6B3QFTXA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.733717463 (474h40m2.604800518s) [noise=true , address=145.79.227.38:4979 , piece_id=Z6V57VN3TAC42WCAEXM3W73P32EIGRZ7OQSOCGDGJAUEF2VAFVIA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.734237743 (474h40m2.605320788s) [noise=true , address=145.79.227.50:4375 , piece_id=EQH7JQWZTJXFJW2K3GR4F4ROTKDNFGW2T4YEZ35BHZH26EXUXLCQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.759370821 (474h40m2.630453886s) [noise=true , address=145.79.227.26:4655 , piece_id=YZWQ2S3UNYMNGSS3VVU54UVWIENMCBCKMUIKLC67PP4I36V6GK6Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.783530896 (474h40m2.654613941s) [noise=true , address=145.79.227.4:4185 , piece_id=Y7OFEQFEFX3T4OF2VOQNBFM6UIJOY6MFTADNCMIIGCTFUOASQW7Q, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:02.965100421 (474h40m2.836183466s) [noise=true , address=45.91.116.6:4974 , piece_id=G2IUDPJC6ZIOT2NDZF4DXWRJKOZQKFIWQN4MPPOHPRX6H6ICPODQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036653279 (474h40m2.907736324s) [noise=true , address=77.90.31.18:4808 , piece_id=BJ7X2N2B7IOXSIG5H2DDBEUK2FW7RGEHPI5DMURHWDP2XXC2OLYA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: <nil> Dec 25 10:09:03.036685410 (474h40m2.907768455s) [seg_pos={0 0}, upload=416645]: Segment reached optimal threshold of 13 pieces. Dec 25 10:09:03.037330078 (474h40m2.908413133s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51774->77.90.31.10:4520: use of closed network connection Dec 25 10:09:03.037344615 (474h40m2.90842766s) [noise=true , address=77.90.31.10:4520 , piece_id=UFVPNRJ6ABACSWGLLW3VOG6D6TOJWGIKCBWWBN3WUKSRB6UZ2YAA, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.037908078 (474h40m2.908991163s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: read tcp 188.166.225.170:40778->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 25 10:09:03.037920753 (474h40m2.909003798s) [noise=true , address=45.91.116.11:4119 , piece_id=NPGJJ5GHTMSXJX4BLFQCXWYWTEWQ7NDMA64UL7QNDDZSIAOA45PQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038165178 (474h40m2.909248223s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: expected piece hash; context canceled; EOF Dec 25 10:09:03.038208601 (474h40m2.909291646s) [noise=true , address=45.91.116.12:4228 , piece_id=SBQT2464EXCU5MQLV5OFJ5OIEAYY27J7HPWHIGLYMRZ2VJRVRHUQ, seg_pos={0 0}, upload=416645]: Long tail cancellation due to optimal threshold Dec 25 10:09:03.038241954 (474h40m2.909324999s) [seg_pos={0 0}, upload=416645]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:09:03.038420314 (474h40m2.909503369s) [upload=416645]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:09:03.038430874 (474h40m2.909513919s) [upload=416645]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:09:03.339547806 (474h40m3.210630861s) [upload=416645]: Done uploading segments... Dec 25 10:09:03.339579046 (474h40m3.210662091s) [upload=416645]: Upload finished. err: <nil> Duration: 30.003918438s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X...........X...............XX..........................X.] duration: 6.125205265s trace: https://jaeger.dev.storj.io/trace/38cbe59e5e3bb35c FAIL (2025-12-25 10:08:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A) context deadline exceeded ======LOGS====== (474h38m5.704587591s) [upload=416618]: Uploading segments... Dec 25 10:07:05.833742549 (474h38m5.704825584s) [upload=416618]: Got next segment. Inline: false Dec 25 10:07:05.834408427 (474h38m5.705491482s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:07:05.834417854 (474h38m5.705500889s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:07:06.081355973 (474h38m5.952439018s) [seg_pos={0 0}, upload=416618]: Begin upload segment... Dec 25 10:07:06.081422761 (474h38m5.952505806s) [seg_pos={0 0}, upload=416618]: Done begin upload segment. Dec 25 10:07:06.081473036 (474h38m5.952556081s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.081510076 (474h38m5.952593121s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082120720 (474h38m5.953203755s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082511433 (474h38m5.953594468s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083013499 (474h38m5.954096544s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083436375 (474h38m5.95451941s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083811178 (474h38m5.954894223s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084218294 (474h38m5.955301339s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084612624 (474h38m5.955695669s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086269009 (474h38m5.957352054s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086516650 (474h38m5.957599695s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.088408151 (474h38m5.959491186s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.090281689 (474h38m5.961364734s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.091837741 (474h38m5.962920776s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.093299495 (474h38m5.96438254s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.094836873 (474h38m5.965919908s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:07.807999056 (474h38m7.679082091s) [upload=416618]: Next returned nil segment Dec 25 10:07:07.808035075 (474h38m7.67911811s) [upload=416618]: Waiting for error group managing segments... Dec 25 10:07:08.095233103 (474h38m7.966316148s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.101219963 (474h38m7.972302998s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103194153 (474h38m7.974277198s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103253335 (474h38m7.97433637s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.113152104 (474h38m7.984235149s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.116079047 (474h38m7.987162093s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.120584797 (474h38m7.991667832s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.125144830 (474h38m7.996227875s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.128055221 (474h38m7.999138256s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.321456498 (474h38m8.192539533s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.372613558 (474h38m8.243696593s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.383542358 (474h38m8.254625393s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449614751 (474h38m8.320697786s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449638396 (474h38m8.320721431s) [seg_pos={0 0}, upload=416618]: Segment reached optimal threshold of 13 pieces. Dec 25 10:07:08.450036203 (474h38m8.321119268s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection Dec 25 10:07:08.450047766 (474h38m8.321130811s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450101538 (474h38m8.321184583s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 10:07:08.450125233 (474h38m8.321208278s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450326667 (474h38m8.321409712s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection Dec 25 10:07:08.450335664 (474h38m8.321418699s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450356743 (474h38m8.321439798s) [seg_pos={0 0}, upload=416618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:07:08.450457124 (474h38m8.321540159s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:07:08.450462344 (474h38m8.321545389s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:07:08.741746594 (474h38m8.612829669s) [upload=416618]: Done uploading segments... Dec 25 10:07:08.741774858 (474h38m8.612857893s) [upload=416618]: Upload finished. err: <nil> Duration: 6.125245291s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X...........X...............XX..........................X] duration: 30.002440363s trace: https://jaeger.dev.storj.io/trace/38cbe59e5e3bb35c FAIL (2025-12-25 10:07:35 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, Piece ID: FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA) context deadline exceeded; piecestore: (Node ID: 1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w, Piece ID: U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: 6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A) context deadline exceeded ======LOGS====== (474h38m5.704587591s) [upload=416618]: Uploading segments... Dec 25 10:07:05.833742549 (474h38m5.704825584s) [upload=416618]: Got next segment. Inline: false Dec 25 10:07:05.834408427 (474h38m5.705491482s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 10:07:05.834417854 (474h38m5.705500889s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 10:07:06.081355973 (474h38m5.952439018s) [seg_pos={0 0}, upload=416618]: Begin upload segment... Dec 25 10:07:06.081422761 (474h38m5.952505806s) [seg_pos={0 0}, upload=416618]: Done begin upload segment. Dec 25 10:07:06.081473036 (474h38m5.952556081s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.081510076 (474h38m5.952593121s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082120720 (474h38m5.953203755s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.082511433 (474h38m5.953594468s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083013499 (474h38m5.954096544s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083436375 (474h38m5.95451941s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.083811178 (474h38m5.954894223s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084218294 (474h38m5.955301339s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.084612624 (474h38m5.955695669s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086269009 (474h38m5.957352054s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.086516650 (474h38m5.957599695s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.088408151 (474h38m5.959491186s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.090281689 (474h38m5.961364734s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.091837741 (474h38m5.962920776s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.093299495 (474h38m5.96438254s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:06.094836873 (474h38m5.965919908s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Uploading piece... Dec 25 10:07:07.807999056 (474h38m7.679082091s) [upload=416618]: Next returned nil segment Dec 25 10:07:07.808035075 (474h38m7.67911811s) [upload=416618]: Waiting for error group managing segments... Dec 25 10:07:08.095233103 (474h38m7.966316148s) [noise=true , address=145.79.227.6:4257 , piece_id=PRXJHTCNQN4DLPM6TEAROLKLT3DBVBGI2OGNN63MHZ7UE27XVHAA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.101219963 (474h38m7.972302998s) [noise=true , address=145.79.227.31:4799 , piece_id=3FLVVMRVTTBQ5RFP2ZIVBA7PCPT5AV3SWYMG54PZBZVRTFPJK72Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103194153 (474h38m7.974277198s) [noise=true , address=145.79.227.32:4835 , piece_id=J5ZKTZUJWC6O3ESTPUNBIC5OITJHCZC7HJLYQVAFLRPGFZ6LLQFA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.103253335 (474h38m7.97433637s) [noise=true , address=145.79.227.35:4481 , piece_id=U2WAS6ZXF63N6ZNR734BX3CMK7RKGD6SB6A3PMFA5YDFAQTHGL6Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.113152104 (474h38m7.984235149s) [noise=true , address=145.79.227.43:4159 , piece_id=2AOZ4I755SO4PK4CQGX47HLIBZFOS74BKJ6YTFHBR76U5IC56CLA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.116079047 (474h38m7.987162093s) [noise=true , address=145.79.227.26:4655 , piece_id=HQZ5IQFJ3FFYJZRTJLZJS7OYIVZEC7ZBLUSZGPC2QBROPY4AXW2Q, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.120584797 (474h38m7.991667832s) [noise=true , address=145.79.227.37:4566 , piece_id=SQN374SWPZXMIDUBBIATIADR4QSB2EHKSCTCS7XWYC7RLGPZUHTA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.125144830 (474h38m7.996227875s) [noise=true , address=145.79.227.2:4113 , piece_id=FHICI4S42XXXYFMX3IDJHUYN2Y4QTHD7AHYH4LSZBYOC57XWV7EQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.128055221 (474h38m7.999138256s) [noise=true , address=145.79.227.47:4265 , piece_id=BINYXHDLHUKQ6LJFNGUKOGLEPCFNJBYHVKKYOSACIMDNKUKF2WMQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.321456498 (474h38m8.192539533s) [noise=true , address=77.90.31.10:4517 , piece_id=FT4Y7MM2NZQGZZPATSUJSPRCJINICFSCOUDP3GAZPSXWZZBSWQSA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.372613558 (474h38m8.243696593s) [noise=true , address=77.90.31.12:4590 , piece_id=6J4MZLAAK4YC3LBW25YTDM2UTZE2Y2TYHRN3G3VERSDRDSRBQO4A, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.383542358 (474h38m8.254625393s) [noise=true , address=77.90.31.14:4664 , piece_id=RZYUEUGUCODQFLZPCUSRFRXFM3MZAWHP3MYH5RSCCXOYWDBAHXGQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449614751 (474h38m8.320697786s) [noise=true , address=45.91.116.7:4010 , piece_id=U7RKVHM2YVD5VIBX4YPZJNDCHODUECYT5JNABTQ3CF6FWAPM3TRQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: <nil> Dec 25 10:07:08.449638396 (474h38m8.320721431s) [seg_pos={0 0}, upload=416618]: Segment reached optimal threshold of 13 pieces. Dec 25 10:07:08.450036203 (474h38m8.321119268s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:58148->77.90.31.11:4556: use of closed network connection Dec 25 10:07:08.450047766 (474h38m8.321130811s) [noise=true , address=77.90.31.11:4556 , piece_id=PL3QC6JDVD3RU4PVI5XZVGUR425NJGIYYCZ5BMALU2W4IGQGBNHQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450101538 (474h38m8.321184583s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1n91vwhc11FPE57QAUCfgDpffrmPrJxWxh7R9ecgt41JEPux6J): protocol: expected piece hash; context canceled; EOF Dec 25 10:07:08.450125233 (474h38m8.321208278s) [noise=true , address=145.79.227.40:4049 , piece_id=QGUXRR7ZX5T7VZC4HEOFUNBDAPYZYB2BIJPNVD2W7WTRILBAWFGA, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450326667 (474h38m8.321409712s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59958->45.91.116.5:4298: use of closed network connection Dec 25 10:07:08.450335664 (474h38m8.321418699s) [noise=true , address=45.91.116.5:4298 , piece_id=EK64Z2IMCU6AUGQI5XWWU7VZGLHCUEQLF6GBRMWDBWKIKP6VVNOQ, seg_pos={0 0}, upload=416618]: Long tail cancellation due to optimal threshold Dec 25 10:07:08.450356743 (474h38m8.321439798s) [seg_pos={0 0}, upload=416618]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 10:07:08.450457124 (474h38m8.321540159s) [upload=416618]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 10:07:08.450462344 (474h38m8.321545389s) [upload=416618]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 10:07:08.741746594 (474h38m8.612829669s) [upload=416618]: Done uploading segments... Dec 25 10:07:08.741774858 (474h38m8.612857893s) [upload=416618]: Upload finished. err: <nil> Duration: 30.002462325s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [....X...........X...............XX..........................] duration: 5.978138819s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:06:51 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.978163688s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 3.684280993s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:06:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.684314146s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.....X...........X...............XX.........................] duration: 5.711735785s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:06:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.711812021s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 3.580609147s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:05:55 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.580634245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 3.590713917s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:05:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.590745786s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [......X...........X...............XX........................] duration: 5.824581235s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:05:08 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.824606293s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 3.728241984s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.728295577s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [....X.......................................................] duration: 3.613290958s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:04:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61332353s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.......X...........X...............XX.......................] duration: 5.843032332s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.843064082s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [........X...........X...............XX......................] duration: 5.711445217s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:03:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.711471546s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.....X......................................................] duration: 3.588832393s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:02:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.588859835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [.........X...........X...............XX.....................] duration: 5.678770961s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:02:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.678798764s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [......X.....................................................] duration: 3.730454006s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.730479905s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.......X....................................................] duration: 4.025367151s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:01:15 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.025399814s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [X.........X...........X...............XX....................] duration: 5.99135213s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 10:01:07 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.991546921s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [........X...................................................] duration: 3.415345567s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 10:00:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.415400711s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.X.........X...........X...............XX...................] duration: 6.011706447s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:59:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.011731505s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.........X..................................................] duration: 4.865385278s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:59:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.865408763s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..X.........X...........X...............XX..................] duration: 5.765173446s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.765221669s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [...X.........X...........X...............XX.................] duration: 5.923426475s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.923450791s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..........X.................................................] duration: 3.424342998s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:20 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.424368097s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........X................................................] duration: 3.405510427s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:58:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.40565302s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [....X.........X...........X...............XX................] duration: 6.242879239s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:58:01 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 6.2429111s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [............X...............................................] duration: 4.470727863s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:30 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.47075268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.............X..............................................] duration: 3.430758276s trace: https://jaeger.dev.storj.io/trace/77246dd96451221 FAIL (2025-12-25 09:57:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.430785548s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.....X.........X...........X...............XX...............] duration: 5.672214878s trace: https://jaeger.dev.storj.io/trace/7702ecd24b71b325 FAIL (2025-12-25 09:56:54 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1bAoTamwp534btqQDaG1TdCMJghkj8ZqTQtqJub1X3kp3VoJn9, Piece ID: UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ) context deadline exceeded; piecestore: (Node ID: 124p2J9pP2rYD3cM73PQNEp6Ykgmpq2xi6aK8wnC6JTiLvHRb7q, Piece ID: VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ) context deadline exceeded ======LOGS====== Dec 25 09:45:20.437951270 (474h16m20.309034305s) [upload=416309]: Starting upload Dec 25 09:45:20.438042362 (474h16m20.309125408s) [upload=416309]: Done starting upload Dec 25 09:45:20.438149506 (474h16m20.309232562s) [upload=416309]: Uploading segments... Dec 25 09:45:20.438300695 (474h16m20.30938374s) [upload=416309]: Got next segment. Inline: false Dec 25 09:45:20.438347874 (474h16m20.30943092s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 25 09:45:20.438355970 (474h16m20.309439015s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 25 09:45:20.679920068 (474h16m20.551003103s) [seg_pos={0 0}, upload=416309]: Begin upload segment... Dec 25 09:45:20.679970434 (474h16m20.55105347s) [seg_pos={0 0}, upload=416309]: Done begin upload segment. Dec 25 09:45:20.680034225 (474h16m20.55111729s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680059934 (474h16m20.551142969s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680528237 (474h16m20.551611272s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680831604 (474h16m20.551914639s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.680846753 (474h16m20.551929788s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681140642 (474h16m20.552223677s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681239551 (474h16m20.552322586s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681578226 (474h16m20.552661271s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.681963730 (474h16m20.553046775s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682326550 (474h16m20.553409585s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.682697637 (474h16m20.553780672s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683145620 (474h16m20.554228665s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683554259 (474h16m20.554637294s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.683960271 (474h16m20.555043316s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684326890 (474h16m20.555409925s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:20.684812565 (474h16m20.5558956s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Uploading piece... Dec 25 09:45:22.300884786 (474h16m22.171967831s) [upload=416309]: Next returned nil segment Dec 25 09:45:22.300900435 (474h16m22.17198348s) [upload=416309]: Waiting for error group managing segments... Dec 25 09:45:22.687169245 (474h16m22.55825229s) [noise=true , address=145.79.227.15:4875 , piece_id=5KAARAINJUWPLNBDNUZFHHFHFV7B7RHC6ZUO54WALNV3TSHY3LFA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.691161237 (474h16m22.562244282s) [noise=true , address=145.79.227.2:4115 , piece_id=RULIAHLDMTFGUWFHOPKFYP4CXJ5KWYBLMA2KZCROKRB73K7RIB7A, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.692239921 (474h16m22.563322956s) [noise=true , address=145.79.227.45:4230 , piece_id=SAACOF5ZTPFKYAF4NDME37NAZFZJ4YXCQ52AR7I47DE2COKXKAXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.694106855 (474h16m22.56518989s) [noise=true , address=145.79.227.39:4015 , piece_id=Y7ELCQYCHPTYMQ73OLSXPJKRUZZKPH5EFRDLJF6HAJRNN5UWV5FQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.696779475 (474h16m22.56786252s) [noise=true , address=145.79.227.4:4186 , piece_id=YUZW2D36K3LDHWVVD5GVKG6QJOJ3ZKLGMHKKEAQ27OIC6LUZGKXA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.698030287 (474h16m22.569113322s) [noise=true , address=145.79.227.16:4910 , piece_id=PKM7STQ7MDLYEYSQRSQ7C2MBVZ33RKXFLZONWAW3WR3OYS7D6GLA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.707903486 (474h16m22.578986521s) [noise=true , address=145.79.227.36:4519 , piece_id=Q34QEC7P6U2U7FVLTW6WEV67W2O3NXKLGM3D7CALP4LL4XJ2UMIA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.708086035 (474h16m22.57916907s) [noise=true , address=145.79.227.26:4655 , piece_id=UKOC3YBMZU456LKHGYFRA5HIC3CXDSITHET3ZUFE5LAWFR6LUMBA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.715594852 (474h16m22.586677897s) [noise=true , address=145.79.227.32:4835 , piece_id=YB4EMBZJQNM3BRWFMK4U5UFSJNZCHJK6PUCHCMD5U6NMLXUILNEA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.835861280 (474h16m22.706944325s) [noise=true , address=145.79.227.33:4871 , piece_id=JC7L2LAD54KICN7LTUCF2ZXUU7JJWASZCQANSMTHPBDS5PU3O2CA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.928530287 (474h16m22.799613322s) [noise=true , address=77.90.31.15:4700 , piece_id=VAEDQVMJCZGQACM7VL5XT4K6E6WUTZK75XMIFSIXBWWG6ZMCCQYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:22.987430763 (474h16m22.858513799s) [noise=true , address=77.90.31.18:4808 , piece_id=UQAWJCHV2BFBMGV7KNR2OG3XHMBGPUC4LJ2LINCZOP2S65ZD3DYQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617953975 (474h16m23.48903702s) [noise=true , address=45.91.116.5:4298 , piece_id=M4D7PKVWVQFF44WF4FD535GCUTXKQKGMVNNLR4N36G3CWVIYQDQA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: <nil> Dec 25 09:45:23.617984062 (474h16m23.489067107s) [seg_pos={0 0}, upload=416309]: Segment reached optimal threshold of 13 pieces. Dec 25 09:45:23.618193090 (474h16m23.489276135s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618245079 (474h16m23.489328134s) [noise=true , address=77.90.31.10:4517 , piece_id=NVOIRLVHN6ESAISNGHH2OOEEOFAXMMACMKY2R6SHDUAET6MHOMRA, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618442595 (474h16m23.48952564s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ffKESNSVpXQzbwJinP6vvgE6zoF3RFz2NRUuuJHvd7bvk1gc9): protocol: expected piece hash; context canceled; EOF Dec 25 09:45:23.618451792 (474h16m23.489534827s) [noise=true , address=145.79.227.34:4905 , piece_id=Q5SGG6SD74A3IMWKZ2UBM54Q4DS5NAXI7WOUUXVNYGKBJBN5YIPQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618468965 (474h16m23.489552s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Qsv8fYepBmZ5RYqo42q94CxRqUUmW2xP5abmXwM5GukQRXfPA): protocol: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:40266->45.91.116.12:4228: use of closed network connection Dec 25 09:45:23.618479696 (474h16m23.489562761s) [noise=true , address=45.91.116.12:4228 , piece_id=ZOTK6FNG4FK4OZVNB5TX4UQOUSHVGBHGQD543VI4TFTTCJLMLDXQ, seg_pos={0 0}, upload=416309]: Long tail cancellation due to optimal threshold Dec 25 09:45:23.618501087 (474h16m23.489584132s) [seg_pos={0 0}, upload=416309]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 25 09:45:23.618591869 (474h16m23.489674924s) [upload=416309]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 25 09:45:23.618599022 (474h16m23.489682067s) [upload=416309]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 25 09:45:23.909139961 (474h16m23.780222996s) [upload=416309]: Done uploading segments... Dec 25 09:45:23.909161873 (474h16m23.780244908s) [upload=416309]: Upload finished. err: <nil> Duration: 5.672241218s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)