Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 353086 reports (Page 202 of 7062)
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [....X..X......X....X...X..X.X.X.....X.......X.X.....X....XX.]
duration: 
5.914171391s
trace: 
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:44:54 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732   , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770     , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838   , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518   , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007   , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151    , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943   , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083    , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374   , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654   , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257    , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517     , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410   , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374   , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838   , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732   , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518   , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410   , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654   , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257    , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007   , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151    , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517     , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943   , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770     , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083    , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 5.914187191s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.....X...........X..............XX...X......X..X..X.....X...]
duration: 
3.574508899s
trace: 
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.574537403s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..................X.........................................]
duration: 
11.646484528s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:44:50 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 11.646562756s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.......X...X......X............X............................]
duration: 
33.412758949s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:44:44 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 33.412783015s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [......X...........X..............XX...X......X..X..X.....X..]
duration: 
4.843980631s
trace: 
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:30 UTC) 
orbiter warning: context deadline exceeded
Duration: 4.844010887s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)
antiblip: 
1 out of 60 is failed (threshold=3) [........................................................X...]
duration: 
10.100985992s
trace: 
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:44:22 UTC) 
orbiter warning: 
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031    , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037    , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038    , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030    , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047    , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035    , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 10.101017122s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...................X........................................]
duration: 
15.157566965s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:44:15 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 15.157682206s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [.....X..X......X....X...X..X.X.X.....X.......X.X.....X....XX]
duration: 
30.003050487s
trace: 
https://jaeger.dev.storj.io/trace/78056e8d4b3848bb
FAIL (2025-12-30 06:44:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:43:43.438454167 (591h14m43.309537212s) [upload=518325]: Starting upload
Dec 30 06:43:43.438541793 (591h14m43.309624849s) [upload=518325]: Done starting upload
Dec 30 06:43:43.438670769 (591h14m43.309753824s) [upload=518325]: Uploading segments...
Dec 30 06:43:43.438849339 (591h14m43.309932394s) [upload=518325]: Got next segment. Inline: false
Dec 30 06:43:43.438883805 (591h14m43.30996685s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:43:43.438889526 (591h14m43.309972561s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:43:43.682039201 (591h14m43.553122246s) [seg_pos={0 0}, upload=518325]: Begin upload segment...
Dec 30 06:43:43.682106038 (591h14m43.553189083s) [seg_pos={0 0}, upload=518325]: Done begin upload segment.
Dec 30 06:43:43.682152857 (591h14m43.553235902s) [noise=true , address=145.79.227.11:4732   , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.682258869 (591h14m43.553341914s) [noise=true , address=77.90.31.17:4770     , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683367860 (591h14m43.554450906s) [noise=true , address=145.79.227.14:4838   , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683377639 (591h14m43.554460684s) [noise=true , address=145.79.227.36:4518   , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.683832937 (591h14m43.554915972s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.684783196 (591h14m43.555866241s) [noise=true , address=145.79.227.18:4007   , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.685038492 (591h14m43.556121527s) [noise=true , address=145.79.227.3:4151    , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686167853 (591h14m43.557250908s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.686258695 (591h14m43.55734174s) [noise=true , address=145.79.227.29:4943   , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687362998 (591h14m43.558446043s) [noise=true , address=45.91.116.10:4083    , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.687636299 (591h14m43.558719334s) [noise=true , address=145.79.227.50:4374   , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688312987 (591h14m43.559396022s) [noise=true , address=145.79.227.26:4654   , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688619340 (591h14m43.559702385s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.688729590 (591h14m43.559812635s) [noise=true , address=145.79.227.6:4257    , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.689649502 (591h14m43.560732537s) [noise=true , address=77.90.31.10:4517     , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:43.690303357 (591h14m43.561386402s) [noise=true , address=145.79.227.51:4410   , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Uploading piece...
Dec 30 06:43:45.296790318 (591h14m45.167873363s) [upload=518325]: Next returned nil segment
Dec 30 06:43:45.296952557 (591h14m45.168035602s) [upload=518325]: Waiting for error group managing segments...
Dec 30 06:43:45.672927161 (591h14m45.544010236s) [noise=true , address=145.79.227.50:4374   , piece_id=W3NSQQDQ72JDIVU3LX7BQSWTCTEM3UAZLH2GU2AM7I7DWIRHFKZA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.679981644 (591h14m45.551064679s) [noise=true , address=145.79.227.14:4838   , piece_id=LNTWJQBCQVT7WXSCH5EIII7DHI4SVOYIVKK4F7IXIKM4KMED2YMQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.681994216 (591h14m45.553077251s) [noise=true , address=145.79.227.11:4732   , piece_id=WOMYTSZBK7KZRKDYYZA3ORJ5M6RCYJUHU2O7GXYY3CSQVFYE3S4A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.682769572 (591h14m45.553852607s) [noise=true , address=145.79.227.36:4518   , piece_id=KWGCY63F2DZC5MNJIY5T73RQHOGDUNIOHOMFBXJXWIQXFQA73PYA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.685368261 (591h14m45.556451296s) [noise=true , address=145.79.227.51:4410   , piece_id=HDGHD2ORSRHHSIJG23HYPYM7GATOSAOD24MYETBMKCICOSTFCJMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.696508394 (591h14m45.567591429s) [noise=true , address=145.79.227.26:4654   , piece_id=J7RRD6V3GTKDEZEPPOUNPBAMD3Q4QHIG7T6Z7FE72EJAJKSDTGBQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.698096838 (591h14m45.569179883s) [noise=true , address=145.79.227.6:4257    , piece_id=IMXIK3WJDAMAF7QXSYTWHGLDHUGQIBXQ3WDIZX45L4ODC35H7ICQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.699199428 (591h14m45.570282463s) [noise=true , address=145.79.227.18:4007   , piece_id=OWWYNA55RHUTUGUQT4Y2ZBI3VQAQQVNJTFKG6ALMRXXDLW3DW4AQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.758676632 (591h14m45.629759677s) [noise=true , address=145.79.227.3:4151    , piece_id=EJ6XIYN4PQ5KGO6EYP4YMQHXEZKZ2NHASAJTNVQ3VF7S5PNHCYKA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.903122208 (591h14m45.774205253s) [noise=true , address=77.90.31.10:4517     , piece_id=JXYAH2NNMGP5LSLNUGVUCG3X5NDPBW3MQ6ISE6HV4XPY7Y2CKSNA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.976028733 (591h14m45.847111778s) [noise=true , address=145.79.227.29:4943   , piece_id=TDSZ3VSCCNAU73ZL2CS5XPWOCA6TRTZZKY7VDS2WSK3E2PFFTNMA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:45.982909666 (591h14m45.853992711s) [noise=true , address=77.90.31.17:4770     , piece_id=EZYUKNUOTNWRXEROZAVMPDNQ2EAYUBO3KP7X2GWSQXU6ZRQ5EJQQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039347542 (591h14m45.910430587s) [noise=true , address=45.91.116.10:4083    , piece_id=AFTCCUF3L7226XLTU4GEFEKYGYQAQPJKHIPKX5N7MT2UCU45L2BQ, seg_pos={0 0}, upload=518325]: Done uploading piece. err: <nil>
Dec 30 06:43:46.039372420 (591h14m45.910455475s) [seg_pos={0 0}, upload=518325]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:43:46.039598750 (591h14m45.910681805s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039606464 (591h14m45.91068951s) [noise=true , address=77.90.31.15:4697     , piece_id=ZQJMG5N6Z5DYDTEMF5EL5YQGQVGKPEWMG6RZN6EXMEJYXJUQT3WA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039875217 (591h14m45.910958282s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039883282 (591h14m45.910966327s) [noise=true , address=45.91.116.9:4263     , piece_id=AS5TS3VJSLQMF7ZHLF7B2VBTDVIB23LHUU63EGI2ENDXGEYPDV5A, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039942615 (591h14m45.91102568s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1y9H9KNVuhzdB1XAPiwzHsFWYR4SrgB8Eff9HY6q6jmsYMTst8): protocol: expected piece hash; context canceled; EOF
Dec 30 06:43:46.039962774 (591h14m45.911045829s) [noise=true , address=45.91.116.11:4120    , piece_id=VTJS4ZRPH6KEIEAYEQ4ODRTTGTX6BA2FWMZRTNT75OF7SH2DZUDA, seg_pos={0 0}, upload=518325]: Long tail cancellation due to optimal threshold
Dec 30 06:43:46.039995456 (591h14m45.911078511s) [seg_pos={0 0}, upload=518325]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:43:46.040216396 (591h14m45.911299441s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:43:46.040225283 (591h14m45.911308328s) [upload=518325]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:43:46.331063484 (591h14m46.202146549s) [upload=518325]: Done uploading segments...
Dec 30 06:43:46.331089314 (591h14m46.202172349s) [upload=518325]: Upload finished. err: <nil>
Duration: 30.003072208s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [.......X...........X..............XX...X......X..X..X.....X.]
duration: 
3.650327237s
trace: 
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:44:12 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.650353809s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [........X...X......X............X...........................]
duration: 
51.831597174s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:44:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 51.831625269s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [....................X.......................................]
duration: 
6.267669655s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:43:48 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.267732995s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
9 out of 60 is failed (threshold=6) [........X...........X..............XX...X......X..X..X.....X]
duration: 
5.001646253s
trace: 
https://jaeger.dev.storj.io/trace/7d0f3c444f9c906c
FAIL (2025-12-30 06:43:44 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001668835s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration: 
7.176797549s
trace: 
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:43:31 UTC) 
orbiter warning: 
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031    , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037    , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038    , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030    , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047    , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035    , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.176819681s core uplink eu-select-1 eu1-placement-15
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.....................X......................................]
duration: 
7.191060517s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:43:31 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.191118076s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [......................X.....................................]
duration: 
8.598872779s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:43:15 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.598915771s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
13 out of 60 is failed (threshold=3) [......X..X......X....X...X..X.X.X.....X.......X.X.....X....X]
duration: 
30.003870321s
trace: 
https://jaeger.dev.storj.io/trace/573a856a59c2addb
FAIL (2025-12-30 06:43:10 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
Dec 30 06:42:40.870163758 (591h13m40.741246793s) [upload=518310]: Starting upload
Dec 30 06:42:40.870307041 (591h13m40.741390076s) [upload=518310]: Done starting upload
Dec 30 06:42:40.870494878 (591h13m40.741577923s) [upload=518310]: Uploading segments...
Dec 30 06:42:40.870589068 (591h13m40.741672103s) [upload=518310]: Got next segment. Inline: false
Dec 30 06:42:40.870625207 (591h13m40.741708252s) [upload=518310]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:42:40.870629625 (591h13m40.74171266s) [upload=518310]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:42:41.118476654 (591h13m40.989559699s) [seg_pos={0 0}, upload=518310]: Begin upload segment...
Dec 30 06:42:41.118524264 (591h13m40.989607319s) [seg_pos={0 0}, upload=518310]: Done begin upload segment.
Dec 30 06:42:41.118550244 (591h13m40.989633289s) [noise=true , address=145.79.227.6:4259    , piece_id=FTC2ODLJEXCNGXUSUIUYXPCDHDWJCYWM3F5WQS447BNW5W7PU4YQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.118723834 (591h13m40.989806879s) [noise=true , address=145.79.227.47:4265   , piece_id=5423K3WBCE33K6YEZ3YO7WDKMU7522RFUNW2KFYGQB7ID7VVYHCQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.118930508 (591h13m40.990013543s) [noise=true , address=145.79.227.41:4085   , piece_id=SORW6IG5KXMSZSYMMJ2GLEEBH46E7DB6B4A2EWBYF5FDLBZ6MCIA, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.119454526 (591h13m40.990537571s) [noise=true , address=145.79.227.9:4365    , piece_id=M4DRE6YZXC3WKC7656SAL4KXFWIQCVDEYN4RHH4F2KMWIVJ4CV4Q, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.119892741 (591h13m40.990975776s) [noise=true , address=145.79.227.38:4979   , piece_id=NSIP7OX6ITE4YREWLWUCKBXCJJ65CR6IOGPHA362HUSCABQWJXHA, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.120972406 (591h13m40.992055441s) [noise=true , address=77.90.31.10:4518     , piece_id=CWUCNGSK5IUXPB2JAUSH7ELABEAC2FGPZDUBKLRMESL5EGBKQPCA, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.121769904 (591h13m40.992852939s) [noise=true , address=45.91.116.11:4119    , piece_id=XIRTULFVU6U5WCZ5OLOTO7IVJ6GAMN3AMOG74QMRI7A4M54DOAEQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.122848398 (591h13m40.993931433s) [noise=true , address=145.79.227.11:4732   , piece_id=2PFJ6ATIXZKUU6MRZT55NKQEHEHEILZFDBX546XOYPPPL3MHEQ3A, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.123653001 (591h13m40.994736036s) [noise=true , address=45.91.116.8:4045     , piece_id=LF3SXSTGJBSEG2TAKWI3IYC4ZYOD24SL2OXQRCOXMOTSXBAFBWVA, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.124359135 (591h13m40.99544217s) [noise=true , address=145.79.227.2:4115    , piece_id=X7AVNTNQI6QJJ3ADEHRG55IGB7XX377DOEM3JYPKIRJRHEG3AOSQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.124762943 (591h13m40.995845988s) [noise=true , address=145.79.227.17:4971   , piece_id=5T5YM3JQGGFFRAM53PDI7ZHKN7F5PP2CZASALBP34DN6FFFZFG6A, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.125043658 (591h13m40.996126693s) [noise=true , address=45.91.116.4:4939     , piece_id=KTIW6KYZ73DK4H6HOGBYMSCZLZPH56K2ASYRFOLBUAZMSTRMPP7Q, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.125475942 (591h13m40.996558977s) [noise=true , address=145.79.227.10:4402   , piece_id=VEFVGMHGMQUH5O7CTBVMYRZDPCZRNVD2MJHLXDZK3EXQMRY4HDWQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.125718944 (591h13m40.996801979s) [noise=true , address=145.79.227.23:4545   , piece_id=CC6E5DWVPMNIAM32NTAV4FTNI7L6LVHSKLOXNWFILITMMVQH2OMQ, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.126002755 (591h13m40.9970858s) [noise=true , address=145.79.227.50:4375   , piece_id=GQJ5H4IW4MB6H4VIGAE47ZMSMDRZUCWQ7CLX6L36EBYJROULSB6A, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:41.127227607 (591h13m40.998310652s) [noise=true , address=45.91.116.5:4297     , piece_id=JYLV5NCR3EUOPMJGKGSHRPSNB6BDIEA3XD7PWMVMILDVUUMQDPXA, seg_pos={0 0}, upload=518310]: Uploading piece...
Dec 30 06:42:42.824282975 (591h13m42.69536601s) [upload=518310]: Next returned nil segment
Dec 30 06:42:42.824294326 (591h13m42.695377362s) [upload=518310]: Waiting for error group managing segments...
Dec 30 06:42:43.096965865 (591h13m42.9680489s) [noise=true , address=145.79.227.9:4365    , piece_id=M4DRE6YZXC3WKC7656SAL4KXFWIQCVDEYN4RHH4F2KMWIVJ4CV4Q, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.106671115 (591h13m42.97775415s) [noise=true , address=145.79.227.11:4732   , piece_id=2PFJ6ATIXZKUU6MRZT55NKQEHEHEILZFDBX546XOYPPPL3MHEQ3A, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.107370848 (591h13m42.978453893s) [noise=true , address=145.79.227.10:4402   , piece_id=VEFVGMHGMQUH5O7CTBVMYRZDPCZRNVD2MJHLXDZK3EXQMRY4HDWQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.126147382 (591h13m42.997230417s) [noise=true , address=145.79.227.23:4545   , piece_id=CC6E5DWVPMNIAM32NTAV4FTNI7L6LVHSKLOXNWFILITMMVQH2OMQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.126512577 (591h13m42.997595613s) [noise=true , address=145.79.227.2:4115    , piece_id=X7AVNTNQI6QJJ3ADEHRG55IGB7XX377DOEM3JYPKIRJRHEG3AOSQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.133357280 (591h13m43.004440315s) [noise=true , address=145.79.227.41:4085   , piece_id=SORW6IG5KXMSZSYMMJ2GLEEBH46E7DB6B4A2EWBYF5FDLBZ6MCIA, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.134848931 (591h13m43.005931966s) [noise=true , address=145.79.227.17:4971   , piece_id=5T5YM3JQGGFFRAM53PDI7ZHKN7F5PP2CZASALBP34DN6FFFZFG6A, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.136662654 (591h13m43.007745699s) [noise=true , address=145.79.227.6:4259    , piece_id=FTC2ODLJEXCNGXUSUIUYXPCDHDWJCYWM3F5WQS447BNW5W7PU4YQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.155476629 (591h13m43.026559664s) [noise=true , address=145.79.227.47:4265   , piece_id=5423K3WBCE33K6YEZ3YO7WDKMU7522RFUNW2KFYGQB7ID7VVYHCQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.189011515 (591h13m43.06009455s) [noise=true , address=145.79.227.50:4375   , piece_id=GQJ5H4IW4MB6H4VIGAE47ZMSMDRZUCWQ7CLX6L36EBYJROULSB6A, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.467154332 (591h13m43.338237367s) [noise=true , address=45.91.116.5:4297     , piece_id=JYLV5NCR3EUOPMJGKGSHRPSNB6BDIEA3XD7PWMVMILDVUUMQDPXA, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.517700870 (591h13m43.388783915s) [noise=true , address=45.91.116.8:4045     , piece_id=LF3SXSTGJBSEG2TAKWI3IYC4ZYOD24SL2OXQRCOXMOTSXBAFBWVA, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.621270773 (591h13m43.492353808s) [noise=true , address=145.79.227.38:4979   , piece_id=NSIP7OX6ITE4YREWLWUCKBXCJJ65CR6IOGPHA362HUSCABQWJXHA, seg_pos={0 0}, upload=518310]: Done uploading piece. err: <nil>
Dec 30 06:42:43.621295771 (591h13m43.492378806s) [seg_pos={0 0}, upload=518310]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:42:43.621595832 (591h13m43.492678867s) [noise=true , address=45.91.116.4:4939     , piece_id=KTIW6KYZ73DK4H6HOGBYMSCZLZPH56K2ASYRFOLBUAZMSTRMPP7Q, seg_pos={0 0}, upload=518310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12K4acSz27pP9v6DZSvETPuUfzFFiR2k7U3awntw2eHNytnPiuX): protocol: expected piece hash; context canceled; EOF
Dec 30 06:42:43.621605761 (591h13m43.492688806s) [noise=true , address=45.91.116.4:4939     , piece_id=KTIW6KYZ73DK4H6HOGBYMSCZLZPH56K2ASYRFOLBUAZMSTRMPP7Q, seg_pos={0 0}, upload=518310]: Long tail cancellation due to optimal threshold
Dec 30 06:42:43.621865054 (591h13m43.492948099s) [noise=true , address=77.90.31.10:4518     , piece_id=CWUCNGSK5IUXPB2JAUSH7ELABEAC2FGPZDUBKLRMESL5EGBKQPCA, seg_pos={0 0}, upload=518310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3): protocol: expected piece hash; context canceled; EOF
Dec 30 06:42:43.621884582 (591h13m43.492967627s) [noise=true , address=77.90.31.10:4518     , piece_id=CWUCNGSK5IUXPB2JAUSH7ELABEAC2FGPZDUBKLRMESL5EGBKQPCA, seg_pos={0 0}, upload=518310]: Long tail cancellation due to optimal threshold
Dec 30 06:42:43.622570719 (591h13m43.493653774s) [noise=true , address=45.91.116.11:4119    , piece_id=XIRTULFVU6U5WCZ5OLOTO7IVJ6GAMN3AMOG74QMRI7A4M54DOAEQ, seg_pos={0 0}, upload=518310]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm): protocol: context canceled; context canceled
Dec 30 06:42:43.622582891 (591h13m43.493665927s) [noise=true , address=45.91.116.11:4119    , piece_id=XIRTULFVU6U5WCZ5OLOTO7IVJ6GAMN3AMOG74QMRI7A4M54DOAEQ, seg_pos={0 0}, upload=518310]: Long tail cancellation due to optimal threshold
Dec 30 06:42:43.622605134 (591h13m43.493688169s) [seg_pos={0 0}, upload=518310]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:42:43.622698682 (591h13m43.493781717s) [upload=518310]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:42:43.622705465 (591h13m43.49378851s) [upload=518310]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:42:43.907727010 (591h13m43.778810045s) [upload=518310]: Done uploading segments...
Dec 30 06:42:43.907745225 (591h13m43.77882827s) [upload=518310]: Upload finished. err: <nil>
Duration: 30.003898795s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.........X...X......X............X..........................]
duration: 
47.587453732s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:43:05 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 47.587481194s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [.........X...........X..............XX...X......X..X..X.....]
duration: 
3.374764085s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:42:46 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.374787678s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration: 
7.193616835s
trace: 
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:42:23 UTC) 
orbiter warning: 
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031    , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037    , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038    , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030    , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047    , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035    , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 7.193641192s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks-placement-15", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.placement_15)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................................................X]
duration: 
1m0.003636879s
trace: 
https://jaeger.dev.storj.io/trace/4c2f4efaff0e0ea0
FAIL (2025-12-30 06:42:12 UTC) 
orbiter warning: 
operation: DeleteObject
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
load=518292]: Uploading piece...
Dec 30 06:41:13.598100152 (591h12m13.469183187s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.598587971 (591h12m13.469671016s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599093143 (591h12m13.470176178s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.599699999 (591h12m13.470783034s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600136169 (591h12m13.471219204s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.600543184 (591h12m13.471626229s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601048286 (591h12m13.472131332s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.601698784 (591h12m13.472781819s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602168259 (591h12m13.473251304s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:13.602782708 (591h12m13.473865753s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Uploading piece...
Dec 30 06:41:14.423395002 (591h12m14.294478047s) [upload=518292]: Next returned nil segment
Dec 30 06:41:14.423414068 (591h12m14.294497103s) [upload=518292]: Waiting for error group managing segments...
Dec 30 06:41:14.817650316 (591h12m14.688733351s) [noise=true , address=185.167.204.205:4029 , piece_id=KUOMMYJ2F4NIK6VN4C5M3Y3TMHYEZH7QFNJ6K3CLOAVI5R6VKTHQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.817870054 (591h12m14.688953099s) [noise=true , address=185.167.204.205:4019 , piece_id=H6AH654VGX6NHBUKODIWGL7I2RKE3DDMY4UM6VDONP2EP3NPDJBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818623698 (591h12m14.689706744s) [noise=true , address=185.167.204.206:4065 , piece_id=LD3J4656Q5QEUCMJ5OKEYATEVRIKSXO7XGARTN25BVM7FDIYOWKA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.818932436 (591h12m14.690015481s) [noise=true , address=185.167.204.206:4061 , piece_id=T2XS4IFPQSVS3T45SPQ5ZDDVT2VMTMXHKL2ZMX4OIOPRR67SCD2Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819147046 (591h12m14.690230091s) [noise=true , address=185.167.204.206:4071 , piece_id=QI2TOY5KCDZVWCCLMAJXVUWD2HB7S7IOTEAV2EHAMUNHKOFKTNYA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819678578 (591h12m14.690761623s) [noise=true , address=185.167.204.206:4059 , piece_id=LMOGTWX2YIRR3EKNROK43O2B27S5L5U6GYB7HBVRBPQPH4WDDWZQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.819905028 (591h12m14.690988063s) [noise=true , address=185.167.204.206:4058 , piece_id=ROOCGK64M5RHIWNAVCSZNYTZ6M2ORWB7MJ27XJVDTFVQIKJZHY4A, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.820479312 (591h12m14.691562357s) [noise=true , address=185.167.204.205:4021 , piece_id=KP7J5XSY4SKSKZG7XXFZMNKMRP4472PXI7CW2UQBZICN4MAFO5RQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.823149296 (591h12m14.694232331s) [noise=true , address=185.167.204.205:4034 , piece_id=OWZHQB5ZDT2QZ2QAHZJVRWQ222MX6CWLKJUOIAAGSMAVLWGJAOBA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.827906996 (591h12m14.698990041s) [noise=true , address=185.167.204.206:4055 , piece_id=VYEDR6XHZBZMTVJ5U5OF67WZOGVMCMQLXXBQUI7DHBCCVQ2RRERQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828003769 (591h12m14.699086814s) [noise=true , address=185.167.204.205:4016 , piece_id=I6MUDLAXUUDJL2C4HFE22C2L56A7D6HSCNWXKWJ7OKO2N4RA4W7Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.828893203 (591h12m14.699976238s) [noise=true , address=185.167.204.206:4064 , piece_id=H4PP7HPHXRX2LOZ4776AMWAJDN7OC435E2K4BXI7XOZYKTB4YUNA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829477074 (591h12m14.700560109s) [noise=true , address=185.167.204.205:4017 , piece_id=CZP65KETMGOHFDMV7KENRL4V36UHBPWRDCBBPUM5NE5L2PV4AQCQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.829774972 (591h12m14.700858007s) [noise=true , address=185.167.204.206:4043 , piece_id=G7VZYID4TXGNPDN63NQULXOVSX42WMFFNPBX6KQJ5L2WSW4D2YGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:14.831734493 (591h12m14.702817528s) [noise=true , address=185.167.204.206:4049 , piece_id=BQ7JEGSGPIFJUGZFBYRPDMBNPZ24BG6TEG4VUM4JUBTE4T2RF4SA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.405688561 (591h12m15.276771606s) [noise=true , address=81.171.14.70:4027    , piece_id=6WEORWX5QOMFSJUEADIIBHHIPKNFP4NJPZAJADBUTPUSG7MZJMOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.417110440 (591h12m15.288193485s) [noise=true , address=81.171.14.70:4031    , piece_id=U6Z4SNK32ZSWQ7NDR7ACXGRC4BADDOPR65ILBR64QLGVNDBADFFQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.425600446 (591h12m15.296683481s) [noise=true , address=81.171.14.70:4037    , piece_id=6A25LEIAQEJB3DNCTXQT6K7ZSKIGOW4QUGMNKLOG3CT2XXH25PSA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.499056228 (591h12m15.370139263s) [noise=true , address=81.171.14.70:4038    , piece_id=3WRHW3VGXWHOEOV5Q6CLYSF7MUVWS7QHTNHLSTCKAUERPHAH3I3Q, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.514998925 (591h12m15.38608197s) [noise=true , address=81.171.14.70:4041    , piece_id=NORNA6WFTGPNOHU7MPTNNB5ONTBS5DJGCN2NWIZAEPYSUSIOSGEQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528579544 (591h12m15.399662589s) [noise=true , address=81.171.14.70:4030    , piece_id=KITOJIT34CHS2QZBXCW2SJPZB4RHEHCMJ6IYQ4Z5KLL24YAWBIUQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.528991398 (591h12m15.400074434s) [noise=true , address=81.171.14.70:4047    , piece_id=ZWV5DP4YC3MSYDWEQH4ZK3VR4WZ4DUTI4VL3WVLQKFIEKIW44LOQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.534844544 (591h12m15.405927579s) [noise=true , address=81.171.14.70:4035    , piece_id=XCLL5SOAPJSSGF5YYQH6ZKGOHMLS6RI6H26FOW5XIJRY3TWYNEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.915614910 (591h12m15.786697945s) [noise=true , address=81.171.14.70:4039    , piece_id=DEJDXFDHOSAMIP2FN5LJBJNBB4QLJSE2AUDW6IMA6IC4VRY7VAGQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:15.967820176 (591h12m15.838903211s) [noise=true , address=81.171.14.70:4046    , piece_id=B2MTUNKMUP5S7XW4SVQ5QI2ZO3VFB5QVAKBBY4BIZIUILM6O7VYQ, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172116128 (591h12m16.043199163s) [noise=true , address=89.149.201.147:4007  , piece_id=HZLSYGSBP4TJH5V5XF3Q3AETJU625ERNDBD3VXW2HUZTTPAHLBMA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: <nil>
Dec 30 06:41:16.172137559 (591h12m16.043220604s) [seg_pos={0 0}, upload=518292]: Segment reached optimal threshold of 26 pieces.
Dec 30 06:41:16.172336949 (591h12m16.043419994s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QN8VP6o7qGw9Dd8EafU9xe7s7Cuf1FyqPMbnRaznrfPxfEUUn): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172344213 (591h12m16.043427258s) [noise=true , address=89.149.201.147:4008  , piece_id=XYFRNJVIFFZA5S36KFPGZPVCPLRL2OW5QBJD3UR37VPPA2RX3RJA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172554002 (591h12m16.043637037s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12YeKHJH7CoouLFJYENJhzVLNkA32hJaqjLz3dXhr78eaSGtcic): protocol: expected piece hash; context canceled; EOF
Dec 30 06:41:16.172565233 (591h12m16.043648278s) [noise=true , address=89.149.201.147:4020  , piece_id=N7G5DGEDYVKBTO5S6BYW3IYEOK7HAMK5JQIBMVPMPYRDHXODWEGA, seg_pos={0 0}, upload=518292]: Long tail cancellation due to optimal threshold
Dec 30 06:41:16.172587506 (591h12m16.043670541s) [seg_pos={0 0}, upload=518292]: Done waiting for segment. successful: 26 optimal: 26 errs: <nil>
Dec 30 06:41:16.172717945 (591h12m16.04380098s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:41:16.172724567 (591h12m16.043807602s) [upload=518292]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:41:16.404092237 (591h12m16.275175292s) [upload=518292]: Done uploading segments...
Dec 30 06:41:16.404119940 (591h12m16.275202975s) [upload=518292]: Upload finished. err: <nil>
Duration: 1m0.00380563s core uplink eu-select-1 eu1-placement-15
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..........X...X......X............X.........................]
duration: 
30.902608345s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:42:06 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 30.90263196s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.......................X....................................]
duration: 
8.699490162s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:42:06 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.699524477s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
12 out of 60 is failed (threshold=3) [.......X..X......X....X...X..X.X.X.....X.......X.X.....X....]
duration: 
6.801076788s
trace: 
https://jaeger.dev.storj.io/trace/70a1058b8eb76ef6
FAIL (2025-12-30 06:42:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ) context deadline exceeded
======LOGS======
Dec 30 06:39:28.036439160 (591h10m27.907522205s) [upload=518263]: Starting upload
Dec 30 06:39:28.036517078 (591h10m27.907600123s) [upload=518263]: Done starting upload
Dec 30 06:39:28.036938501 (591h10m27.908021546s) [upload=518263]: Uploading segments...
Dec 30 06:39:28.037102764 (591h10m27.908185849s) [upload=518263]: Got next segment. Inline: false
Dec 30 06:39:28.037144553 (591h10m27.908227588s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:39:28.037151977 (591h10m27.908235022s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:39:28.284983437 (591h10m28.156066472s) [seg_pos={0 0}, upload=518263]: Begin upload segment...
Dec 30 06:39:28.285027170 (591h10m28.156110205s) [seg_pos={0 0}, upload=518263]: Done begin upload segment.
Dec 30 06:39:28.285053310 (591h10m28.156136356s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288631523 (591h10m28.159714558s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288674476 (591h10m28.159757511s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.289950485 (591h10m28.16103353s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.290760628 (591h10m28.161843683s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.291334110 (591h10m28.162417145s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292463800 (591h10m28.163546845s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292536810 (591h10m28.163619845s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.293704914 (591h10m28.164787949s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294016798 (591h10m28.165099843s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294898607 (591h10m28.165981642s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.295619380 (591h10m28.166702415s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.296137436 (591h10m28.167220471s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.297144263 (591h10m28.168227308s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298734771 (591h10m28.169817816s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298955081 (591h10m28.170038117s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:29.914956649 (591h10m29.786039684s) [upload=518263]: Next returned nil segment
Dec 30 06:39:29.914978040 (591h10m29.786061075s) [upload=518263]: Waiting for error group managing segments...
Dec 30 06:39:30.309860227 (591h10m30.180943262s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.314474824 (591h10m30.185557869s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.315239400 (591h10m30.186322435s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.317380217 (591h10m30.188463252s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.321029955 (591h10m30.192113001s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.369079690 (591h10m30.240162725s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.374119557 (591h10m30.245202592s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.590590933 (591h10m30.461673968s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.609264159 (591h10m30.480347204s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.612038261 (591h10m30.483121296s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.692293589 (591h10m30.563376624s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.783122702 (591h10m30.654205737s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053078901 (591h10m30.924161936s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053101424 (591h10m30.924184459s) [seg_pos={0 0}, upload=518263]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:39:31.053397457 (591h10m30.924480502s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:54804->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:39:31.053404241 (591h10m30.924487286s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053586498 (591h10m30.924669533s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053612037 (591h10m30.924695072s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053785697 (591h10m30.924868742s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053816897 (591h10m30.924899952s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053835582 (591h10m30.924918627s) [seg_pos={0 0}, upload=518263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:39:31.053925263 (591h10m30.925008308s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:39:31.053931435 (591h10m30.92501447s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:39:31.342912920 (591h10m31.213995955s) [upload=518263]: Done uploading segments...
Dec 30 06:39:31.342931876 (591h10m31.214014911s) [upload=518263]: Upload finished. err: <nil>
Duration: 6.801106333s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [..........X...........X..............XX...X......X..X..X....]
duration: 
3.4994622s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:41:53 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.499481847s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [........................X...................................]
duration: 
14.460501019s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:41:45 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 14.460525677s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [...........X...........X..............XX...X......X..X..X...]
duration: 
3.608480476s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:41:35 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.608507398s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.........................X..................................]
duration: 
6.889736216s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:41:10 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.889763718s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...........X...X......X............X........................]
duration: 
34.704881648s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:41:04 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 34.704917457s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
13 out of 60 is failed (threshold=3) [X.......X..X......X....X...X..X.X.X.....X.......X.X.....X...]
duration: 
5.69695207s
trace: 
https://jaeger.dev.storj.io/trace/70a1058b8eb76ef6
FAIL (2025-12-30 06:41:01 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ) context deadline exceeded
======LOGS======
Dec 30 06:39:28.036439160 (591h10m27.907522205s) [upload=518263]: Starting upload
Dec 30 06:39:28.036517078 (591h10m27.907600123s) [upload=518263]: Done starting upload
Dec 30 06:39:28.036938501 (591h10m27.908021546s) [upload=518263]: Uploading segments...
Dec 30 06:39:28.037102764 (591h10m27.908185849s) [upload=518263]: Got next segment. Inline: false
Dec 30 06:39:28.037144553 (591h10m27.908227588s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:39:28.037151977 (591h10m27.908235022s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:39:28.284983437 (591h10m28.156066472s) [seg_pos={0 0}, upload=518263]: Begin upload segment...
Dec 30 06:39:28.285027170 (591h10m28.156110205s) [seg_pos={0 0}, upload=518263]: Done begin upload segment.
Dec 30 06:39:28.285053310 (591h10m28.156136356s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288631523 (591h10m28.159714558s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288674476 (591h10m28.159757511s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.289950485 (591h10m28.16103353s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.290760628 (591h10m28.161843683s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.291334110 (591h10m28.162417145s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292463800 (591h10m28.163546845s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292536810 (591h10m28.163619845s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.293704914 (591h10m28.164787949s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294016798 (591h10m28.165099843s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294898607 (591h10m28.165981642s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.295619380 (591h10m28.166702415s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.296137436 (591h10m28.167220471s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.297144263 (591h10m28.168227308s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298734771 (591h10m28.169817816s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298955081 (591h10m28.170038117s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:29.914956649 (591h10m29.786039684s) [upload=518263]: Next returned nil segment
Dec 30 06:39:29.914978040 (591h10m29.786061075s) [upload=518263]: Waiting for error group managing segments...
Dec 30 06:39:30.309860227 (591h10m30.180943262s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.314474824 (591h10m30.185557869s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.315239400 (591h10m30.186322435s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.317380217 (591h10m30.188463252s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.321029955 (591h10m30.192113001s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.369079690 (591h10m30.240162725s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.374119557 (591h10m30.245202592s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.590590933 (591h10m30.461673968s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.609264159 (591h10m30.480347204s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.612038261 (591h10m30.483121296s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.692293589 (591h10m30.563376624s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.783122702 (591h10m30.654205737s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053078901 (591h10m30.924161936s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053101424 (591h10m30.924184459s) [seg_pos={0 0}, upload=518263]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:39:31.053397457 (591h10m30.924480502s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:54804->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:39:31.053404241 (591h10m30.924487286s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053586498 (591h10m30.924669533s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053612037 (591h10m30.924695072s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053785697 (591h10m30.924868742s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053816897 (591h10m30.924899952s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053835582 (591h10m30.924918627s) [seg_pos={0 0}, upload=518263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:39:31.053925263 (591h10m30.925008308s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:39:31.053931435 (591h10m30.92501447s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:39:31.342912920 (591h10m31.213995955s) [upload=518263]: Done uploading segments...
Dec 30 06:39:31.342931876 (591h10m31.214014911s) [upload=518263]: Upload finished. err: <nil>
Duration: 5.696985454s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..........................X.................................]
duration: 
6.373884196s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:40:51 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.373915006s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [XX.......X..X......X....X...X..X.X.X.....X.......X.X.....X..]
duration: 
5.839416489s
trace: 
https://jaeger.dev.storj.io/trace/70a1058b8eb76ef6
FAIL (2025-12-30 06:40:27 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ) context deadline exceeded
======LOGS======
Dec 30 06:39:28.036439160 (591h10m27.907522205s) [upload=518263]: Starting upload
Dec 30 06:39:28.036517078 (591h10m27.907600123s) [upload=518263]: Done starting upload
Dec 30 06:39:28.036938501 (591h10m27.908021546s) [upload=518263]: Uploading segments...
Dec 30 06:39:28.037102764 (591h10m27.908185849s) [upload=518263]: Got next segment. Inline: false
Dec 30 06:39:28.037144553 (591h10m27.908227588s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:39:28.037151977 (591h10m27.908235022s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:39:28.284983437 (591h10m28.156066472s) [seg_pos={0 0}, upload=518263]: Begin upload segment...
Dec 30 06:39:28.285027170 (591h10m28.156110205s) [seg_pos={0 0}, upload=518263]: Done begin upload segment.
Dec 30 06:39:28.285053310 (591h10m28.156136356s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288631523 (591h10m28.159714558s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288674476 (591h10m28.159757511s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.289950485 (591h10m28.16103353s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.290760628 (591h10m28.161843683s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.291334110 (591h10m28.162417145s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292463800 (591h10m28.163546845s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292536810 (591h10m28.163619845s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.293704914 (591h10m28.164787949s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294016798 (591h10m28.165099843s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294898607 (591h10m28.165981642s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.295619380 (591h10m28.166702415s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.296137436 (591h10m28.167220471s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.297144263 (591h10m28.168227308s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298734771 (591h10m28.169817816s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298955081 (591h10m28.170038117s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:29.914956649 (591h10m29.786039684s) [upload=518263]: Next returned nil segment
Dec 30 06:39:29.914978040 (591h10m29.786061075s) [upload=518263]: Waiting for error group managing segments...
Dec 30 06:39:30.309860227 (591h10m30.180943262s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.314474824 (591h10m30.185557869s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.315239400 (591h10m30.186322435s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.317380217 (591h10m30.188463252s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.321029955 (591h10m30.192113001s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.369079690 (591h10m30.240162725s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.374119557 (591h10m30.245202592s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.590590933 (591h10m30.461673968s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.609264159 (591h10m30.480347204s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.612038261 (591h10m30.483121296s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.692293589 (591h10m30.563376624s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.783122702 (591h10m30.654205737s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053078901 (591h10m30.924161936s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053101424 (591h10m30.924184459s) [seg_pos={0 0}, upload=518263]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:39:31.053397457 (591h10m30.924480502s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:54804->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:39:31.053404241 (591h10m30.924487286s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053586498 (591h10m30.924669533s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053612037 (591h10m30.924695072s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053785697 (591h10m30.924868742s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053816897 (591h10m30.924899952s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053835582 (591h10m30.924918627s) [seg_pos={0 0}, upload=518263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:39:31.053925263 (591h10m30.925008308s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:39:31.053931435 (591h10m30.92501447s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:39:31.342912920 (591h10m31.213995955s) [upload=518263]: Done uploading segments...
Dec 30 06:39:31.342931876 (591h10m31.214014911s) [upload=518263]: Upload finished. err: <nil>
Duration: 5.8394593s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [............X...........X..............XX...X......X..X..X..]
duration: 
3.472302442s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:40:25 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.472327822s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...........................X................................]
duration: 
8.074277459s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:40:20 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 8.074309561s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [.XX.......X..X......X....X...X..X.X.X.....X.......X.X.....X.]
duration: 
6.049016656s
trace: 
https://jaeger.dev.storj.io/trace/70a1058b8eb76ef6
FAIL (2025-12-30 06:40:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ) context deadline exceeded
======LOGS======
Dec 30 06:39:28.036439160 (591h10m27.907522205s) [upload=518263]: Starting upload
Dec 30 06:39:28.036517078 (591h10m27.907600123s) [upload=518263]: Done starting upload
Dec 30 06:39:28.036938501 (591h10m27.908021546s) [upload=518263]: Uploading segments...
Dec 30 06:39:28.037102764 (591h10m27.908185849s) [upload=518263]: Got next segment. Inline: false
Dec 30 06:39:28.037144553 (591h10m27.908227588s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:39:28.037151977 (591h10m27.908235022s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:39:28.284983437 (591h10m28.156066472s) [seg_pos={0 0}, upload=518263]: Begin upload segment...
Dec 30 06:39:28.285027170 (591h10m28.156110205s) [seg_pos={0 0}, upload=518263]: Done begin upload segment.
Dec 30 06:39:28.285053310 (591h10m28.156136356s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288631523 (591h10m28.159714558s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288674476 (591h10m28.159757511s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.289950485 (591h10m28.16103353s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.290760628 (591h10m28.161843683s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.291334110 (591h10m28.162417145s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292463800 (591h10m28.163546845s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292536810 (591h10m28.163619845s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.293704914 (591h10m28.164787949s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294016798 (591h10m28.165099843s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294898607 (591h10m28.165981642s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.295619380 (591h10m28.166702415s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.296137436 (591h10m28.167220471s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.297144263 (591h10m28.168227308s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298734771 (591h10m28.169817816s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298955081 (591h10m28.170038117s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:29.914956649 (591h10m29.786039684s) [upload=518263]: Next returned nil segment
Dec 30 06:39:29.914978040 (591h10m29.786061075s) [upload=518263]: Waiting for error group managing segments...
Dec 30 06:39:30.309860227 (591h10m30.180943262s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.314474824 (591h10m30.185557869s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.315239400 (591h10m30.186322435s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.317380217 (591h10m30.188463252s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.321029955 (591h10m30.192113001s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.369079690 (591h10m30.240162725s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.374119557 (591h10m30.245202592s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.590590933 (591h10m30.461673968s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.609264159 (591h10m30.480347204s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.612038261 (591h10m30.483121296s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.692293589 (591h10m30.563376624s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.783122702 (591h10m30.654205737s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053078901 (591h10m30.924161936s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053101424 (591h10m30.924184459s) [seg_pos={0 0}, upload=518263]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:39:31.053397457 (591h10m30.924480502s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:54804->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:39:31.053404241 (591h10m30.924487286s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053586498 (591h10m30.924669533s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053612037 (591h10m30.924695072s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053785697 (591h10m30.924868742s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053816897 (591h10m30.924899952s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053835582 (591h10m30.924918627s) [seg_pos={0 0}, upload=518263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:39:31.053925263 (591h10m30.925008308s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:39:31.053931435 (591h10m30.92501447s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:39:31.342912920 (591h10m31.213995955s) [upload=518263]: Done uploading segments...
Dec 30 06:39:31.342931876 (591h10m31.214014911s) [upload=518263]: Upload finished. err: <nil>
Duration: 6.049037955s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [............X...X......X............X.......................]
duration: 
15.36570667s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:40:03 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.365743179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [............................X...............................]
duration: 
7.85378635s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:40:02 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 7.853819291s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [..XX.......X..X......X....X...X..X.X.X.....X.......X.X.....X]
duration: 
30.003445768s
trace: 
https://jaeger.dev.storj.io/trace/70a1058b8eb76ef6
FAIL (2025-12-30 06:39:57 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: 4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ) context deadline exceeded; piecestore: (Node ID: 1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, Piece ID: ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ) context deadline exceeded
======LOGS======
Dec 30 06:39:28.036439160 (591h10m27.907522205s) [upload=518263]: Starting upload
Dec 30 06:39:28.036517078 (591h10m27.907600123s) [upload=518263]: Done starting upload
Dec 30 06:39:28.036938501 (591h10m27.908021546s) [upload=518263]: Uploading segments...
Dec 30 06:39:28.037102764 (591h10m27.908185849s) [upload=518263]: Got next segment. Inline: false
Dec 30 06:39:28.037144553 (591h10m27.908227588s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:39:28.037151977 (591h10m27.908235022s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:39:28.284983437 (591h10m28.156066472s) [seg_pos={0 0}, upload=518263]: Begin upload segment...
Dec 30 06:39:28.285027170 (591h10m28.156110205s) [seg_pos={0 0}, upload=518263]: Done begin upload segment.
Dec 30 06:39:28.285053310 (591h10m28.156136356s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288631523 (591h10m28.159714558s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.288674476 (591h10m28.159757511s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.289950485 (591h10m28.16103353s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.290760628 (591h10m28.161843683s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.291334110 (591h10m28.162417145s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292463800 (591h10m28.163546845s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.292536810 (591h10m28.163619845s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.293704914 (591h10m28.164787949s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294016798 (591h10m28.165099843s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.294898607 (591h10m28.165981642s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.295619380 (591h10m28.166702415s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.296137436 (591h10m28.167220471s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.297144263 (591h10m28.168227308s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298734771 (591h10m28.169817816s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:28.298955081 (591h10m28.170038117s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Uploading piece...
Dec 30 06:39:29.914956649 (591h10m29.786039684s) [upload=518263]: Next returned nil segment
Dec 30 06:39:29.914978040 (591h10m29.786061075s) [upload=518263]: Waiting for error group managing segments...
Dec 30 06:39:30.309860227 (591h10m30.180943262s) [noise=true , address=145.79.227.11:4730   , piece_id=BGKPPCNPKII67WMEHS3G6LYQFOWFPC6WL2NWWBFCRAI35BRGH66Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.314474824 (591h10m30.185557869s) [noise=true , address=145.79.227.3:4151    , piece_id=YB2MI6IO4HIKF5ZBCOI5JVWK62T5DO2R3HPR3XSRAVFMIUDKEU7Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.315239400 (591h10m30.186322435s) [noise=true , address=145.79.227.51:4409   , piece_id=S33UGJVGMOKDSPGNH3QSBXSE665RU7II7I4N66SLFXXHJDAJY7TQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.317380217 (591h10m30.188463252s) [noise=true , address=145.79.227.25:4619   , piece_id=6JG6Q5U6ODH3JHXYMT7N6TIZ4V6YMIYIFEFGRSSZ3LN3Z53HFH3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.321029955 (591h10m30.192113001s) [noise=true , address=145.79.227.4:4185    , piece_id=BITLGN427KDEYTFOMDW62VNUZHLYBA4IZKQI6IVVFW3RCLQKPQYA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.369079690 (591h10m30.240162725s) [noise=true , address=145.79.227.16:4912   , piece_id=3IBUMXMY4RBYYWJUL3SXUNVNHAHVUHAS5SZHLFJWX57DNJYUWDSA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.374119557 (591h10m30.245202592s) [noise=true , address=145.79.227.41:4086   , piece_id=K5JH6L27A7TFNBLOBMMKUSJIZPRWPLKRTEO7L3CZR7LCOWHSLNGA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.590590933 (591h10m30.461673968s) [noise=true , address=77.90.31.13:4626     , piece_id=5TSTAG3HAIYVHQIOCRLRLODDL4WWMK7KC6VFCSI7UZX47I6F72UQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.609264159 (591h10m30.480347204s) [noise=true , address=45.91.116.7:4011     , piece_id=FA7PHKNHZAFFVMGQ4CUY7YKBXAQ5XWUH2PTPOWI43CU4ZLEF5RJA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.612038261 (591h10m30.483121296s) [noise=true , address=45.91.116.13:4154    , piece_id=ENSO3M543J23NCFTCIT5OCSWGA5XGQUSPS4PEA4POX3MPMQ56OPQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.692293589 (591h10m30.563376624s) [noise=true , address=77.90.31.12:4590     , piece_id=S5DU2VV3ESLEYFJVZI6XX55PPPZ4SCMRW6XBLY7VO4EFMRE42SXQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:30.783122702 (591h10m30.654205737s) [noise=true , address=45.91.116.9:4264     , piece_id=JAHLYL2TB5VAWL2SIWHSNTYN2MCGSOSDYMKKXE5JLJZ5F4FXWLKQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053078901 (591h10m30.924161936s) [noise=true , address=45.91.116.11:4117    , piece_id=4DA4A5RGZVFA6BTASK2E5QKUBE6FPHIU5YRVJS4NO3VY75LYFEFQ, seg_pos={0 0}, upload=518263]: Done uploading piece. err: <nil>
Dec 30 06:39:31.053101424 (591h10m30.924184459s) [seg_pos={0 0}, upload=518263]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:39:31.053397457 (591h10m30.924480502s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: read tcp 188.166.225.170:54804->45.91.116.5:4298: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Dec 30 06:39:31.053404241 (591h10m30.924487286s) [noise=true , address=45.91.116.5:4298     , piece_id=EIPZSCCDQKMWDDF64W45JXW6KMNNORL446W4FDEKDTSZFCXPP36Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053586498 (591h10m30.924669533s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053612037 (591h10m30.924695072s) [noise=true , address=145.79.227.40:4051   , piece_id=IM5E6GTXYXXUQL7ZM7GK6ZTK3QUKTPPQ5KTG4R7X4MP26HQT27UA, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053785697 (591h10m30.924868742s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:39:31.053816897 (591h10m30.924899952s) [noise=true , address=77.90.31.10:4520     , piece_id=J4UQYJGMUI5AAFREJPNG4BUWZKBVDKM73RK4OQLAEICUMQN4LS3Q, seg_pos={0 0}, upload=518263]: Long tail cancellation due to optimal threshold
Dec 30 06:39:31.053835582 (591h10m30.924918627s) [seg_pos={0 0}, upload=518263]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:39:31.053925263 (591h10m30.925008308s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:39:31.053931435 (591h10m30.92501447s) [upload=518263]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:39:31.342912920 (591h10m31.213995955s) [upload=518263]: Done uploading segments...
Dec 30 06:39:31.342931876 (591h10m31.214014911s) [upload=518263]: Upload finished. err: <nil>
Duration: 30.003468321s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [.............X...X......X............X......................]
duration: 
34.641974649s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:39:40 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 34.641999376s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [.............X...........X..............XX...X......X..X..X.]
duration: 
3.695298538s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:39:16 UTC) 
orbiter warning: context deadline exceeded
Duration: 3.695328195s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [.............................X..............................]
duration: 
6.504938356s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:39:00 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.505059887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [..............X...X......X............X.....................]
duration: 
15.352441864s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:38:50 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.352473135s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [X..XX.......X..X......X....X...X..X.X.X.....X.......X.X.....]
duration: 
6.162576956s
trace: 
https://jaeger.dev.storj.io/trace/883b4f2341882c2
FAIL (2025-12-30 06:38:45 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q) context deadline exceeded
======LOGS======
starting upload
Dec 30 06:35:00.700709664 (591h6m0.571792699s) [upload=518200]: Uploading segments...
Dec 30 06:35:00.700855001 (591h6m0.571938046s) [upload=518200]: Got next segment. Inline: false
Dec 30 06:35:00.700902551 (591h6m0.571985596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:35:00.700907551 (591h6m0.571990586s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:35:00.945523006 (591h6m0.816606071s) [seg_pos={0 0}, upload=518200]: Begin upload segment...
Dec 30 06:35:00.945576387 (591h6m0.816659422s) [seg_pos={0 0}, upload=518200]: Done begin upload segment.
Dec 30 06:35:00.945623758 (591h6m0.816706803s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.945672631 (591h6m0.816755666s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946423301 (591h6m0.817506346s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946941848 (591h6m0.818024893s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947322032 (591h6m0.818405097s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947811344 (591h6m0.818894389s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948024500 (591h6m0.819107535s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948488453 (591h6m0.819571498s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948539040 (591h6m0.819622085s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948975420 (591h6m0.820058455s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949027039 (591h6m0.820110084s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949523544 (591h6m0.820606589s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949539183 (591h6m0.820622218s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950071608 (591h6m0.821154653s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950097677 (591h6m0.821180712s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950788372 (591h6m0.821871417s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:02.579512986 (591h6m2.450596031s) [upload=518200]: Next returned nil segment
Dec 30 06:35:02.579528937 (591h6m2.450611982s) [upload=518200]: Waiting for error group managing segments...
Dec 30 06:35:02.981880046 (591h6m2.852963091s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992028110 (591h6m2.863111165s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992318953 (591h6m2.863402018s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.995375083 (591h6m2.866458128s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.005058321 (591h6m2.876141366s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.060242390 (591h6m2.931325435s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.064706310 (591h6m2.935789355s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.265292297 (591h6m3.136375342s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.290314984 (591h6m3.161398059s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.479217344 (591h6m3.350300389s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.496983904 (591h6m3.368066949s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.645769244 (591h6m3.516852299s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686416803 (591h6m3.557499848s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686440759 (591h6m3.557523804s) [seg_pos={0 0}, upload=518200]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:35:03.686716374 (591h6m3.557799419s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 06:35:03.686725201 (591h6m3.557808246s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687389175 (591h6m3.55847222s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection
Dec 30 06:35:03.687400767 (591h6m3.558483812s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687543790 (591h6m3.558626835s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection
Dec 30 06:35:03.687554680 (591h6m3.558637725s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687579117 (591h6m3.558662162s) [seg_pos={0 0}, upload=518200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:35:03.687670551 (591h6m3.558753596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:35:03.687676532 (591h6m3.558759577s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:35:03.973108018 (591h6m3.844191063s) [upload=518200]: Done uploading segments...
Dec 30 06:35:03.973143596 (591h6m3.844226641s) [upload=518200]: Upload finished. err: <nil>
Duration: 6.162607994s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [.X..XX.......X..X......X....X...X..X.X.X.....X.......X.X....]
duration: 
27.676531313s
trace: 
https://jaeger.dev.storj.io/trace/883b4f2341882c2
FAIL (2025-12-30 06:38:39 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q) context deadline exceeded
======LOGS======
starting upload
Dec 30 06:35:00.700709664 (591h6m0.571792699s) [upload=518200]: Uploading segments...
Dec 30 06:35:00.700855001 (591h6m0.571938046s) [upload=518200]: Got next segment. Inline: false
Dec 30 06:35:00.700902551 (591h6m0.571985596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:35:00.700907551 (591h6m0.571990586s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:35:00.945523006 (591h6m0.816606071s) [seg_pos={0 0}, upload=518200]: Begin upload segment...
Dec 30 06:35:00.945576387 (591h6m0.816659422s) [seg_pos={0 0}, upload=518200]: Done begin upload segment.
Dec 30 06:35:00.945623758 (591h6m0.816706803s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.945672631 (591h6m0.816755666s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946423301 (591h6m0.817506346s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946941848 (591h6m0.818024893s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947322032 (591h6m0.818405097s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947811344 (591h6m0.818894389s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948024500 (591h6m0.819107535s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948488453 (591h6m0.819571498s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948539040 (591h6m0.819622085s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948975420 (591h6m0.820058455s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949027039 (591h6m0.820110084s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949523544 (591h6m0.820606589s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949539183 (591h6m0.820622218s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950071608 (591h6m0.821154653s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950097677 (591h6m0.821180712s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950788372 (591h6m0.821871417s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:02.579512986 (591h6m2.450596031s) [upload=518200]: Next returned nil segment
Dec 30 06:35:02.579528937 (591h6m2.450611982s) [upload=518200]: Waiting for error group managing segments...
Dec 30 06:35:02.981880046 (591h6m2.852963091s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992028110 (591h6m2.863111165s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992318953 (591h6m2.863402018s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.995375083 (591h6m2.866458128s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.005058321 (591h6m2.876141366s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.060242390 (591h6m2.931325435s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.064706310 (591h6m2.935789355s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.265292297 (591h6m3.136375342s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.290314984 (591h6m3.161398059s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.479217344 (591h6m3.350300389s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.496983904 (591h6m3.368066949s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.645769244 (591h6m3.516852299s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686416803 (591h6m3.557499848s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686440759 (591h6m3.557523804s) [seg_pos={0 0}, upload=518200]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:35:03.686716374 (591h6m3.557799419s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 06:35:03.686725201 (591h6m3.557808246s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687389175 (591h6m3.55847222s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection
Dec 30 06:35:03.687400767 (591h6m3.558483812s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687543790 (591h6m3.558626835s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection
Dec 30 06:35:03.687554680 (591h6m3.558637725s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687579117 (591h6m3.558662162s) [seg_pos={0 0}, upload=518200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:35:03.687670551 (591h6m3.558753596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:35:03.687676532 (591h6m3.558759577s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:35:03.973108018 (591h6m3.844191063s) [upload=518200]: Done uploading segments...
Dec 30 06:35:03.973143596 (591h6m3.844226641s) [upload=518200]: Upload finished. err: <nil>
Duration: 27.676559628s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [..............................X.............................]
duration: 
5.814847346s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:38:24 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 5.814882022s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
8 out of 60 is failed (threshold=6) [..............X...........X..............XX...X......X..X..X]
duration: 
5.00126856s
trace: 
https://jaeger.dev.storj.io/trace/638ef26d0865793d
FAIL (2025-12-30 06:38:17 UTC) 
orbiter warning: context deadline exceeded
Duration: 5.001314209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip: 
1 out of 60 is failed (threshold=3) [...............................X............................]
duration: 
6.746723858s
trace: 
https://jaeger.dev.storj.io/trace/deb96436003dee
FAIL (2025-12-30 06:37:51 UTC) 
orbiter warning: read tcp 188.166.225.170:52458->45.38.24.2:443: read: connection reset by peer
Duration: 6.746755056s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [...............X...X......X............X....................]
duration: 
15.170172004s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:37:43 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 15.170202823s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip: 
14 out of 60 is failed (threshold=3) [..X..XX.......X..X......X....X...X..X.X.X.....X.......X.X...]
duration: 
6.419623059s
trace: 
https://jaeger.dev.storj.io/trace/883b4f2341882c2
FAIL (2025-12-30 06:37:12 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ) context deadline exceeded; piecestore: (Node ID: 1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, Piece ID: OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA) context deadline exceeded; piecestore: (Node ID: 12rwZa7GFRR5bJTazST3crJMAuEVFwmVNLm2SnrcoZtYiHsW3FX, Piece ID: QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q) context deadline exceeded
======LOGS======
starting upload
Dec 30 06:35:00.700709664 (591h6m0.571792699s) [upload=518200]: Uploading segments...
Dec 30 06:35:00.700855001 (591h6m0.571938046s) [upload=518200]: Got next segment. Inline: false
Dec 30 06:35:00.700902551 (591h6m0.571985596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Dec 30 06:35:00.700907551 (591h6m0.571990586s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Dec 30 06:35:00.945523006 (591h6m0.816606071s) [seg_pos={0 0}, upload=518200]: Begin upload segment...
Dec 30 06:35:00.945576387 (591h6m0.816659422s) [seg_pos={0 0}, upload=518200]: Done begin upload segment.
Dec 30 06:35:00.945623758 (591h6m0.816706803s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.945672631 (591h6m0.816755666s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946423301 (591h6m0.817506346s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.946941848 (591h6m0.818024893s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947322032 (591h6m0.818405097s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.947811344 (591h6m0.818894389s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948024500 (591h6m0.819107535s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948488453 (591h6m0.819571498s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948539040 (591h6m0.819622085s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.948975420 (591h6m0.820058455s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949027039 (591h6m0.820110084s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949523544 (591h6m0.820606589s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.949539183 (591h6m0.820622218s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950071608 (591h6m0.821154653s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950097677 (591h6m0.821180712s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:00.950788372 (591h6m0.821871417s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Uploading piece...
Dec 30 06:35:02.579512986 (591h6m2.450596031s) [upload=518200]: Next returned nil segment
Dec 30 06:35:02.579528937 (591h6m2.450611982s) [upload=518200]: Waiting for error group managing segments...
Dec 30 06:35:02.981880046 (591h6m2.852963091s) [noise=true , address=145.79.227.45:4230   , piece_id=R4DIRT4O52SPMXRZJWDJX5I3N4RUAYUDPEJ2JKWFRTMVX4DT2GSQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992028110 (591h6m2.863111165s) [noise=true , address=145.79.227.15:4875   , piece_id=LM7OQEBQZB352WGKRHJLDIJXYCGL5Q2LHQYETJFCVV4YRLJRGQAQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.992318953 (591h6m2.863402018s) [noise=true , address=145.79.227.46:4601   , piece_id=YRDRSSYAZGII36AL4LALOG73YTWAXPXVH4C2T4YGUIA6CMPVQTVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:02.995375083 (591h6m2.866458128s) [noise=true , address=145.79.227.49:4339   , piece_id=ELCOJSDBE2KQ7ZG4STQT26CXHYQJECVOLJVVF63CJ6INZPFNXTFQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.005058321 (591h6m2.876141366s) [noise=true , address=145.79.227.13:4804   , piece_id=YSM2HXZW6UUOW7QEOVEADHUUDJUZEDYTDHQAMFLBIW4SQUXDE5DQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.060242390 (591h6m2.931325435s) [noise=true , address=145.79.227.16:4909   , piece_id=QII6PXHXIVCKG473VVHSZC7BFAIERMARS2XE7BMPZVTD75CBN52A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.064706310 (591h6m2.935789355s) [noise=true , address=145.79.227.47:4267   , piece_id=OANW6JUI52D2R6PY6LJ474EREQFX66KB2I6Z75AHRQVJKVOSUDXQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.265292297 (591h6m3.136375342s) [noise=true , address=77.90.31.18:4808     , piece_id=AOFA43TQ4XIOZGUBLM5S4MCMWDLY3EDMA7TADNTMP3XYLZTFGCWA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.290314984 (591h6m3.161398059s) [noise=true , address=77.90.31.21:4877     , piece_id=PWB3DUUKENER2JLQ2THQB3SEU6IMLVAK2NFZMWLYBQJM2F65V72A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.479217344 (591h6m3.350300389s) [noise=true , address=45.91.116.11:4117    , piece_id=WQVIB3ZIRZRQRJLCQL7JEL57ZFL47TA6VVK2W7S4MK46JXXNWIVQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.496983904 (591h6m3.368066949s) [noise=true , address=77.90.31.10:4519     , piece_id=QE5WGSHM4NIFVPUC4WKM7NJFF42UJN7NLLE64SR4PLO3FM2L2P6Q, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.645769244 (591h6m3.516852299s) [noise=true , address=77.90.31.12:4590     , piece_id=OMXR3BZFHSIYM6W4ZXMDZI3TYYMI7RTA22DTCBJ5QGUAT3K44IBA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686416803 (591h6m3.557499848s) [noise=true , address=45.91.116.6:4973     , piece_id=TDJRK4WOVALL2ZRJ2MN3PSLLFDSUNIM65QZJKOKHHD4J7NUXHANA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: <nil>
Dec 30 06:35:03.686440759 (591h6m3.557523804s) [seg_pos={0 0}, upload=518200]: Segment reached optimal threshold of 13 pieces.
Dec 30 06:35:03.686716374 (591h6m3.557799419s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12URx18Ckh7VaAKJ1hdQQucMRu5qiDrAraqaycjYqxETgwZ4uWS): protocol: expected piece hash; context canceled; EOF
Dec 30 06:35:03.686725201 (591h6m3.557808246s) [noise=true , address=145.79.227.34:4907   , piece_id=IFCQP7SQ3R3QWNBHNQIFNXVQZHDI6HWCRYOKJ6SLB2M64W2SZ46A, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687389175 (591h6m3.55847222s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eg62VsecY4ALCTk2z2TrpPFMVnfNLhNkGyUJfBfPzQrq4X8mi): protocol: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34180->45.91.116.9:4261: use of closed network connection
Dec 30 06:35:03.687400767 (591h6m3.558483812s) [noise=true , address=45.91.116.9:4261     , piece_id=5CRNUHVNHHIPZ3R2WIINLHXCJLVPZB6XSYDVY4FURUB5JHJ5A3EQ, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687543790 (591h6m3.558626835s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH): protocol: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42160->45.91.116.12:4227: use of closed network connection
Dec 30 06:35:03.687554680 (591h6m3.558637725s) [noise=true , address=45.91.116.12:4227    , piece_id=3ZNXGVH5KZ5YHILFICJPB75GUEVUQG2GCSET5SGGEOBRB6TVNDPA, seg_pos={0 0}, upload=518200]: Long tail cancellation due to optimal threshold
Dec 30 06:35:03.687579117 (591h6m3.558662162s) [seg_pos={0 0}, upload=518200]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Dec 30 06:35:03.687670551 (591h6m3.558753596s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:35:03.687676532 (591h6m3.558759577s) [upload=518200]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:35:03.973108018 (591h6m3.844191063s) [upload=518200]: Done uploading segments...
Dec 30 06:35:03.973143596 (591h6m3.844226641s) [upload=518200]: Upload finished. err: <nil>
Duration: 6.41963977s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip: 
7 out of 60 is failed (threshold=6) [...............X...........X..............XX...X......X..X..]
duration: 
3.501748404s
trace: 
https://jaeger.dev.storj.io/trace/30c47f8e5a5f412a
FAIL (2025-12-30 06:37:03 UTC) 
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 3.501772851s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)
antiblip: 
4 out of 60 is failed (threshold=3) [................X...X......X............X...................]
duration: 
45.593412602s
trace: 
https://jaeger.dev.storj.io/trace/45b769fc71e13448
FAIL (2025-12-30 06:37:02 UTC) 
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: context deadline exceeded
======LOGS======
.577697683 (590h53m11.448780728s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578173528 (590h53m11.449256573s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578542992 (590h53m11.449626027s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.578846911 (590h53m11.449929946s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579458044 (590h53m11.450541089s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579891209 (590h53m11.450974245s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.579915535 (590h53m11.45099858s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580325466 (590h53m11.451408511s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580379599 (590h53m11.451462644s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580692044 (590h53m11.451775089s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.580857859 (590h53m11.451940894s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581089210 (590h53m11.452172255s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581351320 (590h53m11.452434355s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581457091 (590h53m11.452540126s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.581863144 (590h53m11.452946179s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.584010443 (590h53m11.455093478s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:11.585122791 (590h53m11.456205826s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Uploading piece...
Dec 30 06:22:12.983305402 (590h53m12.854388447s) [upload=517998]: Next returned nil segment
Dec 30 06:22:12.983321313 (590h53m12.854404348s) [upload=517998]: Waiting for error group managing segments...
Dec 30 06:22:13.379703053 (590h53m13.250786088s) [noise=true , address=64.49.13.27:4965     , piece_id=CO4O5F2IBZAAKBZBV3HWNNOXZOGLNNDPALWMQZ3DEM3MREASVFEA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.418055622 (590h53m13.289138657s) [noise=true , address=145.79.227.3:4175    , piece_id=UCV5WQNVWEAMEDBWDNXEVSQEGXPJE7KD6YSWC5A4WXYUKUCK4NLA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.509034160 (590h53m13.380117205s) [noise=true , address=64.49.13.154:4928    , piece_id=SEVBZBMNGURP5TRUEBA4PHRSA6MNANVJUKINALJUWPQWBGKVYNTA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.642886154 (590h53m13.513969199s) [noise=true , address=77.90.31.14:4671     , piece_id=263TRISJ5ADFTX43ULSSGBAF2KMOXAPISJ56LLE53D74XOEMQ5EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.788145520 (590h53m13.659228555s) [noise=true , address=77.90.31.12:4603     , piece_id=2I4L7F2WEGK5OSLODJSJ4672URBA7DUGXWZKTYB6ZNA2TQIGVXWA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.858878757 (590h53m13.729961792s) [noise=true , address=145.79.227.49:4369   , piece_id=XUEXGNJJIJ4DUBDNUV7CKF6J5JYHSLZAEPVGEDGBUBY42BIP4XEQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.879550569 (590h53m13.750633614s) [noise=true , address=145.79.227.34:4909   , piece_id=3QSWTNR5LGUUAY3N5HJKSPL3TRQ6WO5F5OD4VTXKWHWUXUF7PVVQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.940897723 (590h53m13.811980769s) [noise=true , address=77.90.31.19:4860     , piece_id=U2MNFOAD7FHCDBE4FUYOROIP3P33ROJ7K75SF5Z747EFKFTF7RGA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:13.989791936 (590h53m13.860874981s) [noise=true , address=45.91.116.11:4150    , piece_id=E66IDHTJDLYLYJIOWMOIKOOWLXCUNCMBIQ7SQBAD3VTJF6VX5JLQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.289516718 (590h53m14.160599753s) [noise=true , address=145.79.227.44:4203   , piece_id=WHCKQ2FRLYAZE7Q6HYZNTS4EISVHZNAPDDKH34WN33ALK4MVK2EA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.338651537 (590h53m14.209734582s) [noise=true , address=77.90.31.15:4729     , piece_id=NVMXFEILHGT622IIPYYT4FFHC2WZLHMCZM4VT5WUPHRAPH3DX6TA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.344145168 (590h53m14.215228213s) [noise=true , address=145.79.227.30:4772   , piece_id=PVMD5DWRRPVARTSZ6O7HN6MUVTZUYWLYSNQR5GDMNKUJZGXMYPFQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.667232847 (590h53m14.538315892s) [noise=true , address=193.149.164.143:4704 , piece_id=4T2IJIA6WHI5LMIUP5BLJLCD3GRY3A6PXR6G5OWJWUSYZT5ELGHQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671388990 (590h53m14.542472025s) [noise=true , address=145.79.227.29:4945   , piece_id=SHL73IDEI54GRCWYNWHRUSJZZ4UUXREAX54DFJLTG5QPZCF5QKIQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: <nil>
Dec 30 06:22:14.671415190 (590h53m14.542498226s) [seg_pos={0 0}, upload=517998]: Segment reached optimal threshold of 14 pieces.
Dec 30 06:22:14.671803600 (590h53m14.542886655s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12hU1LwL2gX7rVnrNgzaBCYB37ziqf7s7aUgtF2Uc3antReQ6me): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.671814470 (590h53m14.542897525s) [noise=true , address=145.79.227.40:4070   , piece_id=YOAGDXNN4TTCK35LLQDSP24BOWLZXGNSQU7WMKEEVVMPXDVI74JA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672014932 (590h53m14.543097977s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672022637 (590h53m14.543105672s) [noise=true , address=145.79.227.43:4177   , piece_id=YM37ZI3WNZ7SXGTFC224TOVVTHYNFIPRQB2TFLEZ735RNZE553FQ, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672088933 (590h53m14.543171988s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mMBjg1b4MqTZPE5NpNNMYDzmsHqWYotE5jBNmZLyA5FyYLZRp): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672127677 (590h53m14.543210742s) [noise=true , address=145.79.227.33:4881   , piece_id=EX66JF7U7A4ZSRWJMP4MCLQRPW2ZZFZFIUNV72XZXU7E5O3KWX3A, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672312739 (590h53m14.543395774s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1xSXWFiq27Dd2vBDD9vVnhDHVG59UmDKn3hVAhf3oyo5goVNaE): protocol: expected piece hash; context canceled; EOF
Dec 30 06:22:14.672319953 (590h53m14.543402988s) [noise=true , address=45.91.116.14:4217    , piece_id=L4MAHTGKTNWBSN3YTB74E3DBJWVMA3DPU57MQK5FNK35DXRZ6AIA, seg_pos={0 0}, upload=517998]: Long tail cancellation due to optimal threshold
Dec 30 06:22:14.672337847 (590h53m14.543420882s) [seg_pos={0 0}, upload=517998]: Done waiting for segment. successful: 14 optimal: 14 errs: <nil>
Dec 30 06:22:14.672452947 (590h53m14.543535982s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Dec 30 06:22:14.672458507 (590h53m14.543541542s) [upload=517998]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Dec 30 06:22:14.981437069 (590h53m14.852520114s) [upload=517998]: Done uploading segments...
Dec 30 06:22:14.981456546 (590h53m14.852539581s) [upload=517998]: Upload finished. err: <nil>
Duration: 45.593456757s core uplink us-select-1 us1-placement-20