Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX.X..X] duration: 30.003541804s trace: https://jaeger.dev.storj.io/trace/471dd350cb320996 FAIL (2025-12-27 17:15:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq, Piece ID: OF4KDMZALTLPF6BB55LZ4FJBOCYFUIH3EEKNQKOQYIU53QWDZ2MQ) context deadline exceeded; piecestore: (Node ID: 167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6, Piece ID: RGSIT3OLNKOOST2I4DFIESMVJ36QIXVX6TVIJTVH54QAYFX74I4Q) context deadline exceeded; piecestore: (Node ID: 121tumRJ6tGiip2dumJr9ZWMoU5BA2HuLEwU5GxBxzTZjMhv3Pp, Piece ID: 7ZVCPRBCWEXPY4NKI2PTRPQRCSVBMTODM5XF6ESLCCQG3KCKK5NQ) context deadline exceeded ======LOGS====== c 27 17:14:43.517303476 (529h45m43.388386512s) [upload=464628]: Starting upload Dec 27 17:14:43.517364022 (529h45m43.388447067s) [upload=464628]: Done starting upload Dec 27 17:14:43.517440447 (529h45m43.388523492s) [upload=464628]: Uploading segments... Dec 27 17:14:43.517620410 (529h45m43.388703455s) [upload=464628]: Got next segment. Inline: false Dec 27 17:14:43.517712836 (529h45m43.388795881s) [upload=464628]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:14:43.517719318 (529h45m43.388802363s) [upload=464628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:14:43.765542076 (529h45m43.636625111s) [seg_pos={0 0}, upload=464628]: Begin upload segment... Dec 27 17:14:43.765589235 (529h45m43.63667227s) [seg_pos={0 0}, upload=464628]: Done begin upload segment. Dec 27 17:14:43.765617529 (529h45m43.636700564s) [noise=true , address=145.79.227.37:4566 , piece_id=JUI6WZDNSEOULYYIKXXWO35QG4A2SCCZU4J4OIIR7WDBRJA5CGQA, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.765760251 (529h45m43.636843296s) [noise=true , address=45.91.116.6:4976 , piece_id=7ZVCPRBCWEXPY4NKI2PTRPQRCSVBMTODM5XF6ESLCCQG3KCKK5NQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.766105628 (529h45m43.637188674s) [noise=true , address=145.79.227.48:4302 , piece_id=NJH5O6TSGJ4VQAVWDSYZKMVKCUQ4HCNEMWUO62PYVRAW7AFKAICQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.766673980 (529h45m43.637757035s) [noise=true , address=145.79.227.16:4910 , piece_id=6ULLMN5HYSZ6WCCMLPZI2VPZHLHPGGKELX2VZIVVVJOYOL2CIWQQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.767091525 (529h45m43.638174571s) [noise=true , address=145.79.227.9:4365 , piece_id=H2IZ46MK4OQSWTHZH77B3AETGST5DJAMAE6KGE3TFW66HBGZAOVQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.767506075 (529h45m43.63858912s) [noise=true , address=77.90.31.21:4880 , piece_id=HKCBF7CQP2WPQTIGVTSNRAABQPJ2BTVIJUO5DRG664ZMRYRFRFVQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.767952786 (529h45m43.639035821s) [noise=true , address=77.90.31.10:4520 , piece_id=BRXWVFVD5NMSNCCF6ZXOYP5AOBBH7ZLMSVPWIQGQ5P2PVYOVY3WQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.768390900 (529h45m43.639473935s) [noise=true , address=45.91.116.14:4191 , piece_id=GYE7N7WMDBK4QNJEX5ZSIEMGW7HQFBGHP37NDO2LROSSL56AC4PQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.769217504 (529h45m43.640300539s) [noise=true , address=145.79.227.2:4113 , piece_id=ORCZKTOLM42YKYIDM6LCK7HGLWUHOREY456U5SVK3YXLFNK3QWCA, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.769937695 (529h45m43.64102074s) [noise=true , address=145.79.227.45:4229 , piece_id=SYBM4N4WKLZQPWTBXAMLPXQJWQK2VMXF4F5L6D3YXPYN6WRMMUCQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.770510867 (529h45m43.641593912s) [noise=true , address=145.79.227.5:4223 , piece_id=EOMDTYBMLWKFUOXVH7X5FFHDIQPTHVEF3FYHC7EFO4XAIRZZJTAQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.770970302 (529h45m43.642053347s) [noise=true , address=145.79.227.40:4051 , piece_id=TV55SBY5XAPHF6Z6V22SV56FO7YVMCFRCZRZNSQ3TGXC2LMHZOEQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.771521271 (529h45m43.642604306s) [noise=true , address=45.91.116.5:4300 , piece_id=OF4KDMZALTLPF6BB55LZ4FJBOCYFUIH3EEKNQKOQYIU53QWDZ2MQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.771945419 (529h45m43.643028464s) [noise=true , address=45.91.116.11:4117 , piece_id=RGSIT3OLNKOOST2I4DFIESMVJ36QIXVX6TVIJTVH54QAYFX74I4Q, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.772375066 (529h45m43.643458112s) [noise=true , address=145.79.227.14:4839 , piece_id=W332XC3PLIPBC46USDKJIXDEVEKF7XYXNDXYQ4DIEGSBGTV55SSQ, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:43.772964218 (529h45m43.644047263s) [noise=true , address=45.91.116.4:4939 , piece_id=FY62IAJZSONPOZCMBNP5Y6KCNGKJNU4V644Y7F7OSMR5KKSZ35TA, seg_pos={0 0}, upload=464628]: Uploading piece... Dec 27 17:14:45.475216009 (529h45m45.346299055s) [upload=464628]: Next returned nil segment Dec 27 17:14:45.475234615 (529h45m45.34631768s) [upload=464628]: Waiting for error group managing segments... Dec 27 17:14:45.760983981 (529h45m45.632067026s) [noise=true , address=145.79.227.37:4566 , piece_id=JUI6WZDNSEOULYYIKXXWO35QG4A2SCCZU4J4OIIR7WDBRJA5CGQA, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.767450024 (529h45m45.638533059s) [noise=true , address=145.79.227.14:4839 , piece_id=W332XC3PLIPBC46USDKJIXDEVEKF7XYXNDXYQ4DIEGSBGTV55SSQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.770408778 (529h45m45.641491813s) [noise=true , address=145.79.227.9:4365 , piece_id=H2IZ46MK4OQSWTHZH77B3AETGST5DJAMAE6KGE3TFW66HBGZAOVQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.771902913 (529h45m45.642985958s) [noise=true , address=145.79.227.48:4302 , piece_id=NJH5O6TSGJ4VQAVWDSYZKMVKCUQ4HCNEMWUO62PYVRAW7AFKAICQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.785562402 (529h45m45.656645447s) [noise=true , address=145.79.227.45:4229 , piece_id=SYBM4N4WKLZQPWTBXAMLPXQJWQK2VMXF4F5L6D3YXPYN6WRMMUCQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.802417998 (529h45m45.673501033s) [noise=true , address=145.79.227.16:4910 , piece_id=6ULLMN5HYSZ6WCCMLPZI2VPZHLHPGGKELX2VZIVVVJOYOL2CIWQQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:45.804501445 (529h45m45.675584491s) [noise=true , address=145.79.227.2:4113 , piece_id=ORCZKTOLM42YKYIDM6LCK7HGLWUHOREY456U5SVK3YXLFNK3QWCA, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.044180616 (529h45m45.915263651s) [noise=true , address=45.91.116.11:4117 , piece_id=RGSIT3OLNKOOST2I4DFIESMVJ36QIXVX6TVIJTVH54QAYFX74I4Q, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.046182869 (529h45m45.917265914s) [noise=true , address=45.91.116.4:4939 , piece_id=FY62IAJZSONPOZCMBNP5Y6KCNGKJNU4V644Y7F7OSMR5KKSZ35TA, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.062731971 (529h45m45.933815016s) [noise=true , address=77.90.31.10:4520 , piece_id=BRXWVFVD5NMSNCCF6ZXOYP5AOBBH7ZLMSVPWIQGQ5P2PVYOVY3WQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.201585542 (529h45m46.072668577s) [noise=true , address=145.79.227.5:4223 , piece_id=EOMDTYBMLWKFUOXVH7X5FFHDIQPTHVEF3FYHC7EFO4XAIRZZJTAQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.798106000 (529h45m46.669189135s) [noise=true , address=45.91.116.5:4300 , piece_id=OF4KDMZALTLPF6BB55LZ4FJBOCYFUIH3EEKNQKOQYIU53QWDZ2MQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.859088652 (529h45m46.730171687s) [noise=true , address=45.91.116.6:4976 , piece_id=7ZVCPRBCWEXPY4NKI2PTRPQRCSVBMTODM5XF6ESLCCQG3KCKK5NQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: <nil> Dec 27 17:14:46.859109511 (529h45m46.730192546s) [seg_pos={0 0}, upload=464628]: Segment reached optimal threshold of 13 pieces. Dec 27 17:14:46.859347013 (529h45m46.730430058s) [noise=true , address=77.90.31.21:4880 , piece_id=HKCBF7CQP2WPQTIGVTSNRAABQPJ2BTVIJUO5DRG664ZMRYRFRFVQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:59358->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:59358->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59358->77.90.31.21:4880: use of closed network connection Dec 27 17:14:46.859354006 (529h45m46.730437041s) [noise=true , address=77.90.31.21:4880 , piece_id=HKCBF7CQP2WPQTIGVTSNRAABQPJ2BTVIJUO5DRG664ZMRYRFRFVQ, seg_pos={0 0}, upload=464628]: Long tail cancellation due to optimal threshold Dec 27 17:14:46.859381519 (529h45m46.730464554s) [noise=true , address=45.91.116.14:4191 , piece_id=GYE7N7WMDBK4QNJEX5ZSIEMGW7HQFBGHP37NDO2LROSSL56AC4PQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq): protocol: expected piece hash; context canceled; EOF Dec 27 17:14:46.859396968 (529h45m46.730480003s) [noise=true , address=45.91.116.14:4191 , piece_id=GYE7N7WMDBK4QNJEX5ZSIEMGW7HQFBGHP37NDO2LROSSL56AC4PQ, seg_pos={0 0}, upload=464628]: Long tail cancellation due to optimal threshold Dec 27 17:14:46.859518941 (529h45m46.730601976s) [noise=true , address=145.79.227.40:4051 , piece_id=TV55SBY5XAPHF6Z6V22SV56FO7YVMCFRCZRZNSQ3TGXC2LMHZOEQ, seg_pos={0 0}, upload=464628]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12rPbTARUrq7YsD8X75ocnyEvC517itk79s3Dp1wUg72fVuBS8n): protocol: expected piece hash; context canceled; EOF Dec 27 17:14:46.859525092 (529h45m46.730608127s) [noise=true , address=145.79.227.40:4051 , piece_id=TV55SBY5XAPHF6Z6V22SV56FO7YVMCFRCZRZNSQ3TGXC2LMHZOEQ, seg_pos={0 0}, upload=464628]: Long tail cancellation due to optimal threshold Dec 27 17:14:46.859540491 (529h45m46.730623526s) [seg_pos={0 0}, upload=464628]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:14:46.859627297 (529h45m46.730710332s) [upload=464628]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:14:46.859633809 (529h45m46.730716844s) [upload=464628]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:14:47.150967168 (529h45m47.022050213s) [upload=464628]: Done uploading segments... Dec 27 17:14:47.150986014 (529h45m47.022069049s) [upload=464628]: Upload finished. err: <nil> Duration: 30.00356594s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................X..............X...........................] duration: 3.608305083s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:14:53 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.608338225s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................X..............X..........................] duration: 3.493001256s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:14:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.49302432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX.X..] duration: 5.698179234s trace: https://jaeger.dev.storj.io/trace/58978b586312f97f FAIL (2025-12-27 17:14:19 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 17:12:40.951267953 (529h43m40.822350998s) [upload=464596]: Starting upload Dec 27 17:12:40.951332696 (529h43m40.822415731s) [upload=464596]: Done starting upload Dec 27 17:12:40.951543367 (529h43m40.822626412s) [upload=464596]: Uploading segments... Dec 27 17:12:40.951582573 (529h43m40.822665608s) [upload=464596]: Got next segment. Inline: false Dec 27 17:12:40.951612930 (529h43m40.822695965s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:12:40.951617259 (529h43m40.822700294s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:12:41.197676968 (529h43m41.068760003s) [seg_pos={0 0}, upload=464596]: Begin upload segment... Dec 27 17:12:41.197762501 (529h43m41.068845546s) [seg_pos={0 0}, upload=464596]: Done begin upload segment. Dec 27 17:12:41.197793430 (529h43m41.068876475s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197827996 (529h43m41.068911031s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197976219 (529h43m41.069059264s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198507720 (529h43m41.069590765s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198899867 (529h43m41.069982902s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199284449 (529h43m41.070367494s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199646439 (529h43m41.070729494s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199997207 (529h43m41.071080252s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.202600303 (529h43m41.073683348s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.203045942 (529h43m41.074128987s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.204633194 (529h43m41.075716239s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.206042007 (529h43m41.077125042s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.207528868 (529h43m41.078611913s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.208832770 (529h43m41.079915816s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.210176640 (529h43m41.081259675s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.211352479 (529h43m41.082435524s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:42.854624943 (529h43m42.725707978s) [upload=464596]: Next returned nil segment Dec 27 17:12:42.854636175 (529h43m42.72571921s) [upload=464596]: Waiting for error group managing segments... Dec 27 17:12:43.188397032 (529h43m43.059480067s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.198712976 (529h43m43.069796011s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.207683437 (529h43m43.078766472s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.209992845 (529h43m43.08107588s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.211449559 (529h43m43.082532594s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.212574961 (529h43m43.083658007s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.214504295 (529h43m43.08558734s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.234025557 (529h43m43.105108602s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.386948489 (529h43m43.258031534s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.477842877 (529h43m43.348925922s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.486613628 (529h43m43.357696663s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.551242994 (529h43m43.422326039s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.562927985 (529h43m43.43401103s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563003618 (529h43m43.434086663s) [seg_pos={0 0}, upload=464596]: Segment reached optimal threshold of 13 pieces. Dec 27 17:12:43.563254306 (529h43m43.434337351s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563263843 (529h43m43.434346909s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563457433 (529h43m43.434540468s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563638978 (529h43m43.434722023s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563648847 (529h43m43.434731882s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563675447 (529h43m43.434758492s) [seg_pos={0 0}, upload=464596]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 27 17:12:43.563780397 (529h43m43.434863442s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:12:43.563787390 (529h43m43.434870435s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:12:43.862283469 (529h43m43.733366514s) [upload=464596]: Done uploading segments... Dec 27 17:12:43.862370504 (529h43m43.733453549s) [upload=464596]: Upload finished. err: <nil> Duration: 5.698203791s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX.X.] duration: 5.81396413s trace: https://jaeger.dev.storj.io/trace/58978b586312f97f FAIL (2025-12-27 17:13:58 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 17:12:40.951267953 (529h43m40.822350998s) [upload=464596]: Starting upload Dec 27 17:12:40.951332696 (529h43m40.822415731s) [upload=464596]: Done starting upload Dec 27 17:12:40.951543367 (529h43m40.822626412s) [upload=464596]: Uploading segments... Dec 27 17:12:40.951582573 (529h43m40.822665608s) [upload=464596]: Got next segment. Inline: false Dec 27 17:12:40.951612930 (529h43m40.822695965s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:12:40.951617259 (529h43m40.822700294s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:12:41.197676968 (529h43m41.068760003s) [seg_pos={0 0}, upload=464596]: Begin upload segment... Dec 27 17:12:41.197762501 (529h43m41.068845546s) [seg_pos={0 0}, upload=464596]: Done begin upload segment. Dec 27 17:12:41.197793430 (529h43m41.068876475s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197827996 (529h43m41.068911031s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197976219 (529h43m41.069059264s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198507720 (529h43m41.069590765s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198899867 (529h43m41.069982902s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199284449 (529h43m41.070367494s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199646439 (529h43m41.070729494s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199997207 (529h43m41.071080252s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.202600303 (529h43m41.073683348s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.203045942 (529h43m41.074128987s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.204633194 (529h43m41.075716239s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.206042007 (529h43m41.077125042s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.207528868 (529h43m41.078611913s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.208832770 (529h43m41.079915816s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.210176640 (529h43m41.081259675s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.211352479 (529h43m41.082435524s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:42.854624943 (529h43m42.725707978s) [upload=464596]: Next returned nil segment Dec 27 17:12:42.854636175 (529h43m42.72571921s) [upload=464596]: Waiting for error group managing segments... Dec 27 17:12:43.188397032 (529h43m43.059480067s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.198712976 (529h43m43.069796011s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.207683437 (529h43m43.078766472s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.209992845 (529h43m43.08107588s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.211449559 (529h43m43.082532594s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.212574961 (529h43m43.083658007s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.214504295 (529h43m43.08558734s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.234025557 (529h43m43.105108602s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.386948489 (529h43m43.258031534s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.477842877 (529h43m43.348925922s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.486613628 (529h43m43.357696663s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.551242994 (529h43m43.422326039s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.562927985 (529h43m43.43401103s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563003618 (529h43m43.434086663s) [seg_pos={0 0}, upload=464596]: Segment reached optimal threshold of 13 pieces. Dec 27 17:12:43.563254306 (529h43m43.434337351s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563263843 (529h43m43.434346909s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563457433 (529h43m43.434540468s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563638978 (529h43m43.434722023s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563648847 (529h43m43.434731882s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563675447 (529h43m43.434758492s) [seg_pos={0 0}, upload=464596]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 27 17:12:43.563780397 (529h43m43.434863442s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:12:43.563787390 (529h43m43.434870435s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:12:43.862283469 (529h43m43.733366514s) [upload=464596]: Done uploading segments... Dec 27 17:12:43.862370504 (529h43m43.733453549s) [upload=464596]: Upload finished. err: <nil> Duration: 5.814059602s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................X..............X.........................] duration: 3.595641652s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:13:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.5956672s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................X..............X........................] duration: 3.527407549s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:13:17 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.527436415s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXX.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX.X] duration: 30.003813713s trace: https://jaeger.dev.storj.io/trace/58978b586312f97f FAIL (2025-12-27 17:13:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 27 17:12:40.951267953 (529h43m40.822350998s) [upload=464596]: Starting upload Dec 27 17:12:40.951332696 (529h43m40.822415731s) [upload=464596]: Done starting upload Dec 27 17:12:40.951543367 (529h43m40.822626412s) [upload=464596]: Uploading segments... Dec 27 17:12:40.951582573 (529h43m40.822665608s) [upload=464596]: Got next segment. Inline: false Dec 27 17:12:40.951612930 (529h43m40.822695965s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:12:40.951617259 (529h43m40.822700294s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:12:41.197676968 (529h43m41.068760003s) [seg_pos={0 0}, upload=464596]: Begin upload segment... Dec 27 17:12:41.197762501 (529h43m41.068845546s) [seg_pos={0 0}, upload=464596]: Done begin upload segment. Dec 27 17:12:41.197793430 (529h43m41.068876475s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197827996 (529h43m41.068911031s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.197976219 (529h43m41.069059264s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198507720 (529h43m41.069590765s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.198899867 (529h43m41.069982902s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199284449 (529h43m41.070367494s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199646439 (529h43m41.070729494s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.199997207 (529h43m41.071080252s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.202600303 (529h43m41.073683348s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.203045942 (529h43m41.074128987s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.204633194 (529h43m41.075716239s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.206042007 (529h43m41.077125042s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.207528868 (529h43m41.078611913s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.208832770 (529h43m41.079915816s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.210176640 (529h43m41.081259675s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:41.211352479 (529h43m41.082435524s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Uploading piece... Dec 27 17:12:42.854624943 (529h43m42.725707978s) [upload=464596]: Next returned nil segment Dec 27 17:12:42.854636175 (529h43m42.72571921s) [upload=464596]: Waiting for error group managing segments... Dec 27 17:12:43.188397032 (529h43m43.059480067s) [noise=true , address=145.79.227.10:4403 , piece_id=LR4LLC4YIZVDMJAHTR5WDUN5J2C4C4ILRA3YZYYIGTQTWMLAX2LQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.198712976 (529h43m43.069796011s) [noise=true , address=145.79.227.8:4329 , piece_id=MDFVJOZEJSN2VKREPKCLN4SHI3Y4ON5BD2D5HXX6OIUD3PPRMYYA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.207683437 (529h43m43.078766472s) [noise=true , address=145.79.227.46:4603 , piece_id=AWHIOGFMESB4RYLOH465LE22ZHATR44GAELMQKHNUTZ23Y6PA5MQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.209992845 (529h43m43.08107588s) [noise=true , address=145.79.227.51:4409 , piece_id=FHCFPL3SXFM5FNOOTWYG4JWGH7H73KB7MNO2RYKULKHHF5TQTLMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.211449559 (529h43m43.082532594s) [noise=true , address=145.79.227.35:4481 , piece_id=CHBRX3KH26FMR2HXFBISO6GTPRFNCD6K7AE26ZEQKQIKAO6AWH2A, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.212574961 (529h43m43.083658007s) [noise=true , address=145.79.227.38:4979 , piece_id=WH7ARUHOOLSSJS5YZTGR2JYICIDZJ537RQDQHKTKCKHXFCQXX3WQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.214504295 (529h43m43.08558734s) [noise=true , address=145.79.227.28:4725 , piece_id=ROQNMPEOUM7TRSZ3FNOJEU7U2R3E6UI5OSFLMHGXH6XC7QZTGZCQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.234025557 (529h43m43.105108602s) [noise=true , address=145.79.227.48:4301 , piece_id=3INRCDR7VUCBIJNR6PIXC7DB55I3R2YXYMCYLDPVT3TT2QCJCMBQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.386948489 (529h43m43.258031534s) [noise=true , address=77.90.31.16:4736 , piece_id=4NQIUYG43BAW4NLXCH3HL72UPAWRVHPIK2HUNND5CIRG3NZZ2GYQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.477842877 (529h43m43.348925922s) [noise=true , address=45.91.116.11:4118 , piece_id=TZ6DINES4LHIOVN2PGY5POYALUXUXRTCYASXZVM6KUKQWRUAX6AQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.486613628 (529h43m43.357696663s) [noise=true , address=45.91.116.9:4261 , piece_id=NXIPZOVAV3FHY3XSLQ2HPRIKEXNZ55MBWN3TX4C4NJIEA6ULPVMQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.551242994 (529h43m43.422326039s) [noise=true , address=45.91.116.13:4154 , piece_id=ETX4ITXYUKXWHPH6IIAV6CYV6ODN6I44EZKXJOLSV6S3RSALPWBA, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.562927985 (529h43m43.43401103s) [noise=true , address=45.91.116.12:4227 , piece_id=RQPE3CXDSBYJU73BXZI4WPOVF5V64T5P4YAQAMMIBOP3RJS6MQZQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563003618 (529h43m43.434086663s) [seg_pos={0 0}, upload=464596]: Segment reached optimal threshold of 13 pieces. Dec 27 17:12:43.563254306 (529h43m43.434337351s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563263843 (529h43m43.434346909s) [noise=true , address=45.91.116.10:4083 , piece_id=CKZB5YQXNVWS6YHBHFOY6XNHU3E23GILOAH2G6UI5L4XLTPUOVJQ, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563457433 (529h43m43.434540468s) [noise=true , address=45.91.116.7:4011 , piece_id=WRR6S6NOVYUXUPUSZG3U2ZR7WLRZXF7IJWZR5UAE7MWXLB5CRLEQ, seg_pos={0 0}, upload=464596]: Done uploading piece. err: <nil> Dec 27 17:12:43.563638978 (529h43m43.434722023s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1mpRyPuesFpUqZvgaXpQ2HDm8ccFRBJiw1QBfSN9WrRm8QSKdZ): protocol: expected piece hash; context canceled; EOF Dec 27 17:12:43.563648847 (529h43m43.434731882s) [noise=true , address=77.90.31.17:4773 , piece_id=3XD7V7L36EWQ2PP65TZHIZT3IRBIQSO2BUXDAH4ORK6QV4NDCE4Q, seg_pos={0 0}, upload=464596]: Long tail cancellation due to optimal threshold Dec 27 17:12:43.563675447 (529h43m43.434758492s) [seg_pos={0 0}, upload=464596]: Done waiting for segment. successful: 14 optimal: 13 errs: <nil> Dec 27 17:12:43.563780397 (529h43m43.434863442s) [upload=464596]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:12:43.563787390 (529h43m43.434870435s) [upload=464596]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:12:43.862283469 (529h43m43.733366514s) [upload=464596]: Done uploading segments... Dec 27 17:12:43.862370504 (529h43m43.733453549s) [upload=464596]: Upload finished. err: <nil> Duration: 30.00383817s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................X..............X.......................] duration: 3.549778159s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:13:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.54980512s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................X..............X......................] duration: 3.616979932s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:11:58 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.617007494s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX.] duration: 6.132133428s trace: https://jaeger.dev.storj.io/trace/2e442b9ed20dcac5 FAIL (2025-12-27 17:11:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q) context deadline exceeded ======LOGS====== 8795360 (529h41m26.139878405s) [upload=464565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:10:26.511165995 (529h41m26.38224904s) [seg_pos={0 0}, upload=464565]: Begin upload segment... Dec 27 17:10:26.511229476 (529h41m26.382312521s) [seg_pos={0 0}, upload=464565]: Done begin upload segment. Dec 27 17:10:26.511356458 (529h41m26.382439493s) [noise=true , address=45.91.116.13:4156 , piece_id=OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.512701418 (529h41m26.383784473s) [noise=true , address=77.90.31.19:4841 , piece_id=TXHE2G6S5RSO45SCBF45RIMFWU7VTI5KGVRJGS7RT4LAT6PPYGFA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.513604197 (529h41m26.384687233s) [noise=true , address=145.79.227.29:4943 , piece_id=JPHD35SD6R3GR5RMPCM5O6EMXOB3QH5MFZ77RE4C7BXQMBTPUL6Q, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.516251498 (529h41m26.387334543s) [noise=true , address=45.91.116.7:4009 , piece_id=P5CR2XZDFNNMBOPSOXBAIHQ2PML4KHEPGQMRG7BEGVZXDMTVVMEQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.516486155 (529h41m26.387569191s) [noise=true , address=45.91.116.9:4263 , piece_id=3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.518225037 (529h41m26.389308082s) [noise=true , address=145.79.227.14:4839 , piece_id=Y4BSEBUT3NVEDTTWUI66PCEUMR2KTNEEVQJJGUNUZTXI5ZS5YTSQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.519848518 (529h41m26.390931553s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.520586914 (529h41m26.391669959s) [noise=true , address=145.79.227.17:4970 , piece_id=JK4YBKBDMZM3L54VBDSDZJ3ADXCRAZXF2KEQUGYGFDM6VHAPHT7A, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.522013411 (529h41m26.393096456s) [noise=true , address=145.79.227.11:4731 , piece_id=YACBWSX7VBNDW3I5HFO6DYTE2LMFUGBWVV4444TSDB7RA3WG5AWA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.522588866 (529h41m26.393671911s) [noise=true , address=145.79.227.15:4876 , piece_id=4RCFXMURC5IZE745ASI3FZHLP5MBP6ZGBBYSPSOWLIBMLOUCBAJQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.523608518 (529h41m26.394691563s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.524611808 (529h41m26.395694853s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.526138725 (529h41m26.39722176s) [noise=true , address=145.79.227.47:4265 , piece_id=CLM6LQQ4USVGVZLWPIJCROSLSOXQI3D2MI2JBZOT74J7OC24UGPQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.528205461 (529h41m26.399288516s) [noise=true , address=145.79.227.40:4050 , piece_id=FJJ3H7DC5OGUUY2ELCSNZEQDQC6BDQINR7VQQLW57NXSXHKL42YQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.529513301 (529h41m26.400596336s) [noise=true , address=77.90.31.10:4518 , piece_id=K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.530945538 (529h41m26.402028583s) [noise=true , address=145.79.227.48:4303 , piece_id=7JG4EIXGUGH6TX54KZOJX4P4QWQOMZACVQNRJ2TH5LHUN7HZOKLQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:28.260475522 (529h41m28.131558567s) [upload=464565]: Next returned nil segment Dec 27 17:10:28.260493005 (529h41m28.13157605s) [upload=464565]: Waiting for error group managing segments... Dec 27 17:10:28.544537004 (529h41m28.415620039s) [noise=true , address=145.79.227.15:4876 , piece_id=4RCFXMURC5IZE745ASI3FZHLP5MBP6ZGBBYSPSOWLIBMLOUCBAJQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.544714422 (529h41m28.415797467s) [noise=true , address=145.79.227.29:4943 , piece_id=JPHD35SD6R3GR5RMPCM5O6EMXOB3QH5MFZ77RE4C7BXQMBTPUL6Q, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.547448798 (529h41m28.418531843s) [noise=true , address=145.79.227.14:4839 , piece_id=Y4BSEBUT3NVEDTTWUI66PCEUMR2KTNEEVQJJGUNUZTXI5ZS5YTSQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.547637618 (529h41m28.418720663s) [noise=true , address=145.79.227.17:4970 , piece_id=JK4YBKBDMZM3L54VBDSDZJ3ADXCRAZXF2KEQUGYGFDM6VHAPHT7A, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.549319320 (529h41m28.420402365s) [noise=true , address=145.79.227.48:4303 , piece_id=7JG4EIXGUGH6TX54KZOJX4P4QWQOMZACVQNRJ2TH5LHUN7HZOKLQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.551116723 (529h41m28.422199758s) [noise=true , address=145.79.227.47:4265 , piece_id=CLM6LQQ4USVGVZLWPIJCROSLSOXQI3D2MI2JBZOT74J7OC24UGPQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.551583071 (529h41m28.422666116s) [noise=true , address=145.79.227.11:4731 , piece_id=YACBWSX7VBNDW3I5HFO6DYTE2LMFUGBWVV4444TSDB7RA3WG5AWA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.709908279 (529h41m28.580991324s) [noise=true , address=77.90.31.10:4518 , piece_id=K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.833422942 (529h41m28.704505997s) [noise=true , address=45.91.116.7:4009 , piece_id=P5CR2XZDFNNMBOPSOXBAIHQ2PML4KHEPGQMRG7BEGVZXDMTVVMEQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.851701297 (529h41m28.722784332s) [noise=true , address=45.91.116.9:4263 , piece_id=3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.131716945 (529h41m29.00279998s) [noise=true , address=77.90.31.19:4841 , piece_id=TXHE2G6S5RSO45SCBF45RIMFWU7VTI5KGVRJGS7RT4LAT6PPYGFA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.494636937 (529h41m29.365719972s) [noise=true , address=45.91.116.13:4156 , piece_id=OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.605027370 (529h41m29.476110405s) [noise=true , address=145.79.227.40:4050 , piece_id=FJJ3H7DC5OGUUY2ELCSNZEQDQC6BDQINR7VQQLW57NXSXHKL42YQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.605053459 (529h41m29.476136504s) [seg_pos={0 0}, upload=464565]: Segment reached optimal threshold of 13 pieces. Dec 27 17:10:29.605275873 (529h41m29.476358908s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection Dec 27 17:10:29.605284970 (529h41m29.476368005s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605336127 (529h41m29.476419172s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection Dec 27 17:10:29.605347510 (529h41m29.476430565s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605530598 (529h41m29.476613643s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection Dec 27 17:10:29.605539695 (529h41m29.47662273s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605560665 (529h41m29.476643701s) [seg_pos={0 0}, upload=464565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:10:29.605645246 (529h41m29.476728281s) [upload=464565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:10:29.605651759 (529h41m29.476734794s) [upload=464565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:10:29.896490265 (529h41m29.76757331s) [upload=464565]: Done uploading segments... Dec 27 17:10:29.896514552 (529h41m29.767597597s) [upload=464565]: Upload finished. err: <nil> Duration:* 6.132167733s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................X..............X.....................] duration: 3.587478179s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:11:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.58754177s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...XXXX] duration: 30.010351071s trace: https://jaeger.dev.storj.io/trace/2e442b9ed20dcac5 FAIL (2025-12-27 17:10:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, Piece ID: OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ) context deadline exceeded; piecestore: (Node ID: 1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF, Piece ID: 3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ) context deadline exceeded; piecestore: (Node ID: 1mvEBa8qdXMR14Gcz96iRQUKyscWCUDnFXh98NQJC3ndoWkkq3, Piece ID: K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q) context deadline exceeded ======LOGS====== 8795360 (529h41m26.139878405s) [upload=464565]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:10:26.511165995 (529h41m26.38224904s) [seg_pos={0 0}, upload=464565]: Begin upload segment... Dec 27 17:10:26.511229476 (529h41m26.382312521s) [seg_pos={0 0}, upload=464565]: Done begin upload segment. Dec 27 17:10:26.511356458 (529h41m26.382439493s) [noise=true , address=45.91.116.13:4156 , piece_id=OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.512701418 (529h41m26.383784473s) [noise=true , address=77.90.31.19:4841 , piece_id=TXHE2G6S5RSO45SCBF45RIMFWU7VTI5KGVRJGS7RT4LAT6PPYGFA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.513604197 (529h41m26.384687233s) [noise=true , address=145.79.227.29:4943 , piece_id=JPHD35SD6R3GR5RMPCM5O6EMXOB3QH5MFZ77RE4C7BXQMBTPUL6Q, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.516251498 (529h41m26.387334543s) [noise=true , address=45.91.116.7:4009 , piece_id=P5CR2XZDFNNMBOPSOXBAIHQ2PML4KHEPGQMRG7BEGVZXDMTVVMEQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.516486155 (529h41m26.387569191s) [noise=true , address=45.91.116.9:4263 , piece_id=3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.518225037 (529h41m26.389308082s) [noise=true , address=145.79.227.14:4839 , piece_id=Y4BSEBUT3NVEDTTWUI66PCEUMR2KTNEEVQJJGUNUZTXI5ZS5YTSQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.519848518 (529h41m26.390931553s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.520586914 (529h41m26.391669959s) [noise=true , address=145.79.227.17:4970 , piece_id=JK4YBKBDMZM3L54VBDSDZJ3ADXCRAZXF2KEQUGYGFDM6VHAPHT7A, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.522013411 (529h41m26.393096456s) [noise=true , address=145.79.227.11:4731 , piece_id=YACBWSX7VBNDW3I5HFO6DYTE2LMFUGBWVV4444TSDB7RA3WG5AWA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.522588866 (529h41m26.393671911s) [noise=true , address=145.79.227.15:4876 , piece_id=4RCFXMURC5IZE745ASI3FZHLP5MBP6ZGBBYSPSOWLIBMLOUCBAJQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.523608518 (529h41m26.394691563s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.524611808 (529h41m26.395694853s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.526138725 (529h41m26.39722176s) [noise=true , address=145.79.227.47:4265 , piece_id=CLM6LQQ4USVGVZLWPIJCROSLSOXQI3D2MI2JBZOT74J7OC24UGPQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.528205461 (529h41m26.399288516s) [noise=true , address=145.79.227.40:4050 , piece_id=FJJ3H7DC5OGUUY2ELCSNZEQDQC6BDQINR7VQQLW57NXSXHKL42YQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.529513301 (529h41m26.400596336s) [noise=true , address=77.90.31.10:4518 , piece_id=K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:26.530945538 (529h41m26.402028583s) [noise=true , address=145.79.227.48:4303 , piece_id=7JG4EIXGUGH6TX54KZOJX4P4QWQOMZACVQNRJ2TH5LHUN7HZOKLQ, seg_pos={0 0}, upload=464565]: Uploading piece... Dec 27 17:10:28.260475522 (529h41m28.131558567s) [upload=464565]: Next returned nil segment Dec 27 17:10:28.260493005 (529h41m28.13157605s) [upload=464565]: Waiting for error group managing segments... Dec 27 17:10:28.544537004 (529h41m28.415620039s) [noise=true , address=145.79.227.15:4876 , piece_id=4RCFXMURC5IZE745ASI3FZHLP5MBP6ZGBBYSPSOWLIBMLOUCBAJQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.544714422 (529h41m28.415797467s) [noise=true , address=145.79.227.29:4943 , piece_id=JPHD35SD6R3GR5RMPCM5O6EMXOB3QH5MFZ77RE4C7BXQMBTPUL6Q, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.547448798 (529h41m28.418531843s) [noise=true , address=145.79.227.14:4839 , piece_id=Y4BSEBUT3NVEDTTWUI66PCEUMR2KTNEEVQJJGUNUZTXI5ZS5YTSQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.547637618 (529h41m28.418720663s) [noise=true , address=145.79.227.17:4970 , piece_id=JK4YBKBDMZM3L54VBDSDZJ3ADXCRAZXF2KEQUGYGFDM6VHAPHT7A, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.549319320 (529h41m28.420402365s) [noise=true , address=145.79.227.48:4303 , piece_id=7JG4EIXGUGH6TX54KZOJX4P4QWQOMZACVQNRJ2TH5LHUN7HZOKLQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.551116723 (529h41m28.422199758s) [noise=true , address=145.79.227.47:4265 , piece_id=CLM6LQQ4USVGVZLWPIJCROSLSOXQI3D2MI2JBZOT74J7OC24UGPQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.551583071 (529h41m28.422666116s) [noise=true , address=145.79.227.11:4731 , piece_id=YACBWSX7VBNDW3I5HFO6DYTE2LMFUGBWVV4444TSDB7RA3WG5AWA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.709908279 (529h41m28.580991324s) [noise=true , address=77.90.31.10:4518 , piece_id=K4LGDREFIBMCMDKHRKUMFG6OXM6AT4AJG5GWGOAGZA3O6O6QYH2Q, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.833422942 (529h41m28.704505997s) [noise=true , address=45.91.116.7:4009 , piece_id=P5CR2XZDFNNMBOPSOXBAIHQ2PML4KHEPGQMRG7BEGVZXDMTVVMEQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:28.851701297 (529h41m28.722784332s) [noise=true , address=45.91.116.9:4263 , piece_id=3CWLV4SL6E4UFKPFOIAHVTXL7AM4AG7T2GTD56A337N7THHIC2IQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.131716945 (529h41m29.00279998s) [noise=true , address=77.90.31.19:4841 , piece_id=TXHE2G6S5RSO45SCBF45RIMFWU7VTI5KGVRJGS7RT4LAT6PPYGFA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.494636937 (529h41m29.365719972s) [noise=true , address=45.91.116.13:4156 , piece_id=OOQNGKILSJZEJSWK355GETY3QXSASH34GIZ3I33DLOMNK66KJANQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.605027370 (529h41m29.476110405s) [noise=true , address=145.79.227.40:4050 , piece_id=FJJ3H7DC5OGUUY2ELCSNZEQDQC6BDQINR7VQQLW57NXSXHKL42YQ, seg_pos={0 0}, upload=464565]: Done uploading piece. err: <nil> Dec 27 17:10:29.605053459 (529h41m29.476136504s) [seg_pos={0 0}, upload=464565]: Segment reached optimal threshold of 13 pieces. Dec 27 17:10:29.605275873 (529h41m29.476358908s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj): protocol: write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection; write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33496->45.91.116.10:4084: use of closed network connection Dec 27 17:10:29.605284970 (529h41m29.476368005s) [noise=true , address=45.91.116.10:4084 , piece_id=EVSMRLFFGQP4LYTVGTY3DPQU4RMK5UV3EJQUTPFPBQN6L6DQ3AQA, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605336127 (529h41m29.476419172s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x): protocol: write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection; write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52500->145.79.227.4:4187: use of closed network connection Dec 27 17:10:29.605347510 (529h41m29.476430565s) [noise=true , address=145.79.227.4:4187 , piece_id=BT2VCK372HM235ZCMM2TSWHX7J72GXS6634LB7AX2LRA3EZZPHIA, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605530598 (529h41m29.476613643s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection; write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:41024->45.91.116.5:4298: use of closed network connection Dec 27 17:10:29.605539695 (529h41m29.47662273s) [noise=true , address=45.91.116.5:4298 , piece_id=CVW3ILPDDPXOZLVPDDMAZV2UZ4ILGN33WQH5HYZFGPCRE6SLPK5A, seg_pos={0 0}, upload=464565]: Long tail cancellation due to optimal threshold Dec 27 17:10:29.605560665 (529h41m29.476643701s) [seg_pos={0 0}, upload=464565]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:10:29.605645246 (529h41m29.476728281s) [upload=464565]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:10:29.605651759 (529h41m29.476734794s) [upload=464565]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:10:29.896490265 (529h41m29.76757331s) [upload=464565]: Done uploading segments... Dec 27 17:10:29.896514552 (529h41m29.767597597s) [upload=464565]: Upload finished. err: <nil> Duration:* 30.010376519s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................X..............X....................] duration: 3.662087625s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:10:09 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.662121842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...XXX] duration: 30.002444054s trace: https://jaeger.dev.storj.io/trace/6a5d1ee8c3c6d5c8 FAIL (2025-12-27 17:09:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12ZDTj5nit2Gd5eVLYoeY8MeuMbvCHwAMKdzhVMx1j2QKTq3x7p, Piece ID: V25SZW6X6GD33QWT5UDVQJZ7NIQRZGYXN6Z7W2RP4XSXZGUIXDJQ) context deadline exceeded; piecestore: (Node ID: 1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, Piece ID: NMPBXHNIXTUXR5IA75GY5YMCV7UGFGRPW3XKSSYCIF7TUHKVDYQA) context deadline exceeded ======LOGS====== (529h40m20.404831787s) [upload=464550]: Done starting upload Dec 27 17:09:20.534006464 (529h40m20.405089499s) [upload=464550]: Uploading segments... Dec 27 17:09:20.534170926 (529h40m20.405253971s) [upload=464550]: Got next segment. Inline: false Dec 27 17:09:20.534223697 (529h40m20.405306742s) [upload=464550]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:09:20.534230399 (529h40m20.405313435s) [upload=464550]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:09:20.779029489 (529h40m20.650112584s) [seg_pos={0 0}, upload=464550]: Begin upload segment... Dec 27 17:09:20.779123458 (529h40m20.650206493s) [seg_pos={0 0}, upload=464550]: Done begin upload segment. Dec 27 17:09:20.779173183 (529h40m20.650256249s) [noise=true , address=145.79.227.21:4474 , piece_id=7BT7PK3DLERJZOWBVU6NWHPQPWQAXLICDZTUJDHDRUTQUBTXRUKQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.780053419 (529h40m20.651136464s) [noise=true , address=45.91.116.11:4118 , piece_id=V25SZW6X6GD33QWT5UDVQJZ7NIQRZGYXN6Z7W2RP4XSXZGUIXDJQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.780526630 (529h40m20.651609715s) [noise=true , address=145.79.227.38:4978 , piece_id=QXSCYC3LWTLVBUEYKH4CTY6O4JBNKZPKHQNHCZFYDMTFYRJEGGHQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.780998940 (529h40m20.652081985s) [noise=true , address=145.79.227.47:4267 , piece_id=EWIO6VVU5N6TGDSCV2J3ORWA4AD7OWZM44BFC25DHN7M3MSEA5IA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.781466740 (529h40m20.652549785s) [noise=true , address=77.90.31.17:4770 , piece_id=OW4AYHUILEK2CY3W5RDKBNSTCRO5ONJPZJ6LN2PU75AELGWFHU2Q, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.782051253 (529h40m20.653134298s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDSMTQFCY44DDLEPWEOAJTGYL3FE7JBZ6GJNMMN4OSOKG33CWAHA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.782595850 (529h40m20.653678896s) [noise=true , address=145.79.227.41:4087 , piece_id=Z7WMH72KY223NYGREITXLZXKQUBBYFGFSXMOYYARGYBVMBWIMUNA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.783066827 (529h40m20.654149882s) [noise=true , address=77.90.31.19:4842 , piece_id=U5YNDUT36B73DIRXOM7R4GHFBPS6GLREU2WZ6TGOVSGQXQB5OBSQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.783517425 (529h40m20.65460047s) [noise=true , address=77.90.31.10:4520 , piece_id=NMPBXHNIXTUXR5IA75GY5YMCV7UGFGRPW3XKSSYCIF7TUHKVDYQA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.783971980 (529h40m20.655055025s) [noise=true , address=145.79.227.14:4837 , piece_id=TV535FPSXYHPYNGRMBUZHEY4K7TOPDALKM3AO3CMF2IUZGUNBRYA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.784550763 (529h40m20.655633798s) [noise=true , address=145.79.227.3:4151 , piece_id=AKFK2HQGI4PSGFNREXUGPC7GGWOTOEHVUC26MB2LTWFBVYIRDHEA, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.785060223 (529h40m20.656143268s) [noise=true , address=145.79.227.37:4566 , piece_id=UBL26X3RHEYCOL4TGKKQCSWZFWOW2477EBO26RBG3XH3PEVHVTPQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.785503717 (529h40m20.656586762s) [noise=true , address=45.91.116.9:4263 , piece_id=DCOYLYAJR4LYEVKRTCROBCLUGWCN7AGG53U5OGNGGAEUVZBMVZFQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.786094923 (529h40m20.657177958s) [noise=true , address=145.79.227.10:4403 , piece_id=MNXPYHVC4QJ2BYKLT72CZM76OKHUE5C4UCNBFUU3O2RDFMBDHPBQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.786493723 (529h40m20.657576768s) [noise=true , address=145.79.227.50:4374 , piece_id=ZTTG42BD2TGSHSWBC6E5WEHOJYAEKENCUY4FG54HCBQZJOBHKKQQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:20.787021277 (529h40m20.658104312s) [noise=true , address=145.79.227.27:4691 , piece_id=UNVM57IL5BQ3QZV4NPMLDV2VGVDZVOHU3WO35MRWNOAU5LGLISHQ, seg_pos={0 0}, upload=464550]: Uploading piece... Dec 27 17:09:22.405572036 (529h40m22.276655071s) [upload=464550]: Next returned nil segment Dec 27 17:09:22.405617804 (529h40m22.276700839s) [upload=464550]: Waiting for error group managing segments... Dec 27 17:09:22.789311563 (529h40m22.660394608s) [noise=true , address=145.79.227.50:4374 , piece_id=ZTTG42BD2TGSHSWBC6E5WEHOJYAEKENCUY4FG54HCBQZJOBHKKQQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.790475459 (529h40m22.661558504s) [noise=true , address=145.79.227.10:4403 , piece_id=MNXPYHVC4QJ2BYKLT72CZM76OKHUE5C4UCNBFUU3O2RDFMBDHPBQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.792024068 (529h40m22.663107113s) [noise=true , address=145.79.227.3:4151 , piece_id=AKFK2HQGI4PSGFNREXUGPC7GGWOTOEHVUC26MB2LTWFBVYIRDHEA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.792655200 (529h40m22.663738235s) [noise=true , address=145.79.227.41:4087 , piece_id=Z7WMH72KY223NYGREITXLZXKQUBBYFGFSXMOYYARGYBVMBWIMUNA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.793235825 (529h40m22.66431888s) [noise=true , address=145.79.227.27:4691 , piece_id=UNVM57IL5BQ3QZV4NPMLDV2VGVDZVOHU3WO35MRWNOAU5LGLISHQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.793767308 (529h40m22.664850353s) [noise=true , address=145.79.227.21:4474 , piece_id=7BT7PK3DLERJZOWBVU6NWHPQPWQAXLICDZTUJDHDRUTQUBTXRUKQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.804343938 (529h40m22.675426973s) [noise=true , address=145.79.227.38:4978 , piece_id=QXSCYC3LWTLVBUEYKH4CTY6O4JBNKZPKHQNHCZFYDMTFYRJEGGHQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.807413222 (529h40m22.678496267s) [noise=true , address=145.79.227.47:4267 , piece_id=EWIO6VVU5N6TGDSCV2J3ORWA4AD7OWZM44BFC25DHN7M3MSEA5IA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.819241415 (529h40m22.69032446s) [noise=true , address=145.79.227.37:4566 , piece_id=UBL26X3RHEYCOL4TGKKQCSWZFWOW2477EBO26RBG3XH3PEVHVTPQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.971661008 (529h40m22.842744043s) [noise=true , address=77.90.31.14:4663 , piece_id=ZDSMTQFCY44DDLEPWEOAJTGYL3FE7JBZ6GJNMMN4OSOKG33CWAHA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:22.974848897 (529h40m22.845931942s) [noise=true , address=145.79.227.14:4837 , piece_id=TV535FPSXYHPYNGRMBUZHEY4K7TOPDALKM3AO3CMF2IUZGUNBRYA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:23.002929478 (529h40m22.874012523s) [noise=true , address=77.90.31.10:4520 , piece_id=NMPBXHNIXTUXR5IA75GY5YMCV7UGFGRPW3XKSSYCIF7TUHKVDYQA, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:23.030629133 (529h40m22.901712178s) [noise=true , address=45.91.116.11:4118 , piece_id=V25SZW6X6GD33QWT5UDVQJZ7NIQRZGYXN6Z7W2RP4XSXZGUIXDJQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: <nil> Dec 27 17:09:23.030648571 (529h40m22.901731616s) [seg_pos={0 0}, upload=464550]: Segment reached optimal threshold of 13 pieces. Dec 27 17:09:23.031230668 (529h40m22.902313713s) [noise=true , address=77.90.31.17:4770 , piece_id=OW4AYHUILEK2CY3W5RDKBNSTCRO5ONJPZJ6LN2PU75AELGWFHU2Q, seg_pos={0 0}, upload=464550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: read tcp 188.166.225.170:36240->77.90.31.17:4770: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 17:09:23.031239435 (529h40m22.90232248s) [noise=true , address=77.90.31.17:4770 , piece_id=OW4AYHUILEK2CY3W5RDKBNSTCRO5ONJPZJ6LN2PU75AELGWFHU2Q, seg_pos={0 0}, upload=464550]: Long tail cancellation due to optimal threshold Dec 27 17:09:23.031462310 (529h40m22.902545365s) [noise=true , address=77.90.31.19:4842 , piece_id=U5YNDUT36B73DIRXOM7R4GHFBPS6GLREU2WZ6TGOVSGQXQB5OBSQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Dec 27 17:09:23.031471487 (529h40m22.902554552s) [noise=true , address=77.90.31.19:4842 , piece_id=U5YNDUT36B73DIRXOM7R4GHFBPS6GLREU2WZ6TGOVSGQXQB5OBSQ, seg_pos={0 0}, upload=464550]: Long tail cancellation due to optimal threshold Dec 27 17:09:23.031593941 (529h40m22.902676976s) [noise=true , address=45.91.116.9:4263 , piece_id=DCOYLYAJR4LYEVKRTCROBCLUGWCN7AGG53U5OGNGGAEUVZBMVZFQ, seg_pos={0 0}, upload=464550]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1TP3JkhssLqPE6onMrKvTwA49PQUBJfuZWcPnGUeJd8UqR8zrF): protocol: write tcp 188.166.225.170:52404->45.91.116.9:4263: use of closed network connection; write tcp 188.166.225.170:52404->45.91.116.9:4263: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:52404->45.91.116.9:4263: use of closed network connection Dec 27 17:09:23.031603629 (529h40m22.902686664s) [noise=true , address=45.91.116.9:4263 , piece_id=DCOYLYAJR4LYEVKRTCROBCLUGWCN7AGG53U5OGNGGAEUVZBMVZFQ, seg_pos={0 0}, upload=464550]: Long tail cancellation due to optimal threshold Dec 27 17:09:23.031629007 (529h40m22.902712042s) [seg_pos={0 0}, upload=464550]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:09:23.031697227 (529h40m22.902780272s) [upload=464550]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:09:23.031701936 (529h40m22.902784981s) [upload=464550]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:09:23.330949134 (529h40m23.202032189s) [upload=464550]: Done uploading segments... Dec 27 17:09:23.330973691 (529h40m23.202056736s) [upload=464550]: Upload finished. err: <nil> Duration: 30.002467007s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................X..............X...................] duration: 3.602261404s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:09:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.60228595s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...XX] duration: 30.003496877s trace: https://jaeger.dev.storj.io/trace/6cf5bb8a5cb7f688 FAIL (2025-12-27 17:09:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 138doLw3dLzEZXoSRNbCridRmSCajLgdVZ86vBdAQuWXjaDD8n, Piece ID: QAKSHWKP67PMNEA25UK5QN5A325D2MXF5Y3HHSHO3L7OHGOJHQUA) context deadline exceeded; piecestore: (Node ID: 12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, Piece ID: VKA5P4DCWLUMIT67N3EAUTJUQZALI7ZUD3U4CGYSTIUT4BSKBIKA) context deadline exceeded ======LOGS====== ing segments... Dec 27 17:08:45.392998990 (529h39m45.264082085s) [upload=464539]: Got next segment. Inline: false Dec 27 17:08:45.393035359 (529h39m45.264118395s) [upload=464539]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:08:45.393039828 (529h39m45.264122863s) [upload=464539]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:08:45.637408638 (529h39m45.508491683s) [seg_pos={0 0}, upload=464539]: Begin upload segment... Dec 27 17:08:45.637489161 (529h39m45.508572196s) [seg_pos={0 0}, upload=464539]: Done begin upload segment. Dec 27 17:08:45.637499933 (529h39m45.508582968s) [noise=true , address=45.91.116.10:4082 , piece_id=Y235TIOBVG2J2L5YO3VCCVGTBCEQQUABIN3OQBSUH3DOOPPNSGBQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.637530580 (529h39m45.508613615s) [noise=true , address=145.79.227.28:4725 , piece_id=YQV4LQAEEJRQGRMNPEPQK33TTE2NWUE3BKBTCOG7QD2HF3ZXV5MA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.638025984 (529h39m45.509109019s) [noise=true , address=145.79.227.41:4087 , piece_id=GUNWOQPRDLT6EDXOCSW2CKG4AJ2ZTFUEXGFI4W5K2AVPVHDIXZDQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.638499545 (529h39m45.50958258s) [noise=true , address=77.90.31.12:4589 , piece_id=XLCQEOPSSTDNS2GFSFLUNX3OZXJCIND2XJKS3B4K2FS336OEJFTA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.638552316 (529h39m45.509635371s) [noise=true , address=145.79.227.45:4231 , piece_id=EK3FRRVC2TER5NRB6FSDGL3XQG6HMUKYTMAX2U4ZHZJ5ITP4CSRA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.638894809 (529h39m45.509977844s) [noise=true , address=145.79.227.10:4403 , piece_id=SNWINDVZ2CL7BZJEQF6PHV37KILUKPCXQNLYKO53UMSYC3E3QUHQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.638997715 (529h39m45.51008076s) [noise=true , address=145.79.227.36:4517 , piece_id=NW6HDIVNZEL7WPC7BAP72DEOYHKPDJL7UXER2ZP2EI7FC7TMA3HA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.639192375 (529h39m45.51027541s) [noise=true , address=145.79.227.23:4546 , piece_id=ZTP33O2REENRBU3HUQF5S5UG7XBOF64FODSA4XVJC7LNKJN6L6CA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.639455015 (529h39m45.51053805s) [noise=true , address=77.90.31.14:4662 , piece_id=QAKSHWKP67PMNEA25UK5QN5A325D2MXF5Y3HHSHO3L7OHGOJHQUA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.639512093 (529h39m45.510595128s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q64PL7TY4LGNO5MZLKIL6F3V6LR75SDEJHV4OKUR7QHLRBOIMSQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.639844537 (529h39m45.510927582s) [noise=true , address=45.91.116.8:4047 , piece_id=VKA5P4DCWLUMIT67N3EAUTJUQZALI7ZUD3U4CGYSTIUT4BSKBIKA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.640307328 (529h39m45.511390363s) [noise=true , address=145.79.227.48:4303 , piece_id=WWWRF77XOMAWA6WU2QO3W3UBTNKEQRL323NRQCDYL6RHNS5GT2LQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.640587591 (529h39m45.511670626s) [noise=true , address=145.79.227.16:4911 , piece_id=NJDYG3SGYJ4BQBMRJOJBAXCV5M53VXBS2IU6IQRXOTF2ARWF7NGQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.640882153 (529h39m45.511965188s) [noise=true , address=145.79.227.47:4267 , piece_id=TGLKUTKQZXMRKD4AMNNYHAGWAPAAO72PRVOP6XCFSEU6ITQSUCQA, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.641343090 (529h39m45.512426125s) [noise=true , address=145.79.227.15:4876 , piece_id=IIE4M7QBCYYMOQBTGGHW4KVJJCUPPR2KELPYDB63OWCWFYVUFBNQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:45.641632711 (529h39m45.512715756s) [noise=true , address=45.91.116.5:4300 , piece_id=K4A72OAOR7LEMWVDLZQSFKTLPVZW6ZM2QQOUZ4LVWI6B5IKL2DNQ, seg_pos={0 0}, upload=464539]: Uploading piece... Dec 27 17:08:47.353943476 (529h39m47.225026522s) [upload=464539]: Next returned nil segment Dec 27 17:08:47.353960389 (529h39m47.225043434s) [upload=464539]: Waiting for error group managing segments... Dec 27 17:08:47.640428944 (529h39m47.51151201s) [noise=true , address=145.79.227.15:4876 , piece_id=IIE4M7QBCYYMOQBTGGHW4KVJJCUPPR2KELPYDB63OWCWFYVUFBNQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.644648980 (529h39m47.515732015s) [noise=true , address=145.79.227.48:4303 , piece_id=WWWRF77XOMAWA6WU2QO3W3UBTNKEQRL323NRQCDYL6RHNS5GT2LQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.645018212 (529h39m47.516101247s) [noise=true , address=145.79.227.17:4970 , piece_id=2Q64PL7TY4LGNO5MZLKIL6F3V6LR75SDEJHV4OKUR7QHLRBOIMSQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.646425743 (529h39m47.517508778s) [noise=true , address=145.79.227.10:4403 , piece_id=SNWINDVZ2CL7BZJEQF6PHV37KILUKPCXQNLYKO53UMSYC3E3QUHQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.652109345 (529h39m47.52319239s) [noise=true , address=145.79.227.47:4267 , piece_id=TGLKUTKQZXMRKD4AMNNYHAGWAPAAO72PRVOP6XCFSEU6ITQSUCQA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.654779470 (529h39m47.525862505s) [noise=true , address=145.79.227.23:4546 , piece_id=ZTP33O2REENRBU3HUQF5S5UG7XBOF64FODSA4XVJC7LNKJN6L6CA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.657908538 (529h39m47.528991584s) [noise=true , address=145.79.227.41:4087 , piece_id=GUNWOQPRDLT6EDXOCSW2CKG4AJ2ZTFUEXGFI4W5K2AVPVHDIXZDQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.660554707 (529h39m47.531637742s) [noise=true , address=145.79.227.28:4725 , piece_id=YQV4LQAEEJRQGRMNPEPQK33TTE2NWUE3BKBTCOG7QD2HF3ZXV5MA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.665403410 (529h39m47.536486455s) [noise=true , address=145.79.227.36:4517 , piece_id=NW6HDIVNZEL7WPC7BAP72DEOYHKPDJL7UXER2ZP2EI7FC7TMA3HA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.668803354 (529h39m47.539886399s) [noise=true , address=145.79.227.45:4231 , piece_id=EK3FRRVC2TER5NRB6FSDGL3XQG6HMUKYTMAX2U4ZHZJ5ITP4CSRA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.685724124 (529h39m47.556807159s) [noise=true , address=145.79.227.16:4911 , piece_id=NJDYG3SGYJ4BQBMRJOJBAXCV5M53VXBS2IU6IQRXOTF2ARWF7NGQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.840037423 (529h39m47.711120458s) [noise=true , address=77.90.31.14:4662 , piece_id=QAKSHWKP67PMNEA25UK5QN5A325D2MXF5Y3HHSHO3L7OHGOJHQUA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.912519371 (529h39m47.783602416s) [noise=true , address=45.91.116.8:4047 , piece_id=VKA5P4DCWLUMIT67N3EAUTJUQZALI7ZUD3U4CGYSTIUT4BSKBIKA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: <nil> Dec 27 17:08:47.912544289 (529h39m47.783627334s) [seg_pos={0 0}, upload=464539]: Segment reached optimal threshold of 13 pieces. Dec 27 17:08:47.912796820 (529h39m47.783879875s) [noise=true , address=45.91.116.10:4082 , piece_id=Y235TIOBVG2J2L5YO3VCCVGTBCEQQUABIN3OQBSUH3DOOPPNSGBQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: write tcp 188.166.225.170:60866->45.91.116.10:4082: use of closed network connection; write tcp 188.166.225.170:60866->45.91.116.10:4082: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:60866->45.91.116.10:4082: use of closed network connection Dec 27 17:08:47.912847165 (529h39m47.78393021s) [noise=true , address=45.91.116.10:4082 , piece_id=Y235TIOBVG2J2L5YO3VCCVGTBCEQQUABIN3OQBSUH3DOOPPNSGBQ, seg_pos={0 0}, upload=464539]: Long tail cancellation due to optimal threshold Dec 27 17:08:47.912884025 (529h39m47.78396706s) [noise=true , address=45.91.116.5:4300 , piece_id=K4A72OAOR7LEMWVDLZQSFKTLPVZW6ZM2QQOUZ4LVWI6B5IKL2DNQ, seg_pos={0 0}, upload=464539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12t8YAgwwGdjWLFyb9yNgLSFXV8QHbdyv2tnjtSZQAcBc8cZiHq): protocol: expected piece hash; context canceled; EOF Dec 27 17:08:47.912900207 (529h39m47.783983242s) [noise=true , address=45.91.116.5:4300 , piece_id=K4A72OAOR7LEMWVDLZQSFKTLPVZW6ZM2QQOUZ4LVWI6B5IKL2DNQ, seg_pos={0 0}, upload=464539]: Long tail cancellation due to optimal threshold Dec 27 17:08:47.913098154 (529h39m47.784181189s) [noise=true , address=77.90.31.12:4589 , piece_id=XLCQEOPSSTDNS2GFSFLUNX3OZXJCIND2XJKS3B4K2FS336OEJFTA, seg_pos={0 0}, upload=464539]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1jHm28wBr3S4P5omoHpbVbyYpnXvxakxkkmgVm5fthtj4HZbv): protocol: write tcp 188.166.225.170:42244->77.90.31.12:4589: use of closed network connection; write tcp 188.166.225.170:42244->77.90.31.12:4589: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:42244->77.90.31.12:4589: use of closed network connection Dec 27 17:08:47.913105437 (529h39m47.784188483s) [noise=true , address=77.90.31.12:4589 , piece_id=XLCQEOPSSTDNS2GFSFLUNX3OZXJCIND2XJKS3B4K2FS336OEJFTA, seg_pos={0 0}, upload=464539]: Long tail cancellation due to optimal threshold Dec 27 17:08:47.913130084 (529h39m47.784213129s) [seg_pos={0 0}, upload=464539]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:08:47.913229102 (529h39m47.784312137s) [upload=464539]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:08:47.913235885 (529h39m47.78431892s) [upload=464539]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:08:48.215315006 (529h39m48.086398051s) [upload=464539]: Done uploading segments... Dec 27 17:08:48.215334604 (529h39m48.086417639s) [upload=464539]: Upload finished. err: <nil> Duration: 30.00352984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...X] duration: 30.002340417s trace: https://jaeger.dev.storj.io/trace/3b3197aad1e7ef60 FAIL (2025-12-27 17:08:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 12oUsZAf26PYH66pEfjgF9ydtxCoUYspEcsrSRmdhC63woA1hf7, Piece ID: TJTF3TYVM3HHRRTS42QTVDEABWYDY6KF2ZGG3ENWSHPPPOVO7V7A) context deadline exceeded; piecestore: (Node ID: 1EZTRkB6M2F4mwRbYSvsv3ybGk1Gz225pDd18RNBk1xe1SFdGA, Piece ID: JLLSQLU3XWRN5245IUWTQR7UAWQ4G5GMAHK4AFDI55BK54QGY2LQ) context deadline exceeded ======LOGS====== 9h39m6.758521233s) [upload=464531]: Uploading segments... Dec 27 17:08:06.887671142 (529h39m6.758754187s) [upload=464531]: Got next segment. Inline: false Dec 27 17:08:06.887706159 (529h39m6.758789194s) [upload=464531]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:08:06.887710818 (529h39m6.758793863s) [upload=464531]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:08:07.135282427 (529h39m7.006365472s) [seg_pos={0 0}, upload=464531]: Begin upload segment... Dec 27 17:08:07.135340447 (529h39m7.006423482s) [seg_pos={0 0}, upload=464531]: Done begin upload segment. Dec 27 17:08:07.135369773 (529h39m7.006452808s) [noise=true , address=145.79.227.50:4373 , piece_id=XTUYTKSOJ25Z77TKMRQC3YGXI7BNCRCUEGM2CPRSPA7HEQ5CNR5A, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.135444314 (529h39m7.006527349s) [noise=true , address=77.90.31.14:4664 , piece_id=3IIZG5FIQNWPVCFJZIOMGNX7UN6EPTXUOMRIDSMI4SLRSFK2PS2Q, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.136195956 (529h39m7.007279001s) [noise=true , address=145.79.227.37:4565 , piece_id=Z44B5ZAFZNOTMJAF77DNABK3KZDQMUVSDBLDQRLNOEKPHZCXNZKA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.136542366 (529h39m7.007625411s) [noise=true , address=77.90.31.19:4841 , piece_id=JLLSQLU3XWRN5245IUWTQR7UAWQ4G5GMAHK4AFDI55BK54QGY2LQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.137673880 (529h39m7.008756915s) [noise=true , address=145.79.227.45:4231 , piece_id=2US5QJPQDXD4P2L5L7YP5GI64AF6JXBWCPSFQM34Q7JOFLVNRPDA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.138427655 (529h39m7.009510701s) [noise=true , address=145.79.227.24:4581 , piece_id=JCDI7MY2NJR3KBAZPOLKNLDN7U76GLFNKTWEQKU6TZD2T3MLWSLA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.138875599 (529h39m7.009958644s) [noise=true , address=45.91.116.6:4975 , piece_id=TJTF3TYVM3HHRRTS42QTVDEABWYDY6KF2ZGG3ENWSHPPPOVO7V7A, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.139363538 (529h39m7.010446573s) [noise=true , address=145.79.227.2:4114 , piece_id=PFTET7H6Z3DDOGX6GSI7ST64FMHOY565EIZ7QB6KGD3WKRGPVIEA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.139829705 (529h39m7.01091275s) [noise=true , address=45.91.116.5:4298 , piece_id=HNQMGSCOVITDSRQCCDKKODLPXFHWJIPQY7G35VQAOHW37VG4SG4Q, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.140387968 (529h39m7.011471013s) [noise=true , address=145.79.227.11:4729 , piece_id=73YRPI4L6QH56FJWXLWF4WE3YS7TXHO65FWDV5AWHHA6CJ6F2FUQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.140893681 (529h39m7.011976716s) [noise=true , address=145.79.227.47:4266 , piece_id=5JNDBLTHUXQH3BKBQGKXXOEU6PKRZTVT2R7LMT3L2ASOSZM34TGA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.141439260 (529h39m7.012522305s) [noise=true , address=145.79.227.44:4193 , piece_id=DU23MD3J7KOOFTZZHOFU3OFMQX5QRPIAQOS5PWV4CYGO7FSMZCCQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.141990590 (529h39m7.013073635s) [noise=true , address=45.91.116.14:4189 , piece_id=X7JJZT52EMDYJM7I7KN3UE27ZJV56SWQOCYOJIEHYOTIL7UKSZQQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.142526871 (529h39m7.013609906s) [noise=true , address=145.79.227.4:4185 , piece_id=UXYVDD6JSLVHPASX5RKRZBLBQVGV6N5D7PVMWCE47R7SRGSVD5BQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.146381430 (529h39m7.017464465s) [noise=true , address=145.79.227.15:4876 , piece_id=CDQGQ4YVVINLANBWDKBDUUPWV5IIYFBVOFFVPPWDUSLCEB6CBLAQ, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:07.147136068 (529h39m7.018219113s) [noise=true , address=145.79.227.36:4517 , piece_id=6ENRXTQNQXU62XZJKHFUMEDAL3WY7JFLN2QP5OIPCYTOMD7JO4VA, seg_pos={0 0}, upload=464531]: Uploading piece... Dec 27 17:08:08.765278711 (529h39m8.636361746s) [upload=464531]: Next returned nil segment Dec 27 17:08:08.765293799 (529h39m8.636376844s) [upload=464531]: Waiting for error group managing segments... Dec 27 17:08:09.150947310 (529h39m9.022030345s) [noise=true , address=145.79.227.45:4231 , piece_id=2US5QJPQDXD4P2L5L7YP5GI64AF6JXBWCPSFQM34Q7JOFLVNRPDA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.151584934 (529h39m9.022667969s) [noise=true , address=145.79.227.15:4876 , piece_id=CDQGQ4YVVINLANBWDKBDUUPWV5IIYFBVOFFVPPWDUSLCEB6CBLAQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.153669133 (529h39m9.024752168s) [noise=true , address=145.79.227.24:4581 , piece_id=JCDI7MY2NJR3KBAZPOLKNLDN7U76GLFNKTWEQKU6TZD2T3MLWSLA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.154233768 (529h39m9.025316813s) [noise=true , address=145.79.227.11:4729 , piece_id=73YRPI4L6QH56FJWXLWF4WE3YS7TXHO65FWDV5AWHHA6CJ6F2FUQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.156123746 (529h39m9.027206791s) [noise=true , address=145.79.227.37:4565 , piece_id=Z44B5ZAFZNOTMJAF77DNABK3KZDQMUVSDBLDQRLNOEKPHZCXNZKA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.158137912 (529h39m9.029220947s) [noise=true , address=145.79.227.36:4517 , piece_id=6ENRXTQNQXU62XZJKHFUMEDAL3WY7JFLN2QP5OIPCYTOMD7JO4VA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.161349317 (529h39m9.032432352s) [noise=true , address=145.79.227.50:4373 , piece_id=XTUYTKSOJ25Z77TKMRQC3YGXI7BNCRCUEGM2CPRSPA7HEQ5CNR5A, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.174735166 (529h39m9.045818201s) [noise=true , address=145.79.227.47:4266 , piece_id=5JNDBLTHUXQH3BKBQGKXXOEU6PKRZTVT2R7LMT3L2ASOSZM34TGA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.188956827 (529h39m9.060039872s) [noise=true , address=145.79.227.4:4185 , piece_id=UXYVDD6JSLVHPASX5RKRZBLBQVGV6N5D7PVMWCE47R7SRGSVD5BQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.403542535 (529h39m9.27462557s) [noise=true , address=45.91.116.6:4975 , piece_id=TJTF3TYVM3HHRRTS42QTVDEABWYDY6KF2ZGG3ENWSHPPPOVO7V7A, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.432728249 (529h39m9.303811284s) [noise=true , address=45.91.116.5:4298 , piece_id=HNQMGSCOVITDSRQCCDKKODLPXFHWJIPQY7G35VQAOHW37VG4SG4Q, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.443710642 (529h39m9.314793677s) [noise=true , address=77.90.31.14:4664 , piece_id=3IIZG5FIQNWPVCFJZIOMGNX7UN6EPTXUOMRIDSMI4SLRSFK2PS2Q, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.465206383 (529h39m9.336289428s) [noise=true , address=77.90.31.19:4841 , piece_id=JLLSQLU3XWRN5245IUWTQR7UAWQ4G5GMAHK4AFDI55BK54QGY2LQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: <nil> Dec 27 17:08:09.465231431 (529h39m9.336314476s) [seg_pos={0 0}, upload=464531]: Segment reached optimal threshold of 13 pieces. Dec 27 17:08:09.465465176 (529h39m9.336548241s) [noise=true , address=45.91.116.14:4189 , piece_id=X7JJZT52EMDYJM7I7KN3UE27ZJV56SWQOCYOJIEHYOTIL7UKSZQQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JPMPojBpksikw8rtjnpFkz6WrRXWKqTJb1XE6eJpAAVnXtP6N): protocol: write tcp 188.166.225.170:33900->45.91.116.14:4189: use of closed network connection; write tcp 188.166.225.170:33900->45.91.116.14:4189: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:33900->45.91.116.14:4189: use of closed network connection Dec 27 17:08:09.465475085 (529h39m9.33655815s) [noise=true , address=45.91.116.14:4189 , piece_id=X7JJZT52EMDYJM7I7KN3UE27ZJV56SWQOCYOJIEHYOTIL7UKSZQQ, seg_pos={0 0}, upload=464531]: Long tail cancellation due to optimal threshold Dec 27 17:08:09.465894954 (529h39m9.336978009s) [noise=true , address=145.79.227.2:4114 , piece_id=PFTET7H6Z3DDOGX6GSI7ST64FMHOY565EIZ7QB6KGD3WKRGPVIEA, seg_pos={0 0}, upload=464531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN): protocol: write tcp 188.166.225.170:45486->145.79.227.2:4114: use of closed network connection; write tcp 188.166.225.170:45486->145.79.227.2:4114: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:45486->145.79.227.2:4114: use of closed network connection Dec 27 17:08:09.465904613 (529h39m9.336987658s) [noise=true , address=145.79.227.2:4114 , piece_id=PFTET7H6Z3DDOGX6GSI7ST64FMHOY565EIZ7QB6KGD3WKRGPVIEA, seg_pos={0 0}, upload=464531]: Long tail cancellation due to optimal threshold Dec 27 17:08:09.466152585 (529h39m9.33723563s) [noise=true , address=145.79.227.44:4193 , piece_id=DU23MD3J7KOOFTZZHOFU3OFMQX5QRPIAQOS5PWV4CYGO7FSMZCCQ, seg_pos={0 0}, upload=464531]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1EXja3EXj5TJmk2j46Bnc6TXT7N3wgxCLjzAEjbawqvtQU7BF8): protocol: expected piece hash; context canceled; EOF Dec 27 17:08:09.466161753 (529h39m9.337244808s) [noise=true , address=145.79.227.44:4193 , piece_id=DU23MD3J7KOOFTZZHOFU3OFMQX5QRPIAQOS5PWV4CYGO7FSMZCCQ, seg_pos={0 0}, upload=464531]: Long tail cancellation due to optimal threshold Dec 27 17:08:09.466186078 (529h39m9.337269113s) [seg_pos={0 0}, upload=464531]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:08:09.466376822 (529h39m9.337459877s) [upload=464531]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:08:09.466384937 (529h39m9.337467982s) [upload=464531]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:08:09.766805049 (529h39m9.637888084s) [upload=464531]: Done uploading segments... Dec 27 17:08:09.766829155 (529h39m9.6379122s) [upload=464531]: Upload finished. err: <nil> Duration: 30.002384339s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................X..............X..................] duration: 3.564099931s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:08:13 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.564128676s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 19 out of 60 is failed (threshold=3) [..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X...] duration: 5.842571532s trace: https://jaeger.dev.storj.io/trace/24fac740e7c16785 FAIL (2025-12-27 17:07:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 17:05:27.293611525 (529h36m27.164694571s) [upload=464490]: Starting upload Dec 27 17:05:27.293666620 (529h36m27.164749655s) [upload=464490]: Done starting upload Dec 27 17:05:27.293843747 (529h36m27.164926782s) [upload=464490]: Uploading segments... Dec 27 17:05:27.293950671 (529h36m27.165033716s) [upload=464490]: Got next segment. Inline: false Dec 27 17:05:27.294035483 (529h36m27.165118518s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:05:27.294041744 (529h36m27.165124779s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:05:27.539977809 (529h36m27.411060854s) [seg_pos={0 0}, upload=464490]: Begin upload segment... Dec 27 17:05:27.540046590 (529h36m27.411129625s) [seg_pos={0 0}, upload=464490]: Done begin upload segment. Dec 27 17:05:27.540082890 (529h36m27.411165935s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540097768 (529h36m27.411180803s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540603991 (529h36m27.411687027s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541012289 (529h36m27.412095334s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541098564 (529h36m27.412181599s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541477315 (529h36m27.41256036s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541889089 (529h36m27.412972124s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542219598 (529h36m27.413302634s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542601906 (529h36m27.413684952s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542759306 (529h36m27.413842341s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543137366 (529h36m27.414220411s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543317028 (529h36m27.414400063s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543501950 (529h36m27.414584996s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543983107 (529h36m27.415066152s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544407415 (529h36m27.41549045s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544809399 (529h36m27.415892445s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:29.217668143 (529h36m29.088751178s) [upload=464490]: Next returned nil segment Dec 27 17:05:29.217685094 (529h36m29.088768139s) [upload=464490]: Waiting for error group managing segments... Dec 27 17:05:29.550831513 (529h36m29.421914548s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.551352836 (529h36m29.422435871s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.552347419 (529h36m29.423430454s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.553282931 (529h36m29.424365976s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.570896190 (529h36m29.441979235s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.575387261 (529h36m29.446470306s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.577656302 (529h36m29.448739347s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.580787644 (529h36m29.45187068s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.592149670 (529h36m29.463232715s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.782557799 (529h36m29.653640845s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.785166557 (529h36m29.656249592s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.819574727 (529h36m29.690657772s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899277835 (529h36m29.77036088s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899303163 (529h36m29.770386208s) [seg_pos={0 0}, upload=464490]: Segment reached optimal threshold of 13 pieces. Dec 27 17:05:29.899583166 (529h36m29.770666211s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899593816 (529h36m29.770676861s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899763550 (529h36m29.770846595s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection Dec 27 17:05:29.899781204 (529h36m29.770864249s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899899348 (529h36m29.770982393s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899908806 (529h36m29.770991861s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899981795 (529h36m29.77106484s) [seg_pos={0 0}, upload=464490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:05:29.900124537 (529h36m29.771207582s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:05:29.900130659 (529h36m29.771213694s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:05:30.187199958 (529h36m30.058282993s) [upload=464490]: Done uploading segments... Dec 27 17:05:30.187221219 (529h36m30.058304264s) [upload=464490]: Upload finished. err: <nil> Duration: 5.842603444s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................X..............X.................] duration: 3.485652925s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:07:10 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.485684204s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X..] duration: 6.09241545s trace: https://jaeger.dev.storj.io/trace/24fac740e7c16785 FAIL (2025-12-27 17:07:03 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 17:05:27.293611525 (529h36m27.164694571s) [upload=464490]: Starting upload Dec 27 17:05:27.293666620 (529h36m27.164749655s) [upload=464490]: Done starting upload Dec 27 17:05:27.293843747 (529h36m27.164926782s) [upload=464490]: Uploading segments... Dec 27 17:05:27.293950671 (529h36m27.165033716s) [upload=464490]: Got next segment. Inline: false Dec 27 17:05:27.294035483 (529h36m27.165118518s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:05:27.294041744 (529h36m27.165124779s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:05:27.539977809 (529h36m27.411060854s) [seg_pos={0 0}, upload=464490]: Begin upload segment... Dec 27 17:05:27.540046590 (529h36m27.411129625s) [seg_pos={0 0}, upload=464490]: Done begin upload segment. Dec 27 17:05:27.540082890 (529h36m27.411165935s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540097768 (529h36m27.411180803s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540603991 (529h36m27.411687027s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541012289 (529h36m27.412095334s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541098564 (529h36m27.412181599s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541477315 (529h36m27.41256036s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541889089 (529h36m27.412972124s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542219598 (529h36m27.413302634s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542601906 (529h36m27.413684952s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542759306 (529h36m27.413842341s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543137366 (529h36m27.414220411s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543317028 (529h36m27.414400063s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543501950 (529h36m27.414584996s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543983107 (529h36m27.415066152s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544407415 (529h36m27.41549045s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544809399 (529h36m27.415892445s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:29.217668143 (529h36m29.088751178s) [upload=464490]: Next returned nil segment Dec 27 17:05:29.217685094 (529h36m29.088768139s) [upload=464490]: Waiting for error group managing segments... Dec 27 17:05:29.550831513 (529h36m29.421914548s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.551352836 (529h36m29.422435871s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.552347419 (529h36m29.423430454s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.553282931 (529h36m29.424365976s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.570896190 (529h36m29.441979235s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.575387261 (529h36m29.446470306s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.577656302 (529h36m29.448739347s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.580787644 (529h36m29.45187068s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.592149670 (529h36m29.463232715s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.782557799 (529h36m29.653640845s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.785166557 (529h36m29.656249592s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.819574727 (529h36m29.690657772s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899277835 (529h36m29.77036088s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899303163 (529h36m29.770386208s) [seg_pos={0 0}, upload=464490]: Segment reached optimal threshold of 13 pieces. Dec 27 17:05:29.899583166 (529h36m29.770666211s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899593816 (529h36m29.770676861s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899763550 (529h36m29.770846595s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection Dec 27 17:05:29.899781204 (529h36m29.770864249s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899899348 (529h36m29.770982393s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899908806 (529h36m29.770991861s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899981795 (529h36m29.77106484s) [seg_pos={0 0}, upload=464490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:05:29.900124537 (529h36m29.771207582s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:05:29.900130659 (529h36m29.771213694s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:05:30.187199958 (529h36m30.058282993s) [upload=464490]: Done uploading segments... Dec 27 17:05:30.187221219 (529h36m30.058304264s) [upload=464490]: Upload finished. err: <nil> Duration: 6.092445778s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................X..............X................] duration: 3.443996824s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:06:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.444052882s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............................X..............X...............] duration: 3.600670487s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:06:32 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.600689053s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 20 out of 60 is failed (threshold=3) [.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X.] duration: 6.082584679s trace: https://jaeger.dev.storj.io/trace/24fac740e7c16785 FAIL (2025-12-27 17:06:23 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 17:05:27.293611525 (529h36m27.164694571s) [upload=464490]: Starting upload Dec 27 17:05:27.293666620 (529h36m27.164749655s) [upload=464490]: Done starting upload Dec 27 17:05:27.293843747 (529h36m27.164926782s) [upload=464490]: Uploading segments... Dec 27 17:05:27.293950671 (529h36m27.165033716s) [upload=464490]: Got next segment. Inline: false Dec 27 17:05:27.294035483 (529h36m27.165118518s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:05:27.294041744 (529h36m27.165124779s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:05:27.539977809 (529h36m27.411060854s) [seg_pos={0 0}, upload=464490]: Begin upload segment... Dec 27 17:05:27.540046590 (529h36m27.411129625s) [seg_pos={0 0}, upload=464490]: Done begin upload segment. Dec 27 17:05:27.540082890 (529h36m27.411165935s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540097768 (529h36m27.411180803s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540603991 (529h36m27.411687027s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541012289 (529h36m27.412095334s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541098564 (529h36m27.412181599s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541477315 (529h36m27.41256036s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541889089 (529h36m27.412972124s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542219598 (529h36m27.413302634s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542601906 (529h36m27.413684952s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542759306 (529h36m27.413842341s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543137366 (529h36m27.414220411s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543317028 (529h36m27.414400063s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543501950 (529h36m27.414584996s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543983107 (529h36m27.415066152s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544407415 (529h36m27.41549045s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544809399 (529h36m27.415892445s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:29.217668143 (529h36m29.088751178s) [upload=464490]: Next returned nil segment Dec 27 17:05:29.217685094 (529h36m29.088768139s) [upload=464490]: Waiting for error group managing segments... Dec 27 17:05:29.550831513 (529h36m29.421914548s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.551352836 (529h36m29.422435871s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.552347419 (529h36m29.423430454s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.553282931 (529h36m29.424365976s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.570896190 (529h36m29.441979235s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.575387261 (529h36m29.446470306s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.577656302 (529h36m29.448739347s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.580787644 (529h36m29.45187068s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.592149670 (529h36m29.463232715s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.782557799 (529h36m29.653640845s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.785166557 (529h36m29.656249592s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.819574727 (529h36m29.690657772s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899277835 (529h36m29.77036088s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899303163 (529h36m29.770386208s) [seg_pos={0 0}, upload=464490]: Segment reached optimal threshold of 13 pieces. Dec 27 17:05:29.899583166 (529h36m29.770666211s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899593816 (529h36m29.770676861s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899763550 (529h36m29.770846595s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection Dec 27 17:05:29.899781204 (529h36m29.770864249s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899899348 (529h36m29.770982393s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899908806 (529h36m29.770991861s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899981795 (529h36m29.77106484s) [seg_pos={0 0}, upload=464490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:05:29.900124537 (529h36m29.771207582s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:05:29.900130659 (529h36m29.771213694s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:05:30.187199958 (529h36m30.058282993s) [upload=464490]: Done uploading segments... Dec 27 17:05:30.187221219 (529h36m30.058304264s) [upload=464490]: Upload finished. err: <nil> Duration: 6.08261127s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [X.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...X] duration: 30.002972791s trace: https://jaeger.dev.storj.io/trace/24fac740e7c16785 FAIL (2025-12-27 17:05:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== 27 17:05:27.293611525 (529h36m27.164694571s) [upload=464490]: Starting upload Dec 27 17:05:27.293666620 (529h36m27.164749655s) [upload=464490]: Done starting upload Dec 27 17:05:27.293843747 (529h36m27.164926782s) [upload=464490]: Uploading segments... Dec 27 17:05:27.293950671 (529h36m27.165033716s) [upload=464490]: Got next segment. Inline: false Dec 27 17:05:27.294035483 (529h36m27.165118518s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:05:27.294041744 (529h36m27.165124779s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:05:27.539977809 (529h36m27.411060854s) [seg_pos={0 0}, upload=464490]: Begin upload segment... Dec 27 17:05:27.540046590 (529h36m27.411129625s) [seg_pos={0 0}, upload=464490]: Done begin upload segment. Dec 27 17:05:27.540082890 (529h36m27.411165935s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540097768 (529h36m27.411180803s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.540603991 (529h36m27.411687027s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541012289 (529h36m27.412095334s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541098564 (529h36m27.412181599s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541477315 (529h36m27.41256036s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.541889089 (529h36m27.412972124s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542219598 (529h36m27.413302634s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542601906 (529h36m27.413684952s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.542759306 (529h36m27.413842341s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543137366 (529h36m27.414220411s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543317028 (529h36m27.414400063s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543501950 (529h36m27.414584996s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.543983107 (529h36m27.415066152s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544407415 (529h36m27.41549045s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:27.544809399 (529h36m27.415892445s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Uploading piece... Dec 27 17:05:29.217668143 (529h36m29.088751178s) [upload=464490]: Next returned nil segment Dec 27 17:05:29.217685094 (529h36m29.088768139s) [upload=464490]: Waiting for error group managing segments... Dec 27 17:05:29.550831513 (529h36m29.421914548s) [noise=true , address=145.79.227.41:4085 , piece_id=L2SLUSEV7VMI6NGAYRR3LNU47KBLUQEAAAPIOSYJ7DC2UW56I6UQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.551352836 (529h36m29.422435871s) [noise=true , address=145.79.227.11:4732 , piece_id=2XJGFV24YRRPDJ2P7PMMM3OI6WAIZTM65HFNL4N63L6U7HRZEMGQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.552347419 (529h36m29.423430454s) [noise=true , address=145.79.227.15:4874 , piece_id=THQSVMU3BMSL5IGMJFBGO3YD3DXSMXX6DWT73GQBDUZZ5WNRLKLQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.553282931 (529h36m29.424365976s) [noise=true , address=145.79.227.42:4122 , piece_id=OK3F3NN3PMP6RB7W4SO7TX65ZHZZAXLU25O4ZDIIIVQVJ2CVJVWQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.570896190 (529h36m29.441979235s) [noise=true , address=145.79.227.38:4978 , piece_id=UQZ3NNJT5TL4KKVZEP3PURAYRHN6JCC5MITPV3LKT3TMIKEYLVRQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.575387261 (529h36m29.446470306s) [noise=true , address=145.79.227.12:4767 , piece_id=DFOYZDKP6WRAAPONLA7NQWWMRKS7HKHSP5E6XYKJPCR4UURLCXUA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.577656302 (529h36m29.448739347s) [noise=true , address=145.79.227.25:4618 , piece_id=CALKODXM6QD5RHSLJZM2BMPGKRTO2LMXKAM7TNSDYEZYI3WQ7A7Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.580787644 (529h36m29.45187068s) [noise=true , address=145.79.227.47:4267 , piece_id=4IB4BPJC5OZEUHZXFNBDCWGLJYCK2CDP5IGK2LUBH2TEWDNE746Q, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.592149670 (529h36m29.463232715s) [noise=true , address=145.79.227.49:4337 , piece_id=AHICT2IFAU6KOJB7KSKDT77I4E224KUR2IY6AVIETZ6WUWXUE2FQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.782557799 (529h36m29.653640845s) [noise=true , address=45.91.116.13:4155 , piece_id=YFEIXZURABLZRVDV6EYQ73Q2WGLDJTJNDXKMQKJTAZGJ7SE3EOPA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.785166557 (529h36m29.656249592s) [noise=true , address=45.91.116.5:4298 , piece_id=DT7RRIUILMTJ5VQJTXB6XRTPQYWL4CTTHVWH4MJ6ZNE4HILYWZMQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.819574727 (529h36m29.690657772s) [noise=true , address=77.90.31.11:4553 , piece_id=EIMPYPGAPESTWRLCYRMBM5COZE44JDQVZSMOZYNBRUQ4VS4W2LFA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899277835 (529h36m29.77036088s) [noise=true , address=45.91.116.12:4228 , piece_id=BVHFALNX5S7RDW4NDXVR77SSJHVG7JJ3ZRRYMITHV6RCUN52TLXQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: <nil> Dec 27 17:05:29.899303163 (529h36m29.770386208s) [seg_pos={0 0}, upload=464490]: Segment reached optimal threshold of 13 pieces. Dec 27 17:05:29.899583166 (529h36m29.770666211s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899593816 (529h36m29.770676861s) [noise=true , address=45.91.116.10:4083 , piece_id=WCQSPGZMCA32JCN5TLKP4H4M7SA44Z45OPGLGWQ3VQCUPKEEHFHQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899763550 (529h36m29.770846595s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:59996->77.90.31.15:4697: use of closed network connection Dec 27 17:05:29.899781204 (529h36m29.770864249s) [noise=true , address=77.90.31.15:4697 , piece_id=EPRWUNCDWSUYW5BFO67RQ33VV2ZZ4FVET6YPQQMXPPZW674MULXA, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899899348 (529h36m29.770982393s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1288A5aF4EsQG3XrDQg6fqURsqsFKfSRMeLi4kC5aBRYF6uBnFB): protocol: expected piece hash; context canceled; EOF Dec 27 17:05:29.899908806 (529h36m29.770991861s) [noise=true , address=45.91.116.8:4048 , piece_id=6V7463VG4JDMFJSFFNP4DWYJ3OQWRL4SSZJC3PNZOHBU5FI4PBPQ, seg_pos={0 0}, upload=464490]: Long tail cancellation due to optimal threshold Dec 27 17:05:29.899981795 (529h36m29.77106484s) [seg_pos={0 0}, upload=464490]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:05:29.900124537 (529h36m29.771207582s) [upload=464490]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:05:29.900130659 (529h36m29.771213694s) [upload=464490]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:05:30.187199958 (529h36m30.058282993s) [upload=464490]: Done uploading segments... Dec 27 17:05:30.187221219 (529h36m30.058304264s) [upload=464490]: Upload finished. err: <nil> Duration: 30.002997137s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [X.............................X..............X..............] duration: 3.617533766s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:05:48 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.61756087s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X...] duration: 10.182511406s trace: https://jaeger.dev.storj.io/trace/45bfef14be67f70c FAIL (2025-12-27 17:05:21 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA) context deadline exceeded ======LOGS====== Dec 27 17:02:57.839341057 (529h33m57.710424102s) [upload=464453]: Starting upload Dec 27 17:02:57.839432482 (529h33m57.710515527s) [upload=464453]: Done starting upload Dec 27 17:02:57.839489450 (529h33m57.710572495s) [upload=464453]: Uploading segments... Dec 27 17:02:57.839722784 (529h33m57.710805819s) [upload=464453]: Got next segment. Inline: false Dec 27 17:02:57.839771096 (529h33m57.710854141s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:02:57.839776988 (529h33m57.710860033s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:02:58.089116132 (529h33m57.960212263s) [seg_pos={0 0}, upload=464453]: Begin upload segment... Dec 27 17:02:58.089176658 (529h33m57.960259703s) [seg_pos={0 0}, upload=464453]: Done begin upload segment. Dec 27 17:02:58.089215622 (529h33m57.960298667s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089261799 (529h33m57.960344834s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089854598 (529h33m57.960937643s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.090678437 (529h33m57.961761482s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091179001 (529h33m57.962262046s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091564745 (529h33m57.96264778s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.092001256 (529h33m57.963084301s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093338854 (529h33m57.964421899s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093466887 (529h33m57.964549922s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094711687 (529h33m57.965794722s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094834081 (529h33m57.965917126s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.095846138 (529h33m57.966929183s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.096879916 (529h33m57.967962961s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.101781088 (529h33m57.972864124s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.104547907 (529h33m57.975630953s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.106602590 (529h33m57.977685635s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:59.802914459 (529h33m59.673997504s) [upload=464453]: Next returned nil segment Dec 27 17:02:59.802950037 (529h33m59.674033082s) [upload=464453]: Waiting for error group managing segments... Dec 27 17:03:00.091920456 (529h33m59.963003511s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.092178367 (529h33m59.963261402s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.093627737 (529h33m59.964710782s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.103986402 (529h33m59.975069437s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.110609594 (529h33m59.981692639s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117750892 (529h33m59.988833937s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117791819 (529h33m59.988874865s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.119084801 (529h33m59.990167846s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.146634501 (529h34m0.017717546s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.341614196 (529h34m0.212697231s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.399095410 (529h34m0.270178455s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.431765609 (529h34m0.302848654s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476268100 (529h34m0.347351145s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476292257 (529h34m0.347375292s) [seg_pos={0 0}, upload=464453]: Segment reached optimal threshold of 13 pieces. Dec 27 17:03:00.476572250 (529h34m0.347655305s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection Dec 27 17:03:00.476581447 (529h34m0.347664492s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476717506 (529h34m0.347800541s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476740028 (529h34m0.347823063s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476890806 (529h34m0.347973851s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476899893 (529h34m0.347982938s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476922306 (529h34m0.348005351s) [seg_pos={0 0}, upload=464453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:03:00.477010995 (529h34m0.34809403s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:03:00.477019000 (529h34m0.348102045s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:03:00.767804266 (529h34m0.638887311s) [upload=464453]: Done uploading segments... Dec 27 17:03:00.767827058 (529h34m0.638910093s) [upload=464453]: Upload finished. err: <nil> Duration: 10.182570199s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.X.............................X..............X.............] duration: 3.509048612s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:05:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.509082228s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X..] duration: 5.994379s trace: https://jaeger.dev.storj.io/trace/45bfef14be67f70c FAIL (2025-12-27 17:04:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA) context deadline exceeded ======LOGS====== Dec 27 17:02:57.839341057 (529h33m57.710424102s) [upload=464453]: Starting upload Dec 27 17:02:57.839432482 (529h33m57.710515527s) [upload=464453]: Done starting upload Dec 27 17:02:57.839489450 (529h33m57.710572495s) [upload=464453]: Uploading segments... Dec 27 17:02:57.839722784 (529h33m57.710805819s) [upload=464453]: Got next segment. Inline: false Dec 27 17:02:57.839771096 (529h33m57.710854141s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:02:57.839776988 (529h33m57.710860033s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:02:58.089116132 (529h33m57.960212263s) [seg_pos={0 0}, upload=464453]: Begin upload segment... Dec 27 17:02:58.089176658 (529h33m57.960259703s) [seg_pos={0 0}, upload=464453]: Done begin upload segment. Dec 27 17:02:58.089215622 (529h33m57.960298667s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089261799 (529h33m57.960344834s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089854598 (529h33m57.960937643s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.090678437 (529h33m57.961761482s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091179001 (529h33m57.962262046s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091564745 (529h33m57.96264778s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.092001256 (529h33m57.963084301s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093338854 (529h33m57.964421899s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093466887 (529h33m57.964549922s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094711687 (529h33m57.965794722s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094834081 (529h33m57.965917126s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.095846138 (529h33m57.966929183s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.096879916 (529h33m57.967962961s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.101781088 (529h33m57.972864124s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.104547907 (529h33m57.975630953s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.106602590 (529h33m57.977685635s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:59.802914459 (529h33m59.673997504s) [upload=464453]: Next returned nil segment Dec 27 17:02:59.802950037 (529h33m59.674033082s) [upload=464453]: Waiting for error group managing segments... Dec 27 17:03:00.091920456 (529h33m59.963003511s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.092178367 (529h33m59.963261402s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.093627737 (529h33m59.964710782s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.103986402 (529h33m59.975069437s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.110609594 (529h33m59.981692639s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117750892 (529h33m59.988833937s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117791819 (529h33m59.988874865s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.119084801 (529h33m59.990167846s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.146634501 (529h34m0.017717546s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.341614196 (529h34m0.212697231s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.399095410 (529h34m0.270178455s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.431765609 (529h34m0.302848654s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476268100 (529h34m0.347351145s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476292257 (529h34m0.347375292s) [seg_pos={0 0}, upload=464453]: Segment reached optimal threshold of 13 pieces. Dec 27 17:03:00.476572250 (529h34m0.347655305s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection Dec 27 17:03:00.476581447 (529h34m0.347664492s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476717506 (529h34m0.347800541s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476740028 (529h34m0.347823063s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476890806 (529h34m0.347973851s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476899893 (529h34m0.347982938s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476922306 (529h34m0.348005351s) [seg_pos={0 0}, upload=464453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:03:00.477010995 (529h34m0.34809403s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:03:00.477019000 (529h34m0.348102045s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:03:00.767804266 (529h34m0.638887311s) [upload=464453]: Done uploading segments... Dec 27 17:03:00.767827058 (529h34m0.638910093s) [upload=464453]: Upload finished. err: <nil> Duration: 5.994410451s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..X.............................X..............X............] duration: 3.566346088s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:03:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.566372077s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X.] duration: 5.982859386s trace: https://jaeger.dev.storj.io/trace/45bfef14be67f70c FAIL (2025-12-27 17:03:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA) context deadline exceeded ======LOGS====== Dec 27 17:02:57.839341057 (529h33m57.710424102s) [upload=464453]: Starting upload Dec 27 17:02:57.839432482 (529h33m57.710515527s) [upload=464453]: Done starting upload Dec 27 17:02:57.839489450 (529h33m57.710572495s) [upload=464453]: Uploading segments... Dec 27 17:02:57.839722784 (529h33m57.710805819s) [upload=464453]: Got next segment. Inline: false Dec 27 17:02:57.839771096 (529h33m57.710854141s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:02:57.839776988 (529h33m57.710860033s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:02:58.089116132 (529h33m57.960212263s) [seg_pos={0 0}, upload=464453]: Begin upload segment... Dec 27 17:02:58.089176658 (529h33m57.960259703s) [seg_pos={0 0}, upload=464453]: Done begin upload segment. Dec 27 17:02:58.089215622 (529h33m57.960298667s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089261799 (529h33m57.960344834s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089854598 (529h33m57.960937643s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.090678437 (529h33m57.961761482s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091179001 (529h33m57.962262046s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091564745 (529h33m57.96264778s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.092001256 (529h33m57.963084301s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093338854 (529h33m57.964421899s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093466887 (529h33m57.964549922s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094711687 (529h33m57.965794722s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094834081 (529h33m57.965917126s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.095846138 (529h33m57.966929183s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.096879916 (529h33m57.967962961s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.101781088 (529h33m57.972864124s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.104547907 (529h33m57.975630953s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.106602590 (529h33m57.977685635s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:59.802914459 (529h33m59.673997504s) [upload=464453]: Next returned nil segment Dec 27 17:02:59.802950037 (529h33m59.674033082s) [upload=464453]: Waiting for error group managing segments... Dec 27 17:03:00.091920456 (529h33m59.963003511s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.092178367 (529h33m59.963261402s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.093627737 (529h33m59.964710782s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.103986402 (529h33m59.975069437s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.110609594 (529h33m59.981692639s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117750892 (529h33m59.988833937s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117791819 (529h33m59.988874865s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.119084801 (529h33m59.990167846s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.146634501 (529h34m0.017717546s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.341614196 (529h34m0.212697231s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.399095410 (529h34m0.270178455s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.431765609 (529h34m0.302848654s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476268100 (529h34m0.347351145s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476292257 (529h34m0.347375292s) [seg_pos={0 0}, upload=464453]: Segment reached optimal threshold of 13 pieces. Dec 27 17:03:00.476572250 (529h34m0.347655305s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection Dec 27 17:03:00.476581447 (529h34m0.347664492s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476717506 (529h34m0.347800541s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476740028 (529h34m0.347823063s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476890806 (529h34m0.347973851s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476899893 (529h34m0.347982938s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476922306 (529h34m0.348005351s) [seg_pos={0 0}, upload=464453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:03:00.477010995 (529h34m0.34809403s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:03:00.477019000 (529h34m0.348102045s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:03:00.767804266 (529h34m0.638887311s) [upload=464453]: Done uploading segments... Dec 27 17:03:00.767827058 (529h34m0.638910093s) [upload=464453]: Upload finished. err: <nil> Duration: 5.982900945s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [.X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...X] duration: 30.00264052s trace: https://jaeger.dev.storj.io/trace/45bfef14be67f70c FAIL (2025-12-27 17:03:27 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1fSFcAat264kAGFqWt6Kkc14SSpUnXACWby84NqugHqheWcu2b, Piece ID: W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA) context deadline exceeded; piecestore: (Node ID: 1EZ6d8B2pdLZntgDTzJQJzx7o7A2qYK63NQrtN5ke91nmn4idj, Piece ID: IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA) context deadline exceeded ======LOGS====== Dec 27 17:02:57.839341057 (529h33m57.710424102s) [upload=464453]: Starting upload Dec 27 17:02:57.839432482 (529h33m57.710515527s) [upload=464453]: Done starting upload Dec 27 17:02:57.839489450 (529h33m57.710572495s) [upload=464453]: Uploading segments... Dec 27 17:02:57.839722784 (529h33m57.710805819s) [upload=464453]: Got next segment. Inline: false Dec 27 17:02:57.839771096 (529h33m57.710854141s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:02:57.839776988 (529h33m57.710860033s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:02:58.089116132 (529h33m57.960212263s) [seg_pos={0 0}, upload=464453]: Begin upload segment... Dec 27 17:02:58.089176658 (529h33m57.960259703s) [seg_pos={0 0}, upload=464453]: Done begin upload segment. Dec 27 17:02:58.089215622 (529h33m57.960298667s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089261799 (529h33m57.960344834s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.089854598 (529h33m57.960937643s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.090678437 (529h33m57.961761482s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091179001 (529h33m57.962262046s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.091564745 (529h33m57.96264778s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.092001256 (529h33m57.963084301s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093338854 (529h33m57.964421899s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.093466887 (529h33m57.964549922s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094711687 (529h33m57.965794722s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.094834081 (529h33m57.965917126s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.095846138 (529h33m57.966929183s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.096879916 (529h33m57.967962961s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.101781088 (529h33m57.972864124s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.104547907 (529h33m57.975630953s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:58.106602590 (529h33m57.977685635s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Uploading piece... Dec 27 17:02:59.802914459 (529h33m59.673997504s) [upload=464453]: Next returned nil segment Dec 27 17:02:59.802950037 (529h33m59.674033082s) [upload=464453]: Waiting for error group managing segments... Dec 27 17:03:00.091920456 (529h33m59.963003511s) [noise=true , address=145.79.227.16:4911 , piece_id=GKGQC4MW2DSMH6WKR5O2V36ELFXBPHDL6GZIF4BZGN3NTIYJU6GA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.092178367 (529h33m59.963261402s) [noise=true , address=145.79.227.11:4732 , piece_id=SWHWVJP4OX3SGYDRRZBXZNH3B2GUNZX7FLDBPNNL22BXZQI6VH3Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.093627737 (529h33m59.964710782s) [noise=true , address=145.79.227.48:4303 , piece_id=DMXGRWYGNI6VM3IEAINZPECR6NAADJBMWLH2G2YKBGT7RVWF44RA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.103986402 (529h33m59.975069437s) [noise=true , address=145.79.227.17:4971 , piece_id=7I3AUXOR3OGSCR2TMMYONHZVJM4Z7HFSUPKSGBGBSR7NAHOBUOQQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.110609594 (529h33m59.981692639s) [noise=true , address=145.79.227.23:4546 , piece_id=JXWKEY4UIAHRGET7NOZ2DP65JVJGV5YPCBWBHAZPZOKMUZW7BPZA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117750892 (529h33m59.988833937s) [noise=true , address=145.79.227.2:4113 , piece_id=VMG5L6SR64O4ZKCN6MO7W7HBDCRGZXWUOGYPOQJ67KFTAM64Z2LQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.117791819 (529h33m59.988874865s) [noise=true , address=145.79.227.10:4403 , piece_id=PWAPE7DB7F2W4IUV446TLBL25TBVVLSCZEWQ5DC7RIDRAT2S7WBA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.119084801 (529h33m59.990167846s) [noise=true , address=145.79.227.28:4726 , piece_id=FLMDW6GMQZ7MOE35ROHRZUU7MRWO664FGPFPNC5QI6CHXIDDSQJA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.146634501 (529h34m0.017717546s) [noise=true , address=145.79.227.15:4874 , piece_id=2BWF3Q4ZBNI7X7A7XC26JY3WDUWMZ3EBS5ONZFDVFWHZGYPF2RDA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.341614196 (529h34m0.212697231s) [noise=true , address=45.91.116.11:4120 , piece_id=74HD6SHGQJEL4EQPBWUJNCJYYHIKDISS5AWDBVGRYTHGQ7ME5GZQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.399095410 (529h34m0.270178455s) [noise=true , address=45.91.116.13:4153 , piece_id=W7RC46IWDOJYMNTJ7MSXWBTK4SW7MGSYZ7RMLYN4OF45ZTSOAKLA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.431765609 (529h34m0.302848654s) [noise=true , address=45.91.116.10:4084 , piece_id=IQK2MN6OVTM3LYYHKP5CWE2IJXOU2NMDA7WQ5IEY2SSPBKEPNIPA, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476268100 (529h34m0.347351145s) [noise=true , address=45.91.116.7:4009 , piece_id=53E6FECTJI3RCCBV3GP2EF4W3TMZNXWO2PMKZB2E5NY7T4Q2NT4A, seg_pos={0 0}, upload=464453]: Done uploading piece. err: <nil> Dec 27 17:03:00.476292257 (529h34m0.347375292s) [seg_pos={0 0}, upload=464453]: Segment reached optimal threshold of 13 pieces. Dec 27 17:03:00.476572250 (529h34m0.347655305s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FZuwK69Hju5JKoujYS98XohYQgcfm3uea1Wk9bcsMzPW7hXLs): protocol: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:43730->45.91.116.14:4192: use of closed network connection Dec 27 17:03:00.476581447 (529h34m0.347664492s) [noise=true , address=45.91.116.14:4192 , piece_id=PCF3KS5AFI47NRS4ITKTRAZ57J7BD5XZRUXTZRHMFDNRK2XVZTLQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476717506 (529h34m0.347800541s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476740028 (529h34m0.347823063s) [noise=true , address=45.91.116.5:4298 , piece_id=PSX5EES3KI2F46QOUWQCEKPU6SZNSHZKIGRSP2B7LEQ46EFSKPMQ, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476890806 (529h34m0.347973851s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp): protocol: expected piece hash; context canceled; EOF Dec 27 17:03:00.476899893 (529h34m0.347982938s) [noise=true , address=77.90.31.10:4520 , piece_id=DCTQTEF2SE4M7EK2SSP34VG3R4PWFQUPJQGZBA3VFCOWO3QWHC4Q, seg_pos={0 0}, upload=464453]: Long tail cancellation due to optimal threshold Dec 27 17:03:00.476922306 (529h34m0.348005351s) [seg_pos={0 0}, upload=464453]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:03:00.477010995 (529h34m0.34809403s) [upload=464453]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:03:00.477019000 (529h34m0.348102045s) [upload=464453]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:03:00.767804266 (529h34m0.638887311s) [upload=464453]: Done uploading segments... Dec 27 17:03:00.767827058 (529h34m0.638910093s) [upload=464453]: Upload finished. err: <nil> Duration: 30.002663374s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...X.............................X..............X...........] duration: 3.541807019s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:02:43 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.541830895s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 21 out of 60 is failed (threshold=3) [..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X...] duration: 6.784148776s trace: https://jaeger.dev.storj.io/trace/18bea3a0bf0cb7c4 FAIL (2025-12-27 17:02:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ) context deadline exceeded ======LOGS====== Inline: false Dec 27 17:00:39.847438804 (529h31m39.718521849s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:00:39.847445256 (529h31m39.718528291s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:00:40.088296589 (529h31m39.959379624s) [seg_pos={0 0}, upload=464417]: Begin upload segment... Dec 27 17:00:40.088362164 (529h31m39.959445199s) [seg_pos={0 0}, upload=464417]: Done begin upload segment. Dec 27 17:00:40.088377394 (529h31m39.959460429s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088399194 (529h31m39.95948223s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088548069 (529h31m39.959631104s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088840906 (529h31m39.959923941s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089201903 (529h31m39.960285008s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089525780 (529h31m39.960608815s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089922666 (529h31m39.961005711s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090418680 (529h31m39.961501715s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090801188 (529h31m39.961884223s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091221288 (529h31m39.962304323s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091671445 (529h31m39.96275449s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.092771380 (529h31m39.963854415s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094071375 (529h31m39.96515441s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094429308 (529h31m39.965512353s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.095687453 (529h31m39.966770498s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.096853243 (529h31m39.967936278s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:41.819769041 (529h31m41.690852086s) [upload=464417]: Next returned nil segment Dec 27 17:00:41.819782446 (529h31m41.690865491s) [upload=464417]: Waiting for error group managing segments... Dec 27 17:00:42.097915630 (529h31m41.968998685s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.109225655 (529h31m41.9803087s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.115184392 (529h31m41.986267437s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123450492 (529h31m41.994533537s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123585930 (529h31m41.994668965s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123926298 (529h31m41.995009343s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.124947382 (529h31m41.996030427s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.131885263 (529h31m42.002968308s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.132437394 (529h31m42.00352043s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.294701945 (529h31m42.16578498s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.342089909 (529h31m42.213172954s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.351074177 (529h31m42.222157222s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467023354 (529h31m42.338106399s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467048672 (529h31m42.338131707s) [seg_pos={0 0}, upload=464417]: Segment reached optimal threshold of 13 pieces. Dec 27 17:00:42.467302566 (529h31m42.338385601s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection Dec 27 17:00:42.467312004 (529h31m42.338395039s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467324587 (529h31m42.338407642s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection Dec 27 17:00:42.467336120 (529h31m42.338419155s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467610452 (529h31m42.338693518s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60032->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 17:00:42.467619740 (529h31m42.338702805s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467640849 (529h31m42.338723894s) [seg_pos={0 0}, upload=464417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:00:42.467737574 (529h31m42.338820629s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:00:42.467746360 (529h31m42.338829395s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:00:42.750984215 (529h31m42.62206726s) [upload=464417]: Done uploading segments... Dec 27 17:00:42.751009152 (529h31m42.622092187s) [upload=464417]: Upload finished. err: <nil> Duration: 6.784176679s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 22 out of 60 is failed (threshold=3) [X..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X..] duration: 5.818009871s trace: https://jaeger.dev.storj.io/trace/18bea3a0bf0cb7c4 FAIL (2025-12-27 17:02:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ) context deadline exceeded ======LOGS====== Inline: false Dec 27 17:00:39.847438804 (529h31m39.718521849s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:00:39.847445256 (529h31m39.718528291s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:00:40.088296589 (529h31m39.959379624s) [seg_pos={0 0}, upload=464417]: Begin upload segment... Dec 27 17:00:40.088362164 (529h31m39.959445199s) [seg_pos={0 0}, upload=464417]: Done begin upload segment. Dec 27 17:00:40.088377394 (529h31m39.959460429s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088399194 (529h31m39.95948223s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088548069 (529h31m39.959631104s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088840906 (529h31m39.959923941s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089201903 (529h31m39.960285008s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089525780 (529h31m39.960608815s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089922666 (529h31m39.961005711s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090418680 (529h31m39.961501715s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090801188 (529h31m39.961884223s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091221288 (529h31m39.962304323s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091671445 (529h31m39.96275449s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.092771380 (529h31m39.963854415s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094071375 (529h31m39.96515441s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094429308 (529h31m39.965512353s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.095687453 (529h31m39.966770498s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.096853243 (529h31m39.967936278s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:41.819769041 (529h31m41.690852086s) [upload=464417]: Next returned nil segment Dec 27 17:00:41.819782446 (529h31m41.690865491s) [upload=464417]: Waiting for error group managing segments... Dec 27 17:00:42.097915630 (529h31m41.968998685s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.109225655 (529h31m41.9803087s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.115184392 (529h31m41.986267437s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123450492 (529h31m41.994533537s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123585930 (529h31m41.994668965s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123926298 (529h31m41.995009343s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.124947382 (529h31m41.996030427s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.131885263 (529h31m42.002968308s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.132437394 (529h31m42.00352043s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.294701945 (529h31m42.16578498s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.342089909 (529h31m42.213172954s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.351074177 (529h31m42.222157222s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467023354 (529h31m42.338106399s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467048672 (529h31m42.338131707s) [seg_pos={0 0}, upload=464417]: Segment reached optimal threshold of 13 pieces. Dec 27 17:00:42.467302566 (529h31m42.338385601s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection Dec 27 17:00:42.467312004 (529h31m42.338395039s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467324587 (529h31m42.338407642s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection Dec 27 17:00:42.467336120 (529h31m42.338419155s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467610452 (529h31m42.338693518s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60032->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 17:00:42.467619740 (529h31m42.338702805s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467640849 (529h31m42.338723894s) [seg_pos={0 0}, upload=464417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:00:42.467737574 (529h31m42.338820629s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:00:42.467746360 (529h31m42.338829395s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:00:42.750984215 (529h31m42.62206726s) [upload=464417]: Done uploading segments... Dec 27 17:00:42.751009152 (529h31m42.622092187s) [upload=464417]: Upload finished. err: <nil> Duration: 5.818037924s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [X...X.............................X..............X..........] duration: 4.202045358s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:01:35 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.20207287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [XX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X.] duration: 7.857165365s trace: https://jaeger.dev.storj.io/trace/18bea3a0bf0cb7c4 FAIL (2025-12-27 17:01:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ) context deadline exceeded ======LOGS====== Inline: false Dec 27 17:00:39.847438804 (529h31m39.718521849s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:00:39.847445256 (529h31m39.718528291s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:00:40.088296589 (529h31m39.959379624s) [seg_pos={0 0}, upload=464417]: Begin upload segment... Dec 27 17:00:40.088362164 (529h31m39.959445199s) [seg_pos={0 0}, upload=464417]: Done begin upload segment. Dec 27 17:00:40.088377394 (529h31m39.959460429s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088399194 (529h31m39.95948223s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088548069 (529h31m39.959631104s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088840906 (529h31m39.959923941s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089201903 (529h31m39.960285008s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089525780 (529h31m39.960608815s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089922666 (529h31m39.961005711s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090418680 (529h31m39.961501715s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090801188 (529h31m39.961884223s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091221288 (529h31m39.962304323s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091671445 (529h31m39.96275449s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.092771380 (529h31m39.963854415s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094071375 (529h31m39.96515441s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094429308 (529h31m39.965512353s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.095687453 (529h31m39.966770498s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.096853243 (529h31m39.967936278s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:41.819769041 (529h31m41.690852086s) [upload=464417]: Next returned nil segment Dec 27 17:00:41.819782446 (529h31m41.690865491s) [upload=464417]: Waiting for error group managing segments... Dec 27 17:00:42.097915630 (529h31m41.968998685s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.109225655 (529h31m41.9803087s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.115184392 (529h31m41.986267437s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123450492 (529h31m41.994533537s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123585930 (529h31m41.994668965s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123926298 (529h31m41.995009343s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.124947382 (529h31m41.996030427s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.131885263 (529h31m42.002968308s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.132437394 (529h31m42.00352043s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.294701945 (529h31m42.16578498s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.342089909 (529h31m42.213172954s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.351074177 (529h31m42.222157222s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467023354 (529h31m42.338106399s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467048672 (529h31m42.338131707s) [seg_pos={0 0}, upload=464417]: Segment reached optimal threshold of 13 pieces. Dec 27 17:00:42.467302566 (529h31m42.338385601s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection Dec 27 17:00:42.467312004 (529h31m42.338395039s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467324587 (529h31m42.338407642s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection Dec 27 17:00:42.467336120 (529h31m42.338419155s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467610452 (529h31m42.338693518s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60032->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 17:00:42.467619740 (529h31m42.338702805s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467640849 (529h31m42.338723894s) [seg_pos={0 0}, upload=464417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:00:42.467737574 (529h31m42.338820629s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:00:42.467746360 (529h31m42.338829395s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:00:42.750984215 (529h31m42.62206726s) [upload=464417]: Done uploading segments... Dec 27 17:00:42.751009152 (529h31m42.622092187s) [upload=464417]: Upload finished. err: <nil> Duration: 7.857188969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.X] duration: 30.003680835s trace: https://jaeger.dev.storj.io/trace/18bea3a0bf0cb7c4 FAIL (2025-12-27 17:01:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 123WHuy89PAwkQ6AhXenBsQM6HdptYmwJWWqNkGC7P1boNd4K8B, Piece ID: YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ) context deadline exceeded; piecestore: (Node ID: 1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG, Piece ID: QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ) context deadline exceeded ======LOGS====== Inline: false Dec 27 17:00:39.847438804 (529h31m39.718521849s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 17:00:39.847445256 (529h31m39.718528291s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 17:00:40.088296589 (529h31m39.959379624s) [seg_pos={0 0}, upload=464417]: Begin upload segment... Dec 27 17:00:40.088362164 (529h31m39.959445199s) [seg_pos={0 0}, upload=464417]: Done begin upload segment. Dec 27 17:00:40.088377394 (529h31m39.959460429s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088399194 (529h31m39.95948223s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088548069 (529h31m39.959631104s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.088840906 (529h31m39.959923941s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089201903 (529h31m39.960285008s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089525780 (529h31m39.960608815s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.089922666 (529h31m39.961005711s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090418680 (529h31m39.961501715s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.090801188 (529h31m39.961884223s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091221288 (529h31m39.962304323s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.091671445 (529h31m39.96275449s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.092771380 (529h31m39.963854415s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094071375 (529h31m39.96515441s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.094429308 (529h31m39.965512353s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.095687453 (529h31m39.966770498s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:40.096853243 (529h31m39.967936278s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Uploading piece... Dec 27 17:00:41.819769041 (529h31m41.690852086s) [upload=464417]: Next returned nil segment Dec 27 17:00:41.819782446 (529h31m41.690865491s) [upload=464417]: Waiting for error group managing segments... Dec 27 17:00:42.097915630 (529h31m41.968998685s) [noise=true , address=145.79.227.12:4767 , piece_id=CBNTETUX5OOBFGHH2BD3TGYGWGEUEUA4MPVPAV66MWK5HM5ST7SQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.109225655 (529h31m41.9803087s) [noise=true , address=145.79.227.11:4732 , piece_id=QHYJWDYGX3JY5ZBTCKTEGFJ3K55JAQYMPMXS4RVJCZZZLWR3CTUA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.115184392 (529h31m41.986267437s) [noise=true , address=145.79.227.18:4007 , piece_id=IMCK4GKAOBKOOGZWKTZ6WDV32LFMGYJIIFD37DEXGAWYAI37YIYQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123450492 (529h31m41.994533537s) [noise=true , address=145.79.227.2:4114 , piece_id=F5OCWRCRPZJ2KNV3TPONSFBDRSXMIDNPGDYWP4QSXHN42ANNLF7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123585930 (529h31m41.994668965s) [noise=true , address=145.79.227.16:4911 , piece_id=OUYX75Z32IZBG2RPI46MYFSWLDU4VEC3PHTZXHOLAJ4ABDLTLLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.123926298 (529h31m41.995009343s) [noise=true , address=145.79.227.37:4565 , piece_id=TTUBAHMO45D47ZASMPP7LSQ4FIEG6FHXDVAGRQDGCUWU4EJITD7Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.124947382 (529h31m41.996030427s) [noise=true , address=145.79.227.4:4186 , piece_id=Q3MGFS32TMTA53242SQGU5DACNHTRMPU5SVST24I3YNWBB3XUE5A, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.131885263 (529h31m42.002968308s) [noise=true , address=145.79.227.48:4303 , piece_id=QB6I7R27T6SMGVZGAQULB4LQ4JCKLWKVIQAOBSDVEOWPLGYMK24Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.132437394 (529h31m42.00352043s) [noise=true , address=145.79.227.38:4979 , piece_id=B33DMJB3JJNA3UCR7GIURBO4STWIRFGXRQTG7FRGP4TADWHBU45Q, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.294701945 (529h31m42.16578498s) [noise=true , address=77.90.31.17:4773 , piece_id=KQAMNWR4I2YHFYEIKBISVRI45LFA6YJBT3H2LKKQFCXZUCT26TGQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.342089909 (529h31m42.213172954s) [noise=true , address=45.91.116.10:4083 , piece_id=YHB77QYAQITFWG4UFDIA2VEI6APBL4BJIA26CQFYKW54OGM6ZVCQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.351074177 (529h31m42.222157222s) [noise=true , address=45.91.116.13:4155 , piece_id=Y37G4IN2KG4TCGON6R4P5TTPNLM4ED6IP5BMI7AM3FB5VVOPSLOA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467023354 (529h31m42.338106399s) [noise=true , address=45.91.116.14:4190 , piece_id=QSC4NZFALV5W54Y4LDROV4ALEBC6E3V7ASIQWPN7EKR3EEPY7XBQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: <nil> Dec 27 17:00:42.467048672 (529h31m42.338131707s) [seg_pos={0 0}, upload=464417]: Segment reached optimal threshold of 13 pieces. Dec 27 17:00:42.467302566 (529h31m42.338385601s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1vQ1WWxf2CL7wRKRsKsRsWc4nkCbD1h8tRLsA3QYYAft91yUb): protocol: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:49374->77.90.31.21:4880: use of closed network connection Dec 27 17:00:42.467312004 (529h31m42.338395039s) [noise=true , address=77.90.31.21:4880 , piece_id=FSQ6CHZHOV4NCCA42RXK7KAPSJKT7NVGUEYIWJXPYBMKFNYLXWRA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467324587 (529h31m42.338407642s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39160->45.91.116.7:4010: use of closed network connection Dec 27 17:00:42.467336120 (529h31m42.338419155s) [noise=true , address=45.91.116.7:4010 , piece_id=GU74FM2K6UCGL7YEL45INCKHAIVKISS4DDUC373QJZYENNKKJ2EQ, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467610452 (529h31m42.338693518s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks): protocol: read tcp 188.166.225.170:60032->77.90.31.11:4556: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 27 17:00:42.467619740 (529h31m42.338702805s) [noise=true , address=77.90.31.11:4556 , piece_id=DXMVVOLGRFZCFEK4ZRYGQSFDDYV45S2TGU4UESVJDRN6RWIBINZA, seg_pos={0 0}, upload=464417]: Long tail cancellation due to optimal threshold Dec 27 17:00:42.467640849 (529h31m42.338723894s) [seg_pos={0 0}, upload=464417]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 17:00:42.467737574 (529h31m42.338820629s) [upload=464417]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 17:00:42.467746360 (529h31m42.338829395s) [upload=464417]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 17:00:42.750984215 (529h31m42.62206726s) [upload=464417]: Done uploading segments... Dec 27 17:00:42.751009152 (529h31m42.622092187s) [upload=464417]: Upload finished. err: <nil> Duration: 30.00370491s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.X...X.............................X..............X.........] duration: 3.856118373s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:01:06 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.856153209s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [..X...X.............................X..............X........] duration: 5.0014863s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:00:21 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001548018s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [...X...X.............................X..............X.......] duration: 3.442361769s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 17:00:02 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.442390694s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X.] duration: 7.73736013s trace: https://jaeger.dev.storj.io/trace/8184c6de745e846 FAIL (2025-12-27 16:59:32 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A) context deadline exceeded ======LOGS====== 16:58:14.796237598 (529h29m14.667320633s) [upload=464387]: Starting upload Dec 27 16:58:14.796310457 (529h29m14.667393492s) [upload=464387]: Done starting upload Dec 27 16:58:14.796375270 (529h29m14.667458305s) [upload=464387]: Uploading segments... Dec 27 16:58:14.796495579 (529h29m14.667578614s) [upload=464387]: Got next segment. Inline: false Dec 27 16:58:14.796526779 (529h29m14.667609814s) [upload=464387]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:58:14.796530786 (529h29m14.667613821s) [upload=464387]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:58:15.043802495 (529h29m14.91488553s) [seg_pos={0 0}, upload=464387]: Begin upload segment... Dec 27 16:58:15.043848141 (529h29m14.914931176s) [seg_pos={0 0}, upload=464387]: Done begin upload segment. Dec 27 16:58:15.043873971 (529h29m14.914957006s) [noise=true , address=145.79.227.12:4765 , piece_id=CBDYSJUQJG5B74HZVEKD2GPMSXNGZQU6LYVV4Q2JGX2EBNZ2T7GA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.043957479 (529h29m14.915040514s) [noise=true , address=77.90.31.16:4736 , piece_id=LHGEOHR5F5WXQJIX5OD3OB4XBIAVBQTFVVSMZANCRUMJA4MNB67A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.044575046 (529h29m14.915658081s) [noise=true , address=145.79.227.32:4836 , piece_id=IEGE6IQUR5IDJR72O24EZMPKPINL6MBMOLUITKRTSNLY7PNFFJ7Q, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.044874716 (529h29m14.915957751s) [noise=true , address=145.79.227.36:4519 , piece_id=MZHMXL6XOQOS7QUHJJTCQM7GD7B2USH2GIDJVS7LEE4OUF4MYHJA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045148968 (529h29m14.916232003s) [noise=true , address=145.79.227.15:4874 , piece_id=4ZJR57OJQAIQTMID5XDIDSYSALVXDLHSBWOP3YIIBFLIJNJKED2A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045435144 (529h29m14.916518179s) [noise=true , address=145.79.227.26:4655 , piece_id=MUOKF2BZQ4A2M5SAZOI3GYRAYS4XNCTDUMXFPKMH74V5422VAAAA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045752778 (529h29m14.916835813s) [noise=true , address=145.79.227.46:4603 , piece_id=VMI6OW7HK6WQXAKCUSSY2ZKNBBT6ARKVPFB6J6CB6G67TYQ3ZKSQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.046065324 (529h29m14.917148359s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.047571531 (529h29m14.918654566s) [noise=true , address=145.79.227.28:4726 , piece_id=HNRK3B5SXJBZ2RBXNSIF6YOOHEWVQXIGVU7F2BZ7S2CTXXAPAFOA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.048757079 (529h29m14.919840114s) [noise=true , address=77.90.31.19:4844 , piece_id=WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.052149709 (529h29m14.923232744s) [noise=true , address=77.90.31.13:4629 , piece_id=RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.055135825 (529h29m14.92621886s) [noise=true , address=45.91.116.6:4973 , piece_id=TYJQG4A6VWE2D7ANOLEDGRA7ZWM36H5CWEJSNW3CKQ35X74BWVAQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.057145382 (529h29m14.928228417s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.058757241 (529h29m14.929840276s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.060310278 (529h29m14.931393313s) [noise=true , address=77.90.31.18:4808 , piece_id=2ARVM3ZHNDQDRSFE3LWMLAJKDW5X4FQEVSD4DU34YCXXXT3KGLSA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.060511962 (529h29m14.931594998s) [noise=true , address=145.79.227.7:4295 , piece_id=LJPGLGL3DT7RC4JYVUA2WUWQ7HU7IHFGNEFRDPHHXOHLCCV2YFYQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:16.751359066 (529h29m16.622442121s) [upload=464387]: Next returned nil segment Dec 27 16:58:16.751382791 (529h29m16.622465836s) [upload=464387]: Waiting for error group managing segments... Dec 27 16:58:17.034163023 (529h29m16.905246058s) [noise=true , address=145.79.227.46:4603 , piece_id=VMI6OW7HK6WQXAKCUSSY2ZKNBBT6ARKVPFB6J6CB6G67TYQ3ZKSQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.034200835 (529h29m16.90528388s) [noise=true , address=145.79.227.15:4874 , piece_id=4ZJR57OJQAIQTMID5XDIDSYSALVXDLHSBWOP3YIIBFLIJNJKED2A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.049516669 (529h29m16.920599704s) [noise=true , address=145.79.227.26:4655 , piece_id=MUOKF2BZQ4A2M5SAZOI3GYRAYS4XNCTDUMXFPKMH74V5422VAAAA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.050199358 (529h29m16.921282394s) [noise=true , address=145.79.227.32:4836 , piece_id=IEGE6IQUR5IDJR72O24EZMPKPINL6MBMOLUITKRTSNLY7PNFFJ7Q, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.052116750 (529h29m16.923199796s) [noise=true , address=145.79.227.36:4519 , piece_id=MZHMXL6XOQOS7QUHJJTCQM7GD7B2USH2GIDJVS7LEE4OUF4MYHJA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.061440293 (529h29m16.932523328s) [noise=true , address=145.79.227.12:4765 , piece_id=CBDYSJUQJG5B74HZVEKD2GPMSXNGZQU6LYVV4Q2JGX2EBNZ2T7GA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.067222725 (529h29m16.93830576s) [noise=true , address=145.79.227.7:4295 , piece_id=LJPGLGL3DT7RC4JYVUA2WUWQ7HU7IHFGNEFRDPHHXOHLCCV2YFYQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.081116651 (529h29m16.952199696s) [noise=true , address=145.79.227.28:4726 , piece_id=HNRK3B5SXJBZ2RBXNSIF6YOOHEWVQXIGVU7F2BZ7S2CTXXAPAFOA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.228728760 (529h29m17.099811805s) [noise=true , address=77.90.31.13:4629 , piece_id=RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.297702948 (529h29m17.168785983s) [noise=true , address=77.90.31.18:4808 , piece_id=2ARVM3ZHNDQDRSFE3LWMLAJKDW5X4FQEVSD4DU34YCXXXT3KGLSA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.299562089 (529h29m17.170645134s) [noise=true , address=45.91.116.6:4973 , piece_id=TYJQG4A6VWE2D7ANOLEDGRA7ZWM36H5CWEJSNW3CKQ35X74BWVAQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.314160025 (529h29m17.18524306s) [noise=true , address=77.90.31.19:4844 , piece_id=WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.356797384 (529h29m17.22788043s) [noise=true , address=77.90.31.16:4736 , piece_id=LHGEOHR5F5WXQJIX5OD3OB4XBIAVBQTFVVSMZANCRUMJA4MNB67A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.356818955 (529h29m17.22790199s) [seg_pos={0 0}, upload=464387]: Segment reached optimal threshold of 13 pieces. Dec 27 16:58:17.357022833 (529h29m17.228105878s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection Dec 27 16:58:17.357029766 (529h29m17.228112801s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357046699 (529h29m17.228129734s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 16:58:17.357058151 (529h29m17.228141186s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357246279 (529h29m17.228329314s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 16:58:17.357252731 (529h29m17.228335766s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357268271 (529h29m17.228351306s) [seg_pos={0 0}, upload=464387]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:58:17.357334847 (529h29m17.228417883s) [upload=464387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:58:17.357339436 (529h29m17.228422471s) [upload=464387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:58:17.647876318 (529h29m17.518959363s) [upload=464387]: Done uploading segments... Dec 27 16:58:17.647901316 (529h29m17.518984361s) [upload=464387]: Upload finished. err: <nil> Duration: 7.737390086s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [....X...X.............................X..............X......] duration: 3.454117544s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:59:22 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.454133474s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....X] duration: 30.00358864s trace: https://jaeger.dev.storj.io/trace/8184c6de745e846 FAIL (2025-12-27 16:58:44 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1rJArRXLFnyoegSL7XGNequq3wd1c14uzeumaR1HAkd5NdQB1t, Piece ID: WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ) context deadline exceeded; piecestore: (Node ID: 12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, Piece ID: RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A) context deadline exceeded ======LOGS====== 16:58:14.796237598 (529h29m14.667320633s) [upload=464387]: Starting upload Dec 27 16:58:14.796310457 (529h29m14.667393492s) [upload=464387]: Done starting upload Dec 27 16:58:14.796375270 (529h29m14.667458305s) [upload=464387]: Uploading segments... Dec 27 16:58:14.796495579 (529h29m14.667578614s) [upload=464387]: Got next segment. Inline: false Dec 27 16:58:14.796526779 (529h29m14.667609814s) [upload=464387]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:58:14.796530786 (529h29m14.667613821s) [upload=464387]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:58:15.043802495 (529h29m14.91488553s) [seg_pos={0 0}, upload=464387]: Begin upload segment... Dec 27 16:58:15.043848141 (529h29m14.914931176s) [seg_pos={0 0}, upload=464387]: Done begin upload segment. Dec 27 16:58:15.043873971 (529h29m14.914957006s) [noise=true , address=145.79.227.12:4765 , piece_id=CBDYSJUQJG5B74HZVEKD2GPMSXNGZQU6LYVV4Q2JGX2EBNZ2T7GA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.043957479 (529h29m14.915040514s) [noise=true , address=77.90.31.16:4736 , piece_id=LHGEOHR5F5WXQJIX5OD3OB4XBIAVBQTFVVSMZANCRUMJA4MNB67A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.044575046 (529h29m14.915658081s) [noise=true , address=145.79.227.32:4836 , piece_id=IEGE6IQUR5IDJR72O24EZMPKPINL6MBMOLUITKRTSNLY7PNFFJ7Q, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.044874716 (529h29m14.915957751s) [noise=true , address=145.79.227.36:4519 , piece_id=MZHMXL6XOQOS7QUHJJTCQM7GD7B2USH2GIDJVS7LEE4OUF4MYHJA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045148968 (529h29m14.916232003s) [noise=true , address=145.79.227.15:4874 , piece_id=4ZJR57OJQAIQTMID5XDIDSYSALVXDLHSBWOP3YIIBFLIJNJKED2A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045435144 (529h29m14.916518179s) [noise=true , address=145.79.227.26:4655 , piece_id=MUOKF2BZQ4A2M5SAZOI3GYRAYS4XNCTDUMXFPKMH74V5422VAAAA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.045752778 (529h29m14.916835813s) [noise=true , address=145.79.227.46:4603 , piece_id=VMI6OW7HK6WQXAKCUSSY2ZKNBBT6ARKVPFB6J6CB6G67TYQ3ZKSQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.046065324 (529h29m14.917148359s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.047571531 (529h29m14.918654566s) [noise=true , address=145.79.227.28:4726 , piece_id=HNRK3B5SXJBZ2RBXNSIF6YOOHEWVQXIGVU7F2BZ7S2CTXXAPAFOA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.048757079 (529h29m14.919840114s) [noise=true , address=77.90.31.19:4844 , piece_id=WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.052149709 (529h29m14.923232744s) [noise=true , address=77.90.31.13:4629 , piece_id=RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.055135825 (529h29m14.92621886s) [noise=true , address=45.91.116.6:4973 , piece_id=TYJQG4A6VWE2D7ANOLEDGRA7ZWM36H5CWEJSNW3CKQ35X74BWVAQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.057145382 (529h29m14.928228417s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.058757241 (529h29m14.929840276s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.060310278 (529h29m14.931393313s) [noise=true , address=77.90.31.18:4808 , piece_id=2ARVM3ZHNDQDRSFE3LWMLAJKDW5X4FQEVSD4DU34YCXXXT3KGLSA, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:15.060511962 (529h29m14.931594998s) [noise=true , address=145.79.227.7:4295 , piece_id=LJPGLGL3DT7RC4JYVUA2WUWQ7HU7IHFGNEFRDPHHXOHLCCV2YFYQ, seg_pos={0 0}, upload=464387]: Uploading piece... Dec 27 16:58:16.751359066 (529h29m16.622442121s) [upload=464387]: Next returned nil segment Dec 27 16:58:16.751382791 (529h29m16.622465836s) [upload=464387]: Waiting for error group managing segments... Dec 27 16:58:17.034163023 (529h29m16.905246058s) [noise=true , address=145.79.227.46:4603 , piece_id=VMI6OW7HK6WQXAKCUSSY2ZKNBBT6ARKVPFB6J6CB6G67TYQ3ZKSQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.034200835 (529h29m16.90528388s) [noise=true , address=145.79.227.15:4874 , piece_id=4ZJR57OJQAIQTMID5XDIDSYSALVXDLHSBWOP3YIIBFLIJNJKED2A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.049516669 (529h29m16.920599704s) [noise=true , address=145.79.227.26:4655 , piece_id=MUOKF2BZQ4A2M5SAZOI3GYRAYS4XNCTDUMXFPKMH74V5422VAAAA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.050199358 (529h29m16.921282394s) [noise=true , address=145.79.227.32:4836 , piece_id=IEGE6IQUR5IDJR72O24EZMPKPINL6MBMOLUITKRTSNLY7PNFFJ7Q, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.052116750 (529h29m16.923199796s) [noise=true , address=145.79.227.36:4519 , piece_id=MZHMXL6XOQOS7QUHJJTCQM7GD7B2USH2GIDJVS7LEE4OUF4MYHJA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.061440293 (529h29m16.932523328s) [noise=true , address=145.79.227.12:4765 , piece_id=CBDYSJUQJG5B74HZVEKD2GPMSXNGZQU6LYVV4Q2JGX2EBNZ2T7GA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.067222725 (529h29m16.93830576s) [noise=true , address=145.79.227.7:4295 , piece_id=LJPGLGL3DT7RC4JYVUA2WUWQ7HU7IHFGNEFRDPHHXOHLCCV2YFYQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.081116651 (529h29m16.952199696s) [noise=true , address=145.79.227.28:4726 , piece_id=HNRK3B5SXJBZ2RBXNSIF6YOOHEWVQXIGVU7F2BZ7S2CTXXAPAFOA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.228728760 (529h29m17.099811805s) [noise=true , address=77.90.31.13:4629 , piece_id=RFT667GCA3IZCN7K5UJ4ZHKUC5CUDFMBXJVGNS6BK7IM6DXQS77A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.297702948 (529h29m17.168785983s) [noise=true , address=77.90.31.18:4808 , piece_id=2ARVM3ZHNDQDRSFE3LWMLAJKDW5X4FQEVSD4DU34YCXXXT3KGLSA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.299562089 (529h29m17.170645134s) [noise=true , address=45.91.116.6:4973 , piece_id=TYJQG4A6VWE2D7ANOLEDGRA7ZWM36H5CWEJSNW3CKQ35X74BWVAQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.314160025 (529h29m17.18524306s) [noise=true , address=77.90.31.19:4844 , piece_id=WJAK7TV4ZVKPIVYXEFJKQDPLGEBKRX5O44KL7MXU5NMQMCPLR7AQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.356797384 (529h29m17.22788043s) [noise=true , address=77.90.31.16:4736 , piece_id=LHGEOHR5F5WXQJIX5OD3OB4XBIAVBQTFVVSMZANCRUMJA4MNB67A, seg_pos={0 0}, upload=464387]: Done uploading piece. err: <nil> Dec 27 16:58:17.356818955 (529h29m17.22790199s) [seg_pos={0 0}, upload=464387]: Segment reached optimal threshold of 13 pieces. Dec 27 16:58:17.357022833 (529h29m17.228105878s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr): protocol: write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection; write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:44062->145.79.227.19:4078: use of closed network connection Dec 27 16:58:17.357029766 (529h29m17.228112801s) [noise=true , address=145.79.227.19:4078 , piece_id=HS566A5YYJ7XC43YLZFQWHOVSHKMWGZZ5BKSDUIZ6GEUAEFQIDHQ, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357046699 (529h29m17.228129734s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: expected piece hash; context canceled; EOF Dec 27 16:58:17.357058151 (529h29m17.228141186s) [noise=true , address=77.90.31.15:4697 , piece_id=4757TKPPMUTE3CFGBHTTKNANDBFBDYC462AOYPYZ6U5CP4Z7PZWA, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357246279 (529h29m17.228329314s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:167JkPLSCrURmtr6PmVHr4i6huBvTzVXXxhLjkB1RkfC6aqox6): protocol: expected piece hash; context canceled; EOF Dec 27 16:58:17.357252731 (529h29m17.228335766s) [noise=true , address=45.91.116.11:4117 , piece_id=JEELYX7O2LSZT3Z64BZOGC3U4T6BITHQDWP64X5R7NUHLDZBBWDQ, seg_pos={0 0}, upload=464387]: Long tail cancellation due to optimal threshold Dec 27 16:58:17.357268271 (529h29m17.228351306s) [seg_pos={0 0}, upload=464387]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:58:17.357334847 (529h29m17.228417883s) [upload=464387]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:58:17.357339436 (529h29m17.228422471s) [upload=464387]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:58:17.647876318 (529h29m17.518959363s) [upload=464387]: Done uploading segments... Dec 27 16:58:17.647901316 (529h29m17.518984361s) [upload=464387]: Upload finished. err: <nil> Duration: 30.003624258s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.....X...X.............................X..............X.....] duration: 4.507149948s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:58:40 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.507167151s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [X.....X...X.............................X..............X....] duration: 3.563967696s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:57:57 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.563992363s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [.X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X.....] duration: 5.929031425s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:57:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 5.929060671s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 23 out of 60 is failed (threshold=3) [..X.XXX..X.XX.X..X..XXXX.XX.......XX.......XX.XXX......X....] duration: 5.92566246s trace: https://jaeger.dev.storj.io/trace/645f8a46074b6d72 FAIL (2025-12-27 16:57:09 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, Piece ID: FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ) context deadline exceeded; piecestore: (Node ID: 1QNr5B8kdrSa59rHAb82TcvSYpru7m7zbdxXqqxw3FpcFyfpks, Piece ID: IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA) context deadline exceeded ======LOGS====== Dec 27 16:54:09.395244253 (529h25m9.266327288s) [upload=464328]: Starting upload Dec 27 16:54:09.395315138 (529h25m9.266398183s) [upload=464328]: Done starting upload Dec 27 16:54:09.395538925 (529h25m9.26662197s) [upload=464328]: Uploading segments... Dec 27 16:54:09.395664463 (529h25m9.266747518s) [upload=464328]: Got next segment. Inline: false Dec 27 16:54:09.395729166 (529h25m9.266812201s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 27 16:54:09.395736420 (529h25m9.266819465s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 27 16:54:09.640946212 (529h25m9.512029257s) [seg_pos={0 0}, upload=464328]: Begin upload segment... Dec 27 16:54:09.641002107 (529h25m9.512085153s) [seg_pos={0 0}, upload=464328]: Done begin upload segment. Dec 27 16:54:09.641033758 (529h25m9.512116823s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641104863 (529h25m9.512187898s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641605948 (529h25m9.512688983s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.641793555 (529h25m9.51287659s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642172406 (529h25m9.513255441s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.642571036 (529h25m9.513654071s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643041541 (529h25m9.514124576s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643124990 (529h25m9.514208035s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643844540 (529h25m9.514927575s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.643909765 (529h25m9.51499281s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644564812 (529h25m9.515647847s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.644616149 (529h25m9.515699184s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645316272 (529h25m9.516399307s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.645380525 (529h25m9.51646357s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.646264679 (529h25m9.517347714s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:09.647019506 (529h25m9.518102541s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Uploading piece... Dec 27 16:54:11.251571098 (529h25m11.122654133s) [upload=464328]: Next returned nil segment Dec 27 16:54:11.251584153 (529h25m11.122667198s) [upload=464328]: Waiting for error group managing segments... Dec 27 16:54:11.635070998 (529h25m11.506154043s) [noise=true , address=145.79.227.16:4909 , piece_id=YTQAW3MMMPTS3M7BO4NOVLIWFKIQZWVK6ZIMXH3U2ZYOULQECMAA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.637746432 (529h25m11.508829467s) [noise=true , address=145.79.227.9:4365 , piece_id=MCXE2KYU5JBU3OXS6TT64SHOUFIAPBKTR7ARFSLTS2SM2G2DJQXA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.642795406 (529h25m11.513878462s) [noise=true , address=145.79.227.14:4839 , piece_id=W6JMTNAW6D3MXYOWJ2PHWHHTBMW427FQPG7PJ6553Z3GWS4SWO4A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.646797187 (529h25m11.517880222s) [noise=true , address=145.79.227.36:4518 , piece_id=OKWDCKQ54JV3DIDYXXKDWYU4QC2UA64CV4H6HIIIS52MPCQ2TY6A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.648722423 (529h25m11.519805468s) [noise=true , address=145.79.227.37:4566 , piece_id=EN6SVXPEQ4PJPFBXTWC7QYYJCTONKGA7GQCVY543WZYKOOPZMBZQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.653115148 (529h25m11.524198183s) [noise=true , address=145.79.227.17:4970 , piece_id=ZLWSSBY5ILQKDPSG3DQ6LJOGQ6FSNOIX3UVJDDBQPZWKFIAM556A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.669518742 (529h25m11.540601777s) [noise=true , address=145.79.227.15:4874 , piece_id=WEUBEZCGQ3MR4CVJUT5NUOLVIXRC2RR4SSACVY7A7NXB2PECN6KA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.673846613 (529h25m11.544929658s) [noise=true , address=145.79.227.32:4836 , piece_id=76D444RQRVBU7FXEDT4JEEFIVBCGVZMGM6GNQBHVMUK6PRX36WFQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.682323394 (529h25m11.553406429s) [noise=true , address=145.79.227.48:4302 , piece_id=SJ2JWQEHY5QA2W2W46RYUPTUAISCJMN4CH5M5SSAXHJGMYGZI5XA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.939294221 (529h25m11.810377266s) [noise=true , address=145.79.227.42:4121 , piece_id=PGPOOZZG67RBT5EOHJUMV6KFJOF47ZX733MV572AI6POZO3WUPLQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:11.959324092 (529h25m11.830407127s) [noise=true , address=77.90.31.16:4736 , piece_id=FBMSGDFE7Z3ZABZLI77XQVJSBEXQUDZOZ5TEGNSGXQXW5HA2METQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.042123345 (529h25m11.91320639s) [noise=true , address=45.91.116.4:4937 , piece_id=XMN7TEOWIX7MBMF2MJQV63N332QXMU2TH22OVM3QXJ2TJMANUYYQ, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409785817 (529h25m12.280868883s) [noise=true , address=77.90.31.11:4556 , piece_id=IADCEXPJGNTPYSJ6YEJYBNXYZEZ23TNPSKCSWZEMQD56MIEA5QUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: <nil> Dec 27 16:54:12.409842215 (529h25m12.28092526s) [seg_pos={0 0}, upload=464328]: Segment reached optimal threshold of 13 pieces. Dec 27 16:54:12.410069919 (529h25m12.281152974s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410079758 (529h25m12.281162803s) [noise=true , address=45.91.116.7:4010 , piece_id=VW6WG4C2IVJHMWGRA2JBJIOWTRGGGBGOSX3HSPYKEQDEZWGUW55A, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410393385 (529h25m12.28147643s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410409776 (529h25m12.281492821s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1No1jHEK6YFSye92GGBri48Yv2DYUiESY6Gs3Z9PENjmBynS9b): protocol: expected piece hash; context canceled; EOF Dec 27 16:54:12.410432419 (529h25m12.281515464s) [noise=true , address=145.79.227.30:4762 , piece_id=5VFR7JF3U4BYI3APF74YTR3ASQ4MELLAUDB374MZTCG62SFQP3WA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410449933 (529h25m12.281532968s) [noise=true , address=45.91.116.10:4082 , piece_id=CBVZKKPGXSZA5JPAU4SGGC52INF6N2UVWIYDBHDZL5YD2LM4AUUA, seg_pos={0 0}, upload=464328]: Long tail cancellation due to optimal threshold Dec 27 16:54:12.410468418 (529h25m12.281551463s) [seg_pos={0 0}, upload=464328]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Dec 27 16:54:12.410589569 (529h25m12.281672654s) [upload=464328]: Flush batch item: pb.BatchRequestItem_SegmentCommit Dec 27 16:54:12.410597073 (529h25m12.281680108s) [upload=464328]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 27 16:54:12.695997364 (529h25m12.567080409s) [upload=464328]: Done uploading segments... Dec 27 16:54:12.696056086 (529h25m12.567139121s) [upload=464328]: Upload finished. err: <nil> Duration: 5.925693459s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 5 out of 60 is failed (threshold=6) [.X.....X...X.............................X..............X...] duration: 3.624632904s trace: https://jaeger.dev.storj.io/trace/63f3b817fe611960 FAIL (2025-12-27 16:56:49 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 3.624659504s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)