Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.271455752s FAIL (2025-09-29 06:45:04 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 29 06:41:00.181194479 (137h9m45.643286952s) [upload=120722]: Starting upload Sep 29 06:41:00.181297906 (137h9m45.643390379s) [upload=120722]: Done starting upload Sep 29 06:41:00.181482096 (137h9m45.643574569s) [upload=120722]: Uploading segments... Sep 29 06:41:00.181556399 (137h9m45.643648892s) [upload=120722]: Got next segment. Inline: false Sep 29 06:41:00.181601985 (137h9m45.643694468s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 29 06:41:00.181607776 (137h9m45.643700259s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 29 06:41:00.432663241 (137h9m45.894755714s) [seg_pos={0 0}, upload=120722]: Begin upload segment... Sep 29 06:41:00.432728205 (137h9m45.894820678s) [seg_pos={0 0}, upload=120722]: Done begin upload segment. Sep 29 06:41:00.432754124 (137h9m45.894846598s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.432861900 (137h9m45.894954383s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.433368625 (137h9m45.895461108s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.436558298 (137h9m45.898650781s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.437470897 (137h9m45.89956338s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438368234 (137h9m45.900460717s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438847187 (137h9m45.90093966s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.439976147 (137h9m45.90206863s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.440125170 (137h9m45.902217654s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.441438081 (137h9m45.903530564s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.443691411 (137h9m45.905783884s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.445119691 (137h9m45.907212174s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.446594028 (137h9m45.908686512s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.448493746 (137h9m45.910586229s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.449140788 (137h9m45.911233271s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.450874088 (137h9m45.912966572s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:02.111479968 (137h9m47.573572451s) [upload=120722]: Next returned nil segment Sep 29 06:41:02.111535274 (137h9m47.573627757s) [upload=120722]: Waiting for error group managing segments... Sep 29 06:41:02.505279655 (137h9m47.967372148s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507060486 (137h9m47.969152969s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507741463 (137h9m47.969833946s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.516901164 (137h9m47.978993647s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.517526335 (137h9m47.979618818s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.548263313 (137h9m48.010355806s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.554649243 (137h9m48.016741736s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.575602401 (137h9m48.037694884s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.594535884 (137h9m48.056628367s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.597991635 (137h9m48.060084128s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.605536753 (137h9m48.067629236s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.606701620 (137h9m48.068794103s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618151523 (137h9m48.080244016s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618182532 (137h9m48.080275015s) [seg_pos={0 0}, upload=120722]: Segment reached optimal threshold of 13 pieces. Sep 29 06:41:02.618346805 (137h9m48.080439288s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618359068 (137h9m48.080451551s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618428571 (137h9m48.080521074s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618441706 (137h9m48.080534189s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618475199 (137h9m48.080567682s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618483836 (137h9m48.080576319s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618509605 (137h9m48.080602088s) [seg_pos={0 0}, upload=120722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 29 06:41:02.618619545 (137h9m48.080712028s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 29 06:41:02.618629133 (137h9m48.080721616s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 29 06:41:02.942512436 (137h9m48.404604919s) [upload=120722]: Done uploading segments... Sep 29 06:41:02.942536411 (137h9m48.404628894s) [upload=120722]: Upload finished. err: <nil> Duration: 6.27147622s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 6.186961858s FAIL (2025-09-29 06:44:22 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 29 06:41:00.181194479 (137h9m45.643286952s) [upload=120722]: Starting upload Sep 29 06:41:00.181297906 (137h9m45.643390379s) [upload=120722]: Done starting upload Sep 29 06:41:00.181482096 (137h9m45.643574569s) [upload=120722]: Uploading segments... Sep 29 06:41:00.181556399 (137h9m45.643648892s) [upload=120722]: Got next segment. Inline: false Sep 29 06:41:00.181601985 (137h9m45.643694468s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 29 06:41:00.181607776 (137h9m45.643700259s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 29 06:41:00.432663241 (137h9m45.894755714s) [seg_pos={0 0}, upload=120722]: Begin upload segment... Sep 29 06:41:00.432728205 (137h9m45.894820678s) [seg_pos={0 0}, upload=120722]: Done begin upload segment. Sep 29 06:41:00.432754124 (137h9m45.894846598s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.432861900 (137h9m45.894954383s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.433368625 (137h9m45.895461108s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.436558298 (137h9m45.898650781s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.437470897 (137h9m45.89956338s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438368234 (137h9m45.900460717s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438847187 (137h9m45.90093966s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.439976147 (137h9m45.90206863s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.440125170 (137h9m45.902217654s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.441438081 (137h9m45.903530564s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.443691411 (137h9m45.905783884s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.445119691 (137h9m45.907212174s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.446594028 (137h9m45.908686512s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.448493746 (137h9m45.910586229s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.449140788 (137h9m45.911233271s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.450874088 (137h9m45.912966572s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:02.111479968 (137h9m47.573572451s) [upload=120722]: Next returned nil segment Sep 29 06:41:02.111535274 (137h9m47.573627757s) [upload=120722]: Waiting for error group managing segments... Sep 29 06:41:02.505279655 (137h9m47.967372148s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507060486 (137h9m47.969152969s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507741463 (137h9m47.969833946s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.516901164 (137h9m47.978993647s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.517526335 (137h9m47.979618818s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.548263313 (137h9m48.010355806s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.554649243 (137h9m48.016741736s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.575602401 (137h9m48.037694884s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.594535884 (137h9m48.056628367s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.597991635 (137h9m48.060084128s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.605536753 (137h9m48.067629236s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.606701620 (137h9m48.068794103s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618151523 (137h9m48.080244016s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618182532 (137h9m48.080275015s) [seg_pos={0 0}, upload=120722]: Segment reached optimal threshold of 13 pieces. Sep 29 06:41:02.618346805 (137h9m48.080439288s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618359068 (137h9m48.080451551s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618428571 (137h9m48.080521074s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618441706 (137h9m48.080534189s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618475199 (137h9m48.080567682s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618483836 (137h9m48.080576319s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618509605 (137h9m48.080602088s) [seg_pos={0 0}, upload=120722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 29 06:41:02.618619545 (137h9m48.080712028s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 29 06:41:02.618629133 (137h9m48.080721616s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 29 06:41:02.942512436 (137h9m48.404604919s) [upload=120722]: Done uploading segments... Sep 29 06:41:02.942536411 (137h9m48.404628894s) [upload=120722]: Upload finished. err: <nil> Duration: 6.186993049s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 6.317592193s FAIL (2025-09-29 06:43:21 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 29 06:41:00.181194479 (137h9m45.643286952s) [upload=120722]: Starting upload Sep 29 06:41:00.181297906 (137h9m45.643390379s) [upload=120722]: Done starting upload Sep 29 06:41:00.181482096 (137h9m45.643574569s) [upload=120722]: Uploading segments... Sep 29 06:41:00.181556399 (137h9m45.643648892s) [upload=120722]: Got next segment. Inline: false Sep 29 06:41:00.181601985 (137h9m45.643694468s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 29 06:41:00.181607776 (137h9m45.643700259s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 29 06:41:00.432663241 (137h9m45.894755714s) [seg_pos={0 0}, upload=120722]: Begin upload segment... Sep 29 06:41:00.432728205 (137h9m45.894820678s) [seg_pos={0 0}, upload=120722]: Done begin upload segment. Sep 29 06:41:00.432754124 (137h9m45.894846598s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.432861900 (137h9m45.894954383s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.433368625 (137h9m45.895461108s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.436558298 (137h9m45.898650781s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.437470897 (137h9m45.89956338s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438368234 (137h9m45.900460717s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438847187 (137h9m45.90093966s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.439976147 (137h9m45.90206863s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.440125170 (137h9m45.902217654s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.441438081 (137h9m45.903530564s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.443691411 (137h9m45.905783884s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.445119691 (137h9m45.907212174s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.446594028 (137h9m45.908686512s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.448493746 (137h9m45.910586229s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.449140788 (137h9m45.911233271s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.450874088 (137h9m45.912966572s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:02.111479968 (137h9m47.573572451s) [upload=120722]: Next returned nil segment Sep 29 06:41:02.111535274 (137h9m47.573627757s) [upload=120722]: Waiting for error group managing segments... Sep 29 06:41:02.505279655 (137h9m47.967372148s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507060486 (137h9m47.969152969s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507741463 (137h9m47.969833946s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.516901164 (137h9m47.978993647s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.517526335 (137h9m47.979618818s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.548263313 (137h9m48.010355806s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.554649243 (137h9m48.016741736s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.575602401 (137h9m48.037694884s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.594535884 (137h9m48.056628367s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.597991635 (137h9m48.060084128s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.605536753 (137h9m48.067629236s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.606701620 (137h9m48.068794103s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618151523 (137h9m48.080244016s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618182532 (137h9m48.080275015s) [seg_pos={0 0}, upload=120722]: Segment reached optimal threshold of 13 pieces. Sep 29 06:41:02.618346805 (137h9m48.080439288s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618359068 (137h9m48.080451551s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618428571 (137h9m48.080521074s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618441706 (137h9m48.080534189s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618475199 (137h9m48.080567682s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618483836 (137h9m48.080576319s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618509605 (137h9m48.080602088s) [seg_pos={0 0}, upload=120722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 29 06:41:02.618619545 (137h9m48.080712028s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 29 06:41:02.618629133 (137h9m48.080721616s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 29 06:41:02.942512436 (137h9m48.404604919s) [upload=120722]: Done uploading segments... Sep 29 06:41:02.942536411 (137h9m48.404628894s) [upload=120722]: Upload finished. err: <nil> Duration: 6.317629473s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.110067469s FAIL (2025-09-29 06:42:07 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 29 06:41:00.181194479 (137h9m45.643286952s) [upload=120722]: Starting upload Sep 29 06:41:00.181297906 (137h9m45.643390379s) [upload=120722]: Done starting upload Sep 29 06:41:00.181482096 (137h9m45.643574569s) [upload=120722]: Uploading segments... Sep 29 06:41:00.181556399 (137h9m45.643648892s) [upload=120722]: Got next segment. Inline: false Sep 29 06:41:00.181601985 (137h9m45.643694468s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 29 06:41:00.181607776 (137h9m45.643700259s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 29 06:41:00.432663241 (137h9m45.894755714s) [seg_pos={0 0}, upload=120722]: Begin upload segment... Sep 29 06:41:00.432728205 (137h9m45.894820678s) [seg_pos={0 0}, upload=120722]: Done begin upload segment. Sep 29 06:41:00.432754124 (137h9m45.894846598s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.432861900 (137h9m45.894954383s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.433368625 (137h9m45.895461108s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.436558298 (137h9m45.898650781s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.437470897 (137h9m45.89956338s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438368234 (137h9m45.900460717s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438847187 (137h9m45.90093966s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.439976147 (137h9m45.90206863s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.440125170 (137h9m45.902217654s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.441438081 (137h9m45.903530564s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.443691411 (137h9m45.905783884s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.445119691 (137h9m45.907212174s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.446594028 (137h9m45.908686512s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.448493746 (137h9m45.910586229s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.449140788 (137h9m45.911233271s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.450874088 (137h9m45.912966572s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:02.111479968 (137h9m47.573572451s) [upload=120722]: Next returned nil segment Sep 29 06:41:02.111535274 (137h9m47.573627757s) [upload=120722]: Waiting for error group managing segments... Sep 29 06:41:02.505279655 (137h9m47.967372148s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507060486 (137h9m47.969152969s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507741463 (137h9m47.969833946s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.516901164 (137h9m47.978993647s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.517526335 (137h9m47.979618818s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.548263313 (137h9m48.010355806s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.554649243 (137h9m48.016741736s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.575602401 (137h9m48.037694884s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.594535884 (137h9m48.056628367s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.597991635 (137h9m48.060084128s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.605536753 (137h9m48.067629236s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.606701620 (137h9m48.068794103s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618151523 (137h9m48.080244016s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618182532 (137h9m48.080275015s) [seg_pos={0 0}, upload=120722]: Segment reached optimal threshold of 13 pieces. Sep 29 06:41:02.618346805 (137h9m48.080439288s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618359068 (137h9m48.080451551s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618428571 (137h9m48.080521074s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618441706 (137h9m48.080534189s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618475199 (137h9m48.080567682s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618483836 (137h9m48.080576319s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618509605 (137h9m48.080602088s) [seg_pos={0 0}, upload=120722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 29 06:41:02.618619545 (137h9m48.080712028s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 29 06:41:02.618629133 (137h9m48.080721616s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 29 06:41:02.942512436 (137h9m48.404604919s) [upload=120722]: Done uploading segments... Sep 29 06:41:02.942536411 (137h9m48.404628894s) [upload=120722]: Upload finished. err: <nil> Duration: 6.110098388s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002357891s FAIL (2025-09-29 06:41:29 UTC) orbiter warning: operation: ListObjects ======ERROR====== uplink: metainfo: metaclient: context deadline exceeded ======LOGS====== Sep 29 06:41:00.181194479 (137h9m45.643286952s) [upload=120722]: Starting upload Sep 29 06:41:00.181297906 (137h9m45.643390379s) [upload=120722]: Done starting upload Sep 29 06:41:00.181482096 (137h9m45.643574569s) [upload=120722]: Uploading segments... Sep 29 06:41:00.181556399 (137h9m45.643648892s) [upload=120722]: Got next segment. Inline: false Sep 29 06:41:00.181601985 (137h9m45.643694468s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 29 06:41:00.181607776 (137h9m45.643700259s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 29 06:41:00.432663241 (137h9m45.894755714s) [seg_pos={0 0}, upload=120722]: Begin upload segment... Sep 29 06:41:00.432728205 (137h9m45.894820678s) [seg_pos={0 0}, upload=120722]: Done begin upload segment. Sep 29 06:41:00.432754124 (137h9m45.894846598s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.432861900 (137h9m45.894954383s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.433368625 (137h9m45.895461108s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.436558298 (137h9m45.898650781s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.437470897 (137h9m45.89956338s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438368234 (137h9m45.900460717s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.438847187 (137h9m45.90093966s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.439976147 (137h9m45.90206863s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.440125170 (137h9m45.902217654s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.441438081 (137h9m45.903530564s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.443691411 (137h9m45.905783884s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.445119691 (137h9m45.907212174s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.446594028 (137h9m45.908686512s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.448493746 (137h9m45.910586229s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.449140788 (137h9m45.911233271s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:00.450874088 (137h9m45.912966572s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Uploading piece... Sep 29 06:41:02.111479968 (137h9m47.573572451s) [upload=120722]: Next returned nil segment Sep 29 06:41:02.111535274 (137h9m47.573627757s) [upload=120722]: Waiting for error group managing segments... Sep 29 06:41:02.505279655 (137h9m47.967372148s) [noise=true , address=145.79.227.18:4005 , piece_id=ZLH3JC6QO7NMEIUWIPV7DFL456AY66MWUEQ7WJXCNF6XGSAOAA6Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507060486 (137h9m47.969152969s) [noise=true , address=145.79.227.11:4731 , piece_id=E55SCM4VEWXJQQ4PCMS522P453JPMT564XGHPWXQGIU2YQCRT7VQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.507741463 (137h9m47.969833946s) [noise=true , address=145.79.227.6:4257 , piece_id=RWK2HJDPU47LWMKZSA436ZI4T2UVGHNGM2FH66346TDL2UIES5IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.516901164 (137h9m47.978993647s) [noise=true , address=145.79.227.4:4186 , piece_id=2CAZDPXNZ2O4P4SSK53XPMZSP4MVBZS7IQX7ZHZEMO2I66F5OYGA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.517526335 (137h9m47.979618818s) [noise=true , address=145.79.227.19:4079 , piece_id=WO3OQM5QNZDC4OCO4RXXE2BEOSNN43XM3AJK7L2H4PS7QIWBOOBA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.548263313 (137h9m48.010355806s) [noise=true , address=45.91.116.12:4227 , piece_id=7YYYQSEPPCSJWUYBIMJUDEI2FP2M6J3T2UZTFBPMYJWZKWHVN4UA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.554649243 (137h9m48.016741736s) [noise=true , address=45.91.116.11:4117 , piece_id=RLF3UC45C2KZPM2VUUCMOIYXVFKRX656NZHAYCSH2G6QJKL23VDA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.575602401 (137h9m48.037694884s) [noise=true , address=45.91.116.13:4154 , piece_id=QYO63B33KZ7CMJOUZ4YPZFGGPFISQW5DQON4O3PPPIBARN2D7C2A, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.594535884 (137h9m48.056628367s) [noise=true , address=45.91.116.4:4939 , piece_id=X6KEO4UPCYW3HWXK6F6J7PDUWBAL5DEE77RAYMNH2PXIQKOLL76Q, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.597991635 (137h9m48.060084128s) [noise=true , address=77.90.31.19:4842 , piece_id=7P33B534BW5ODFADWJAGSJENBF6USQBEXJ3IRF2QZGNRKW5SGRIA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.605536753 (137h9m48.067629236s) [noise=true , address=45.91.116.11:4119 , piece_id=N7R6XVPVKPYWSTPPVN33OS2AUH6X4KAZIPEML566SELWWPKDRKHA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.606701620 (137h9m48.068794103s) [noise=true , address=77.90.31.15:4699 , piece_id=KZ4ZWE423BBZ2JEC7KURVTJWOQMCIKRQAPF3VT2SAMRKARBFM5OA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618151523 (137h9m48.080244016s) [noise=true , address=45.91.116.6:4974 , piece_id=DWJOU65HBHOVIWQ6UZ6JWTU2OXKNSYR2UPF3FNY4CU5G2S2SU3EQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: <nil> Sep 29 06:41:02.618182532 (137h9m48.080275015s) [seg_pos={0 0}, upload=120722]: Segment reached optimal threshold of 13 pieces. Sep 29 06:41:02.618346805 (137h9m48.080439288s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618359068 (137h9m48.080451551s) [noise=true , address=77.90.31.11:4554 , piece_id=NYKGYO7Z53BTV6NBYAZ7FLQHYZLXVK4BUJ6SXYZGJVUBUB2A65IQ, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618428571 (137h9m48.080521074s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12faSN2qTnzvZquF4ekHA6PX5tDSAyrVTkWuy1Z3FxgByxZ4jue): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618441706 (137h9m48.080534189s) [noise=true , address=77.90.31.14:4661 , piece_id=NBN4TMHIMVLITA67MYNGBB2FKMYZFTPOU4MIKZT2ROV7ISX4TNLA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618475199 (137h9m48.080567682s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12UqimG6nUH8jMQVyovyRz8PXqRzRGbFF2u1UjScB5RdsvHECqG): protocol: expected piece hash; context canceled; EOF Sep 29 06:41:02.618483836 (137h9m48.080576319s) [noise=true , address=45.91.116.10:4081 , piece_id=FESUGLRMWMOXK7I35DNR6SACW5XTC4VAKWX4W7H6DFCPXG7MLPTA, seg_pos={0 0}, upload=120722]: Long tail cancellation due to optimal threshold Sep 29 06:41:02.618509605 (137h9m48.080602088s) [seg_pos={0 0}, upload=120722]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 29 06:41:02.618619545 (137h9m48.080712028s) [upload=120722]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 29 06:41:02.618629133 (137h9m48.080721616s) [upload=120722]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 29 06:41:02.942512436 (137h9m48.404604919s) [upload=120722]: Done uploading segments... Sep 29 06:41:02.942536411 (137h9m48.404628894s) [upload=120722]: Upload finished. err: <nil> Duration: 30.002413456s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 6.342525583s FAIL (2025-09-28 23:38:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.342557273s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 6.695607213s FAIL (2025-09-28 23:37:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.69563184s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 6.242015007s FAIL (2025-09-28 23:36:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.242045144s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 6.230954815s FAIL (2025-09-28 23:35:40 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.230984472s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 6.027128174s FAIL (2025-09-28 23:34:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.027155546s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 6.08689406s FAIL (2025-09-28 23:33:29 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.08692557s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 2.069073873s FAIL (2025-09-28 23:32:44 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.069103368s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.945270186s FAIL (2025-09-28 23:32:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.945299761s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.849392322s FAIL (2025-09-28 23:31:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.849425153s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 2.268567752s FAIL (2025-09-28 23:31:30 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.268605043s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 6.19384977s FAIL (2025-09-28 23:30:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.193874969s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 2.004064256s FAIL (2025-09-28 23:30:23 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.004090846s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 6.172837278s FAIL (2025-09-28 23:29:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.172869569s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 2.289491422s FAIL (2025-09-28 23:29:37 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.289514586s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 6.16093714s FAIL (2025-09-28 23:29:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.16099505s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 2.07269782s FAIL (2025-09-28 23:28:42 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.072752534s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 6.244445319s FAIL (2025-09-28 23:28:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.244480636s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 2.078602904s FAIL (2025-09-28 23:27:59 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.078641027s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 2.008955314s FAIL (2025-09-28 23:27:22 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.008980231s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 6.221622887s FAIL (2025-09-28 23:27:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.221655239s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 2.082704003s FAIL (2025-09-28 23:27:03 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.082741354s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 6.112027272s FAIL (2025-09-28 23:26:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.11205787s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 2.115572049s FAIL (2025-09-28 23:25:55 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.115613347s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 2.17717937s FAIL (2025-09-28 23:25:19 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.177216601s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 6.116611948s FAIL (2025-09-28 23:25:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.116647185s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 2.10093698s FAIL (2025-09-28 23:24:51 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.100980905s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 6.452670574s FAIL (2025-09-28 23:24:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.452700492s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 2.100529483s FAIL (2025-09-28 23:24:01 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.100570543s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 6.322589656s FAIL (2025-09-28 23:23:50 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.322621728s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 2.092286278s FAIL (2025-09-28 23:22:53 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.092322177s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 6.225650407s FAIL (2025-09-28 23:22:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.225687709s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 2.111958839s FAIL (2025-09-28 23:22:27 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.11199061s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.986068808s FAIL (2025-09-28 23:22:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.986099146s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 2.07887947s FAIL (2025-09-28 23:21:54 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.078914186s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 6.755266091s FAIL (2025-09-28 23:21:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.75529752s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 2.283605462s FAIL (2025-09-28 23:21:02 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.283663121s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 6.464090565s FAIL (2025-09-28 23:20:44 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.46412452s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 2.122549916s FAIL (2025-09-28 23:20:21 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.122603718s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 6.375506033s FAIL (2025-09-28 23:20:06 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.375539597s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 6.198810911s FAIL (2025-09-28 23:19:41 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.198869703s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 2.233653313s FAIL (2025-09-28 23:19:14 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.233684232s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 6.565740393s FAIL (2025-09-28 23:19:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.565788525s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 5.999656417s FAIL (2025-09-28 23:18:17 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 5.999693117s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 2.077928519s FAIL (2025-09-28 23:18:15 UTC) orbiter warning: operation: DeleteObject uplink: metaclient: context deadline exceeded Duration: 2.077972984s core uplink us1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 6.328274368s FAIL (2025-09-28 23:17:52 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Sep 28 22:50:24.901683454 (129h19m10.363775927s) [upload=113836]: Starting upload Sep 28 22:50:24.901752295 (129h19m10.363844768s) [upload=113836]: Done starting upload Sep 28 22:50:24.901879066 (129h19m10.363971539s) [upload=113836]: Uploading segments... Sep 28 22:50:24.902030415 (129h19m10.364122898s) [upload=113836]: Got next segment. Inline: false Sep 28 22:50:24.902077033 (129h19m10.364169516s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 28 22:50:24.902083035 (129h19m10.364175508s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 28 22:50:26.852393655 (129h19m12.314486128s) [seg_pos={0 0}, upload=113836]: Begin upload segment... Sep 28 22:50:26.852460853 (129h19m12.314553326s) [seg_pos={0 0}, upload=113836]: Done begin upload segment. Sep 28 22:50:26.852475400 (129h19m12.314567873s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.852488045 (129h19m12.314580518s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853029766 (129h19m12.315122249s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853296834 (129h19m12.315389317s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853540498 (129h19m12.315632991s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853566488 (129h19m12.315658971s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853606073 (129h19m12.315698556s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.853931903 (129h19m12.316024376s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854100585 (129h19m12.316193068s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854231675 (129h19m12.316324148s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854504174 (129h19m12.316596647s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854593263 (129h19m12.316685736s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.854764679 (129h19m12.316857152s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855066444 (129h19m12.317158917s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855570193 (129h19m12.317662676s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:26.855841500 (129h19m12.317933983s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Uploading piece... Sep 28 22:50:28.582844923 (129h19m14.044937406s) [upload=113836]: Next returned nil segment Sep 28 22:50:28.582866864 (129h19m14.044959347s) [upload=113836]: Waiting for error group managing segments... Sep 28 22:50:28.881030230 (129h19m14.343122713s) [noise=true , address=145.79.227.4:4186 , piece_id=DZPVZ6TTS523WUKPQBVTDHWDLRFMRWMT673YDGKIOGD6Y6LJLLKQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.887290389 (129h19m14.349382872s) [noise=true , address=145.79.227.16:4910 , piece_id=YNU3QIF56QQQFB5UQITRXEKNDQDBO3FJ4Q4WFSEGG74FHCNNKDWQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888070155 (129h19m14.350162638s) [noise=true , address=145.79.227.7:4294 , piece_id=SSW3KWRKDWZJ2JCNATQQUSFT3YGDD2AYBRCQQ466BSIS243MX7JQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888212486 (129h19m14.350304979s) [noise=true , address=145.79.227.5:4221 , piece_id=AISUWYT5DEQNHRO4XCSCAWMRWQZJALZM34H46MHYQU6FZ366VE2Q, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.888968586 (129h19m14.351061059s) [noise=true , address=145.79.227.12:4765 , piece_id=LGXQUKQ23WX23FACNH6WXAV5X2K4PUVQFVG5OLCMZTJEFJJDAXCQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.901633701 (129h19m14.363726184s) [noise=true , address=145.79.227.6:4258 , piece_id=GW2CCZ7LS52Q4BZW6D7LMCY4IIK3P7TG5J4VPSS4CKLRCF7CSOQA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905686599 (129h19m14.367779082s) [noise=true , address=145.79.227.13:4801 , piece_id=NCEQJYSYM37IJT724RGYQZT3BEQT6EMHW2PN6YYLJHKHNDZ7IB4A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:28.905792000 (129h19m14.367884483s) [noise=true , address=145.79.227.3:4151 , piece_id=6L2HCDQHVN43MLKGSCSQDYYHOWGPURZ2AHIDPT6A577RZS22ECXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.038837341 (129h19m14.500929824s) [noise=true , address=45.91.116.7:4011 , piece_id=VRHQ7UKQYOYZH6O4IP55CN5SP6YTOIYKA3B4M3IQUAOKLQV2PRMQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.052413222 (129h19m14.514505705s) [noise=true , address=45.91.116.6:4974 , piece_id=ZSNORZRG5XAPDFYHO2NZ2EZZWU2767SUYXYBWBU77T25VLXGRXFA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.124522220 (129h19m14.586614693s) [noise=true , address=77.90.31.18:4807 , piece_id=PAE3COMCDOBJKH73EPUDMTSVPNILJH6SZ7HWXOWBJVP3CUBNXBSA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.166734780 (129h19m14.628827273s) [noise=true , address=45.91.116.5:4297 , piece_id=WOTCIC7IJ2MSJBNUIXRMC5M5BOORRVKI6DMBTKO7SMPCGP2KXNXQ, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.169989688 (129h19m14.632082172s) [noise=true , address=45.91.116.10:4081 , piece_id=FYELLZRKVZ346QV2ZZ76UB7SXS7H64THXPWSHQJHZ4HLZRJCC35A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: <nil> Sep 28 22:50:29.170018583 (129h19m14.632111066s) [seg_pos={0 0}, upload=113836]: Segment reached optimal threshold of 13 pieces. Sep 28 22:50:29.170184509 (129h19m14.632276992s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170196101 (129h19m14.632288584s) [noise=true , address=45.91.116.12:4226 , piece_id=7CKQIFZIRGMM6ZIMTUNG45GPTYBK4BV2V5E4D3PBGDXRONY6YYZA, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170339364 (129h19m14.632431847s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170347790 (129h19m14.632440263s) [noise=true , address=77.90.31.13:4626 , piece_id=B6VZSRKLCVJ5P2KJCHH7KWWLLIPGNTVDQ7QUOTDTCDHH6Q33CP5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170461887 (129h19m14.63255437s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK): protocol: expected piece hash; context canceled; EOF Sep 28 22:50:29.170476124 (129h19m14.632568627s) [noise=true , address=77.90.31.21:4878 , piece_id=NAPYUKEGFXO5KIGC6WJI5WPJPKLHGWB365L6N4MDBYYNC7ZCDC5A, seg_pos={0 0}, upload=113836]: Long tail cancellation due to optimal threshold Sep 28 22:50:29.170500641 (129h19m14.632593114s) [seg_pos={0 0}, upload=113836]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Sep 28 22:50:29.170600972 (129h19m14.632693445s) [upload=113836]: Flush batch item: pb.BatchRequestItem_SegmentCommit Sep 28 22:50:29.170609267 (129h19m14.63270175s) [upload=113836]: Flush batch item: pb.BatchRequestItem_ObjectCommit Sep 28 22:50:46.053429497 (129h19m31.51552197s) [upload=113836]: Done uploading segments... Sep 28 22:50:46.053458162 (129h19m31.515550635s) [upload=113836]: Upload finished. err: metaclient: context deadline exceeded Duration: 6.328301381s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)