Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 12.48974463s FAIL (2025-10-01 13:50:42 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 12.489786158s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 21.660161291s FAIL (2025-10-01 13:50:14 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 21.660194656s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 9.536128951s FAIL (2025-10-01 13:49:31 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 9.536165108s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 14.885203332s FAIL (2025-10-01 13:48:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 14.885277673s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................X..........................] duration: 11.281226554s FAIL (2025-10-01 13:47:18 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 11.281259206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................X.........................] duration: 29.073522413s FAIL (2025-10-01 13:46:28 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 29.073561947s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 16.043771108s FAIL (2025-10-01 13:45:12 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 16.043808039s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 15.791453427s FAIL (2025-10-01 13:44:11 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 15.79148616s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 6.276546396s FAIL (2025-10-01 13:43:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.276597163s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 15.609368895s FAIL (2025-10-01 13:43:16 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 15.609519623s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 11.016256448s FAIL (2025-10-01 13:42:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 11.016292007s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 17.247779355s FAIL (2025-10-01 13:41:50 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 17.2478143s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 7.843408605s FAIL (2025-10-01 13:40:55 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 7.84344206s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 13.747980511s FAIL (2025-10-01 13:40:05 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 13.748017892s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 18.627607614s FAIL (2025-10-01 13:39:34 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 18.627644253s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 6.590552924s FAIL (2025-10-01 13:38:46 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.590583984s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 6.434606377s FAIL (2025-10-01 13:37:40 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.434639229s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 11.927493737s FAIL (2025-10-01 13:36:39 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 11.927524044s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 6.209233964s FAIL (2025-10-01 13:35:53 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.209269523s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 16.518589109s FAIL (2025-10-01 13:35:37 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 16.518620689s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 6.056083812s FAIL (2025-10-01 13:34:56 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.056108539s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 22.022232014s FAIL (2025-10-01 13:34:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 22.022265507s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 11.782077594s FAIL (2025-10-01 13:34:06 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 11.782115467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 6.454210817s FAIL (2025-10-01 13:33:10 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.454249269s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 9.534141566s FAIL (2025-10-01 13:32:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 9.5341749s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 7.307972493s FAIL (2025-10-01 13:32:17 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 7.308005467s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 6.600604667s FAIL (2025-10-01 13:31:25 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.600637561s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 7.361696709s FAIL (2025-10-01 13:30:49 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 7.361725254s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 18.325691966s FAIL (2025-10-01 13:30:36 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 18.325729077s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 6.088776174s FAIL (2025-10-01 13:30:00 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 6.088808315s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.002003716s FAIL (2025-10-01 13:29:52 UTC) orbiter warning: operation: sync2.Copy ======ERROR====== uplink: piecestore: (Node ID: 1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, Piece ID: S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ) piecestore close: write tcp 188.166.225.170:45912->145.79.227.3:4151: use of closed network connection; piecestore: (Node ID: 1gJNAhKazJp5TabeEkXsEUFjoBu87mfYDBdeKhfGi4gub3yaSJ, Piece ID: WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ) context deadline exceeded ======LOGS====== Oct 1 13:29:23.185667767 (191h58m8.64776024s) [upload=168789]: Starting upload Oct 1 13:29:23.185749512 (191h58m8.647841995s) [upload=168789]: Done starting upload Oct 1 13:29:23.185859331 (191h58m8.647951814s) [upload=168789]: Uploading segments... Oct 1 13:29:23.186031810 (191h58m8.648124293s) [upload=168789]: Got next segment. Inline: false Oct 1 13:29:23.186074322 (191h58m8.648166795s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectBegin Oct 1 13:29:23.186078770 (191h58m8.648171253s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentBegin Oct 1 13:29:25.093078564 (191h58m10.555171047s) [seg_pos={0 0}, upload=168789]: Begin upload segment... Oct 1 13:29:25.093132697 (191h58m10.55522518s) [seg_pos={0 0}, upload=168789]: Done begin upload segment. Oct 1 13:29:25.093162043 (191h58m10.555254526s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093175509 (191h58m10.555267982s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093717451 (191h58m10.555809934s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.093736607 (191h58m10.55582908s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094010839 (191h58m10.556103312s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094266065 (191h58m10.556358548s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094619298 (191h58m10.556711771s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094727354 (191h58m10.556819827s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.094924879 (191h58m10.557017362s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095210755 (191h58m10.557303228s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095528149 (191h58m10.557620632s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.095849731 (191h58m10.557942214s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096003253 (191h58m10.558095726s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096345114 (191h58m10.558437587s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.096869803 (191h58m10.558962286s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:25.097391557 (191h58m10.55948404s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Uploading piece... Oct 1 13:29:26.777641255 (191h58m12.239733738s) [upload=168789]: Next returned nil segment Oct 1 13:29:26.777656914 (191h58m12.239749397s) [upload=168789]: Waiting for error group managing segments... Oct 1 13:29:27.056140973 (191h58m12.518233446s) [noise=true , address=145.79.227.5:4222 , piece_id=YPAQCS25LG3A2GAWCBRXQIVYIZJJ7FOQS23STXSKUNI25FC4PIHQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.057665787 (191h58m12.51975827s) [noise=true , address=145.79.227.14:4837 , piece_id=6C6CKPYDEB3RHOZOBCTZYJQZQWPVN6B2SE5V6CY47IFQOGFUK6JA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.071803667 (191h58m12.533896151s) [noise=true , address=145.79.227.13:4804 , piece_id=MBPW7LEVVY2DDCMBHFJ4MSMOKWYGMVRYZWOGJO5UZQMDODGAZIWA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.076345947 (191h58m12.53843843s) [noise=true , address=145.79.227.3:4151 , piece_id=S4FSTP2TOAIVRPGF26KZSNKFKYDO7STC5S5QILVQYSMSU4NQKGSQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.078171263 (191h58m12.540263736s) [noise=true , address=145.79.227.11:4731 , piece_id=3RSMAQVBGI35QG6MU2775MIDJTJCEPA6MTEFUMSG7JREUV5QNQFA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.088895083 (191h58m12.550987567s) [noise=true , address=145.79.227.6:4257 , piece_id=FDWBHVPZQZEFR4NUISKAY4UKQFKM6VTDLOAMHMKHDGP4STMMO5RA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.114178799 (191h58m12.576271282s) [noise=true , address=145.79.227.4:4186 , piece_id=I3KJOXWG6QGVUDYX66UCGLFFH423EHIB5XPQ3PJVLYBSGVBC6DMQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.483944827 (191h58m12.94603731s) [noise=true , address=45.91.116.12:4227 , piece_id=EGFLHJY2EA7TWPMHOCROKGNTFBUZIEVXEJ52J5CBURLPQLC3LTIQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.500766197 (191h58m12.96285867s) [noise=true , address=77.90.31.18:4806 , piece_id=WWWPUCIFHAVAROVPGTQWCB7TDV5TDYH6UM2GBGLZOIAMWPWUITDQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.528960564 (191h58m12.991053048s) [noise=true , address=45.91.116.9:4263 , piece_id=RTHY7SNNWU2UFI4TYM6EZ7NGCWUGHJ33XG6ZXCS2JVFYPURNACDA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.529348513 (191h58m12.991440986s) [noise=true , address=45.91.116.10:4081 , piece_id=7TJESVZL7SKIIKBS7RNTVN35KLWSODEEANHCZCVXZ6H7GZWR5ZIA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.530143647 (191h58m12.99223612s) [noise=true , address=45.91.116.5:4299 , piece_id=OACDDMPKEJ3L4DXVBUXUYF63NSJAET6EWL6SWNQBQWBS5P3MAPRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594922740 (191h58m13.057015223s) [noise=true , address=45.91.116.14:4190 , piece_id=P5X3H7K2NL775YRMRXVG6XTHNNP6AEKWKG77IVZ6DXE6B7GGZ3NQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: <nil> Oct 1 13:29:27.594944040 (191h58m13.057036513s) [seg_pos={0 0}, upload=168789]: Segment reached optimal threshold of 13 pieces. Oct 1 13:29:27.595054670 (191h58m13.057147143s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12Z23fYAQ6w3Wv3mXVfoqQcLR7HxW5yhrPe2xgViVeiKFsmRVcV): protocol: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:39422->45.91.116.6:4973: use of closed network connection Oct 1 13:29:27.595062266 (191h58m13.057154739s) [noise=true , address=45.91.116.6:4973 , piece_id=F5OC7PKYJYCPJFI327JQULEGBXQC2B6PBG6OYOLWNIWMVIHZ4MTA, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597504526 (191h58m13.059597009s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1XXhhMfCnZi1QRQeyH9L7xq346Y4bLR4m6VPAfz8fBdAtcc7w): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597519294 (191h58m13.059611777s) [noise=true , address=45.91.116.7:4010 , piece_id=2I6JAZMNWBOIFZDSXCPONAKZR7ILB74JOMV7U5RWYAL5NYRPVDRQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597604366 (191h58m13.059696839s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12k2VGS6TVLT1L3KSF5eUEii6VQ2jnkZaCHVrckQaCvANqFHh5C): protocol: expected piece hash; context canceled; EOF Oct 1 13:29:27.597612361 (191h58m13.059704844s) [noise=true , address=45.91.116.5:4298 , piece_id=YXJPBHKXUKCLAJS4NH7KPQZCNU6KMFBVREPAQFWLWWWKEWOSHKOQ, seg_pos={0 0}, upload=168789]: Long tail cancellation due to optimal threshold Oct 1 13:29:27.597627910 (191h58m13.059720393s) [seg_pos={0 0}, upload=168789]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Oct 1 13:29:27.597691431 (191h58m13.059783915s) [upload=168789]: Flush batch item: pb.BatchRequestItem_SegmentCommit Oct 1 13:29:27.597697343 (191h58m13.059789816s) [upload=168789]: Flush batch item: pb.BatchRequestItem_ObjectCommit Oct 1 13:29:27.954162644 (191h58m13.416255127s) [upload=168789]: Done uploading segments... Oct 1 13:29:27.954184856 (191h58m13.416277339s) [upload=168789]: Upload finished. err: <nil> Duration: 30.002027991s core uplink us-select-1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 5.398266143s FAIL (2025-10-01 12:33:31 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.398304567s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 5.589188954s FAIL (2025-10-01 12:32:41 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.589225534s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 5.597342309s FAIL (2025-10-01 12:32:19 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.597380492s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 5.730422063s FAIL (2025-10-01 12:31:58 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.730454764s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 5.464826667s FAIL (2025-10-01 12:30:53 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.464862024s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 5.591199915s FAIL (2025-10-01 12:30:08 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.591227739s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 5.600354797s FAIL (2025-10-01 12:29:37 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.600385645s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 5.419844694s FAIL (2025-10-01 12:28:52 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.419876844s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 5.782033501s FAIL (2025-10-01 12:27:52 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.782077775s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 5.958268685s FAIL (2025-10-01 12:27:31 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.958296057s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 5.663808997s FAIL (2025-10-01 12:26:55 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.663843423s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 5.442050467s FAIL (2025-10-01 12:25:43 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.44208284s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 5.320135407s FAIL (2025-10-01 12:25:04 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.320176165s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 5.364137953s FAIL (2025-10-01 12:24:49 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.364237733s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 5.764790482s FAIL (2025-10-01 12:23:37 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.764828164s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 5.947804879s FAIL (2025-10-01 12:23:22 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.947838873s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 5.871527038s FAIL (2025-10-01 12:22:34 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.871557666s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 5.916284496s FAIL (2025-10-01 12:22:18 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.916325936s core uplink eu1
checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 5.730696553s FAIL (2025-10-01 12:21:46 UTC) orbiter warning: operation: ListObjects uplink: metainfo: metaclient: context deadline exceeded Duration: 5.730739103s core uplink eu1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)

checkers.Uplink{Satellite: "12L9ZFwhzVpuEKMUNUqkaTLGzwY9G24tbiigLiXpmZWKwmcNDDs@eu1.storj.io:7777", Bucket: "uptime-checks", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.eu1.remote)