Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..XX............XXXX.X.XX.....X.X....................X......] duration: 5.975924769s FAIL (2025-09-22 05:34:37 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.975956278s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................XX....X..X...X.X......X.........X.X..X...] duration: 5.92964312s FAIL (2025-09-22 05:34:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.92991599s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...XX............XXXX.X.XX.....X.X....................X.....] duration: 6.148896014s FAIL (2025-09-22 05:34:11 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.148931864s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................X...X...................................] duration: 5.856900725s FAIL (2025-09-22 05:34:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.856922968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....................X...X..................................] duration: 6.044363332s FAIL (2025-09-22 05:33:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.044411965s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....XX............XXXX.X.XX.....X.X....................X....] duration: 6.017748625s FAIL (2025-09-22 05:33:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.017782399s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [....................XX....X..X...X.X......X.........X.X..X..] duration: 6.155704534s FAIL (2025-09-22 05:33:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.155721595s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [......................X...X.................................] duration: 6.106312706s FAIL (2025-09-22 05:33:02 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.106345429s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [.....................XX....X..X...X.X......X.........X.X..X.] duration: 6.027343826s FAIL (2025-09-22 05:32:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.027369686s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......................X...X................................] duration: 5.802518892s FAIL (2025-09-22 05:32:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.802557324s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [X....XX............XXXX.X.XX.....X.X....................X...] duration: 6.082780444s FAIL (2025-09-22 05:32:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.082817575s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [......................XX....X..X...X.X......X.........X.X..X] duration: 30.00148111s FAIL (2025-09-22 05:32:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001508642s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.X....XX............XXXX.X.XX.....X.X....................X..] duration: 5.930494605s FAIL (2025-09-22 05:31:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 5.930525675s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......................XX....X..X...X.X......X.........X.X..] duration: 6.244487425s FAIL (2025-09-22 05:31:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.244706212s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................X...X...............................] duration: 5.9327144s FAIL (2025-09-22 05:31:22 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.932752914s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.X....XX............XXXX.X.XX.....X.X....................X.] duration: 6.034862242s FAIL (2025-09-22 05:31:00 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 6.034915414s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [........................XX....X..X...X.X......X.........X.X.] duration: 6.113324646s FAIL (2025-09-22 05:30:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.113336779s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.........................XX....X..X...X.X......X.........X.X] duration: 30.000763971s FAIL (2025-09-22 05:30:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000794179s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................X...X..............................] duration: 6.005449764s FAIL (2025-09-22 05:30:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.005478049s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.X....XX............XXXX.X.XX.....X.X....................X] duration: 30.251273562s FAIL (2025-09-22 05:30:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== 3.337157461s) [upload=330511]: Starting upload Sep 22 05:29:47.297668972 (376h1m23.337242102s) [upload=330511]: Done starting upload Sep 22 05:29:47.297899231 (376h1m23.337472361s) [upload=330511]: Uploading segments... Sep 22 05:29:47.298002678 (376h1m23.337575818s) [upload=330511]: Got next segment. Inline: false Sep 22 05:29:47.298059145 (376h1m23.337632275s) [upload=330511]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:29:47.298066109 (376h1m23.337639229s) [upload=330511]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:29:47.552911815 (376h1m23.592484945s) [seg_pos={0 0}, upload=330511]: Begin upload segment... Sep 22 05:29:47.552975406 (376h1m23.592548536s) [seg_pos={0 0}, upload=330511]: Done begin upload segment. Sep 22 05:29:47.553013599 (376h1m23.592586719s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553528389 (376h1m23.593101539s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553925034 (376h1m23.593498154s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.553961283 (376h1m23.593534413s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554326960 (376h1m23.59390009s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554351497 (376h1m23.593924627s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554588828 (376h1m23.594161948s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554724957 (376h1m23.594298087s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.554875986 (376h1m23.594449116s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555093950 (376h1m23.59466708s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555135429 (376h1m23.594708559s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555464797 (376h1m23.595037917s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.555726194 (376h1m23.595299314s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556009483 (376h1m23.595582614s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556364609 (376h1m23.59593773s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:47.556611299 (376h1m23.596184429s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Uploading piece... Sep 22 05:29:49.308510933 (376h1m25.348084063s) [upload=330511]: Next returned nil segment Sep 22 05:29:49.308533666 (376h1m25.348106796s) [upload=330511]: Waiting for error group managing segments... Sep 22 05:29:49.602590138 (376h1m25.642163268s) [noise=true , address=45.91.116.11:4117 , piece_id=NIFPLA2N3L6XDWTIFUM3YYRGBMFQJKWFVCGYXD2GBFYVYDXW6OEQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.633269677 (376h1m25.672842807s) [noise=true , address=77.90.31.15:4699 , piece_id=6XZN64DOILMELBYDU4D72ER33XDKMRIJRSAXX5ITFYPG46YEBITQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.637382749 (376h1m25.676955879s) [noise=true , address=45.91.116.6:4973 , piece_id=WPWUFOFQHQZFRPTR33GTDST2NYCVPXKLDVCJQFPQ4A7PI44LJKOQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.648543782 (376h1m25.688116902s) [noise=true , address=77.90.31.16:4735 , piece_id=K5KYKKY3BQV6UVJK4G3DYC24DFFIEFQX7KGGFLS4P35KJRFHA6UQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.653934647 (376h1m25.693507767s) [noise=true , address=45.91.116.14:4189 , piece_id=6VJ6XCCEV2CZSARNRF3ATZ4KCUI75QKWIWQWJL5QH4XTQG4MCLUQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.669096699 (376h1m25.708669829s) [noise=true , address=45.91.116.9:4261 , piece_id=REZZWYP25KZOP4UBJWJPHGCDUWCW3EBOJG2DOLCDYZY5B43KBMSQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.673672402 (376h1m25.713245532s) [noise=true , address=77.90.31.14:4663 , piece_id=LYGO7UTA55Q5Q7APPDJZNNWZ4S2XHAT4LS4V27VF27A3435DVJQA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:49.741841689 (376h1m25.781414809s) [noise=true , address=77.90.31.18:4807 , piece_id=JFOMLF75MIXEDP5CKDHH7J3PWGWESB7BIKK3SCAIOVNX74DJDC3A, seg_pos={0 0}, upload=330511]: Done uploading piece. err: <nil> Sep 22 05:29:58.021015047 (376h1m34.060588167s) [noise=true , address=145.79.227.5:4222 , piece_id=CZNVNWHETTFQS6JR2YLE65LNQU4WX74DAROY5I3KAYHG4QDLDOKA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:145.79.227.5:4222): protocol: rpc: dial tcp 145.79.227.5:4222: connect: no route to host; conn closed: op; piecestore: piecestore close: manager closed: conn closed: op Sep 22 05:30:05.028605724 (376h1m41.068178854s) [noise=true , address=145.79.227.3:4151 , piece_id=O6AU3AOBKJRIUMEAWPHMWWDDNRTA52WL72JNUTK5JFLVG2TNV5IA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:145.79.227.3:4151): protocol: rpc: dial tcp 145.79.227.3:4151: connect: no route to host; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586571260 (376h1m52.62614439s) [noise=true , address=145.79.227.12:4767 , piece_id=OUVFSM62B7UZ4U4WGFFBOHCPHXEXR6A6H2FQL3DTG5L3DEPUVYVQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:145.79.227.12:4767): protocol: rpc: dial tcp 145.79.227.12:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586679978 (376h1m52.626253118s) [noise=true , address=145.79.227.6:4258 , piece_id=D7HMSQH7ZLLKNTUZGJLVHIW2KEQZDL3HSS6VO2BSNRZJAC3USFAQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:145.79.227.6:4258): protocol: rpc: dial tcp 145.79.227.6:4258: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586708171 (376h1m52.626281301s) [noise=true , address=145.79.227.16:4912 , piece_id=AG2FUP2KFIJ7W55H2VRYTJYWQ57MAXCTEMTDWSR4FUJAYE3HFKBQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:145.79.227.16:4912): protocol: rpc: dial tcp 145.79.227.16:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586749971 (376h1m52.626323101s) [noise=true , address=145.79.227.15:4875 , piece_id=ZEWO52263YKTTRYDVF4PV52JTPEDJR3XGVRG3BFE5QQKDQSCE2LQ, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:145.79.227.15:4875): protocol: rpc: dial tcp 145.79.227.15:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586795768 (376h1m52.626368888s) [noise=true , address=145.79.227.17:4972 , piece_id=Q3KQQ57XWDUCXNHP33STZ7DN6VQ6FVYZTDPLINEPZASFEVY5XDJA, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:145.79.227.17:4972): protocol: rpc: dial tcp 145.79.227.17:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586806088 (376h1m52.626379218s) [noise=true , address=145.79.227.19:4077 , piece_id=EDBYIPZUOPFRRIZBZJJEP2U7NZFCUHRLDZESZ7J5CQZFKADNLE6Q, seg_pos={0 0}, upload=330511]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:145.79.227.19:4077): protocol: rpc: dial tcp 145.79.227.19:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:30:16.586831426 (376h1m52.626404556s) [seg_pos={0 0}, upload=330511]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Sep 22 05:30:16.834944071 (376h1m52.874517201s) [upload=330511]: Done uploading segments... Sep 22 05:30:16.834981573 (376h1m52.874554703s) [upload=330511]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.251312727s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................X...X.............................] duration: 6.058373148s FAIL (2025-09-22 05:29:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.058439876s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [X.........................XX....X..X...X.X......X.........X.] duration: 6.141066073s FAIL (2025-09-22 05:29:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.14109532s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........................X...X............................] duration: 5.875412768s FAIL (2025-09-22 05:29:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.875448787s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [..X.X....XX............XXXX.X.XX.....X.X....................] duration: 6.108193397s FAIL (2025-09-22 05:28:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.108227682s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [XX.........................XX....X..X...X.X......X.........X] duration: 30.000750253s FAIL (2025-09-22 05:28:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000772355s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X.X....XX............XXXX.X.XX.....X.X...................] duration: 6.023478653s FAIL (2025-09-22 05:28:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.023616796s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [............................X...X...........................] duration: 6.097088242s FAIL (2025-09-22 05:28:21 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.097115223s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [....X.X....XX............XXXX.X.XX.....X.X..................] duration: 5.995864191s FAIL (2025-09-22 05:27:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.995901s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............................X...X..........................] duration: 5.918121771s FAIL (2025-09-22 05:27:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.9181526s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.XX.........................XX....X..X...X.X......X.........] duration: 6.182754227s FAIL (2025-09-22 05:27:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.182783001s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.....X.X....XX............XXXX.X.XX.....X.X.................] duration: 6.083725614s FAIL (2025-09-22 05:26:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.083756622s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [..XX.........................XX....X..X...X.X......X........] duration: 6.178746835s FAIL (2025-09-22 05:26:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.178768426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [X.....X.X....XX............XXXX.X.XX.....X.X................] duration: 5.946542952s FAIL (2025-09-22 05:26:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.946571967s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............................X...X.........................] duration: 5.643810802s FAIL (2025-09-22 05:26:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.643847099s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............................X...X........................] duration: 6.052940728s FAIL (2025-09-22 05:25:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.052971947s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.X.....X.X....XX............XXXX.X.XX.....X.X...............] duration: 5.966775888s FAIL (2025-09-22 05:25:35 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.966809754s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [................................X...X.......................] duration: 5.851177434s FAIL (2025-09-22 05:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.85121129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [...XX.........................XX....X..X...X.X......X.......] duration: 5.979325607s FAIL (2025-09-22 05:25:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.979353941s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X.....X.X....XX............XXXX.X.XX.....X.X..............] duration: 6.171711076s FAIL (2025-09-22 05:24:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.171746474s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................................X...X......................] duration: 5.79844304s FAIL (2025-09-22 05:24:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.798476846s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [....XX.........................XX....X..X...X.X......X......] duration: 6.092126744s FAIL (2025-09-22 05:24:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.092170047s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...X.....X.X....XX............XXXX.X.XX.....X.X.............] duration: 5.916817991s FAIL (2025-09-22 05:23:57 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.916853888s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..................................X...X.....................] duration: 6.068218056s FAIL (2025-09-22 05:23:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 6.068254465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [....X.....X.X....XX............XXXX.X.XX.....X.X............] duration: 5.919086438s FAIL (2025-09-22 05:23:15 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 5.919165829s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.....XX.........................XX....X..X...X.X......X.....] duration: 5.940408168s FAIL (2025-09-22 05:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.940437163s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [...................................X...X....................] duration: 5.931525825s FAIL (2025-09-22 05:22:56 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.931559689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [......XX.........................XX....X..X...X.X......X....] duration: 5.801370797s FAIL (2025-09-22 05:22:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.801401025s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X....X.....X.X....XX............XXXX.X.XX.....X.X...........] duration: 6.214053054s FAIL (2025-09-22 05:22:24 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Sep 22 05:14:18.030489637 (375h45m54.070062767s) [upload=330284]: Starting upload Sep 22 05:14:18.030578265 (375h45m54.070151395s) [upload=330284]: Done starting upload Sep 22 05:14:18.030668377 (375h45m54.070241497s) [upload=330284]: Uploading segments... Sep 22 05:14:18.030818172 (375h45m54.070391302s) [upload=330284]: Got next segment. Inline: false Sep 22 05:14:18.030863758 (375h45m54.070436888s) [upload=330284]: Flush batch item: pb.BatchRequestItem_ObjectBegin Sep 22 05:14:18.030870672 (375h45m54.070443802s) [upload=330284]: Flush batch item: pb.BatchRequestItem_SegmentBegin Sep 22 05:14:18.274581174 (375h45m54.314154314s) [seg_pos={0 0}, upload=330284]: Begin upload segment... Sep 22 05:14:18.274644795 (375h45m54.314217926s) [seg_pos={0 0}, upload=330284]: Done begin upload segment. Sep 22 05:14:18.274678940 (375h45m54.314252071s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275194623 (375h45m54.314767753s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.275603681 (375h45m54.315176811s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276007791 (375h45m54.315580921s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276482074 (375h45m54.316055214s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.276877677 (375h45m54.316450807s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277378280 (375h45m54.31695141s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.277875187 (375h45m54.317448317s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278323050 (375h45m54.31789618s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.278740645 (375h45m54.318313785s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.279965447 (375h45m54.319538587s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.280691610 (375h45m54.32026475s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.281410097 (375h45m54.320983227s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.282060286 (375h45m54.321633416s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.283294026 (375h45m54.322867166s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:18.284496795 (375h45m54.324069935s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Uploading piece... Sep 22 05:14:20.053687445 (375h45m56.093260575s) [upload=330284]: Next returned nil segment Sep 22 05:14:20.053711810 (375h45m56.09328494s) [upload=330284]: Waiting for error group managing segments... Sep 22 05:14:20.350046921 (375h45m56.389620051s) [noise=true , address=45.91.116.5:4297 , piece_id=RFLFFJMM7UYDA6WX4FQMHOO4QD3BTVPMOYIWFXNAY45QUPINKVZQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.386568805 (375h45m56.426141926s) [noise=true , address=45.91.116.12:4225 , piece_id=B27PUA74XVTURGHGNL5GJXM7ZBV3I5NDWLII7NYK4BUKJMAY7WHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.408100086 (375h45m56.447673216s) [noise=true , address=45.91.116.8:4047 , piece_id=LGIHNMNB43DWVQTVOILCF3MRHB32PBEBZPV7DJMAVNS6ZJC24JWQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.410316977 (375h45m56.449890097s) [noise=true , address=77.90.31.16:4733 , piece_id=LANYC6IP5XYLPDW6TWG2MA7LD43FBPG4YE2JJRD3AG3SHO35WY2Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.428495202 (375h45m56.468068322s) [noise=true , address=45.91.116.14:4191 , piece_id=K67QTQSVSO3NHQGYBACLC57ZNPQGIAEDDIGEOYZS43Y5P6QSEBUQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.431901278 (375h45m56.471474408s) [noise=true , address=45.91.116.6:4974 , piece_id=DD3OROUAI26G72AXHHUPS3IBFH2ACXLAOHIAISKVDERXDER2YWCA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.460823843 (375h45m56.500396963s) [noise=true , address=77.90.31.10:4520 , piece_id=ESM2HBOTEGTDCYD2MXYD2XI54Y26M2OJ5ZCLRRRUZHOYX3JIXWOA, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.478192215 (375h45m56.517765345s) [noise=true , address=45.91.116.14:4189 , piece_id=ZXBC4XUSQKTM3TCM3OS2RBJMUD3U4JBRMHE5JLR2HFTABS254C5A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.486474456 (375h45m56.526047576s) [noise=true , address=77.90.31.16:4735 , piece_id=IKDOH7NHQOQ7ZENCO7HXMGMDBBV4SONDZ3XRCGFZ7UHTGAA3EQKQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.498839753 (375h45m56.538412873s) [noise=true , address=45.91.116.9:4262 , piece_id=75AMDTWNN4V25ZO6E7V6PKYRRFQEZYIKFWUQQXGORQJWEJ7PZBSQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.523338643 (375h45m56.562911773s) [noise=true , address=77.90.31.17:4770 , piece_id=2EAI3P53ZFWNSGMPEIGZINMQQHWBNVD5V6OY7LZMNBGSX6IBD46A, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:20.583611294 (375h45m56.623184424s) [noise=true , address=77.90.31.12:4590 , piece_id=6V2ILRXWCA36V25N4HMQ24D7I5VO7LTBMHAMW5C25LLV6MHMHFYQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: <nil> Sep 22 05:14:47.356515816 (375h46m23.396088936s) [noise=true , address=145.79.227.2:4115 , piece_id=BJSDTOSRHB46CDV34TXHGWSKNK5WE7N62G3SJIUTF375OSQBX34Q, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:145.79.227.2:4115): protocol: rpc: dial tcp 145.79.227.2:4115: i/o timeout; conn closed: op Sep 22 05:14:47.356553858 (375h46m23.396126988s) [noise=true , address=145.79.227.14:4839 , piece_id=YKDAYGS5UTMX4AZMLLKDC5JKRKUGY5BKY6JNFJMBTPSFAX4STHHQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:145.79.227.14:4839): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356730084 (375h46m23.396303214s) [noise=true , address=145.79.227.2:4114 , piece_id=BLZCISF673GTFC3525W7QVZKCG5FORIFWG324XFK4KTHUL32UZCQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:145.79.227.2:4114): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356737578 (375h46m23.396310708s) [noise=true , address=145.79.227.12:4765 , piece_id=Y7NS4BKYMXVIS3UX2XSDGAAHYWBLZBJWGTP6EQZIEGJAKVMPFBQQ, seg_pos={0 0}, upload=330284]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:145.79.227.12:4765): protocol: context deadline exceeded; context deadline exceeded Sep 22 05:14:47.356761373 (375h46m23.396334503s) [seg_pos={0 0}, upload=330284]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Sep 22 05:14:47.594095784 (375h46m23.633668914s) [upload=330284]: Done uploading segments... Sep 22 05:14:47.594165507 (375h46m23.633738637s) [upload=330284]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 6.21410899s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 9 out of 60 is failed (threshold=3) [.......XX.........................XX....X..X...X.X......X...] duration: 6.028961404s FAIL (2025-09-22 05:22:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 6.028982274s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [....................................X...X...................] duration: 5.854599554s FAIL (2025-09-22 05:21:49 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 5h37m59.543525066s) [seg_pos={0 0}, upload=330171]: Begin upload segment... Sep 22 05:06:23.504018422 (375h37m59.543591552s) [seg_pos={0 0}, upload=330171]: Done begin upload segment. Sep 22 05:06:23.504059370 (375h37m59.543632501s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.504681175 (375h37m59.544254355s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505278141 (375h37m59.544851301s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505743808 (375h37m59.545316948s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.505904454 (375h37m59.545477584s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506859502 (375h37m59.546432622s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.506958631 (375h37m59.546531761s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.507277077 (375h37m59.546850217s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.508336856 (375h37m59.547909986s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513150782 (375h37m59.552723912s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.513934875 (375h37m59.553508006s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.514566187 (375h37m59.554139318s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515189424 (375h37m59.554762544s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.515805377 (375h37m59.555378508s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.516494880 (375h37m59.55606801s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.526689722 (375h37m59.566262863s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.531527154 (375h37m59.571100294s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:23.537921861 (375h37m59.577494991s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Uploading piece... Sep 22 05:06:24.564378058 (375h38m0.603951188s) [noise=true , address=145.79.227.5:4247 , piece_id=QGN2S5CET2J6LGW6YPQYDWEVOGEMHWRSRMJG2IUQ36KA23OG2R3A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1262xxX6aUPTYgKdvRxAyc77PmPBWJtq5cXRJFoELxHbfn18DLE, address:145.79.227.5:4247): protocol: rpc: dial tcp 145.79.227.5:4247: connect: no route to host; conn closed: op Sep 22 05:06:24.986206552 (375h38m1.025779682s) [upload=330171]: Next returned nil segment Sep 22 05:06:24.986226830 (375h38m1.02579995s) [upload=330171]: Waiting for error group managing segments... Sep 22 05:06:25.383578176 (375h38m1.423151306s) [noise=true , address=145.79.227.24:4594 , piece_id=V55Y7DFR7QJWBBQRKSTUWYNEZSCAIWAY2XHAOZOF2H7JDFSX6IBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.383806571 (375h38m1.423379702s) [noise=true , address=145.79.227.33:4881 , piece_id=AIKNNVT5PMYFYKRMUFN5DQMY7TWEUN3FUC64IQQYKAMV723TKKZA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.398811493 (375h38m1.438384634s) [noise=true , address=145.79.227.46:4601 , piece_id=EVWU7M2P2W4FQCNDEQ2OO3SR4KHPTXQRM7GEI6Q6DHYMDHK4PP4Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.399530783 (375h38m1.439103913s) [noise=true , address=145.79.227.39:4039 , piece_id=MZRBH2VHR7CCJK4OCCXINOQTNBP7PUP236H7DB2DFL7ZFDBNI2IQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.406508421 (375h38m1.446081551s) [noise=true , address=145.79.227.49:4338 , piece_id=NBR62GETN7QBXC7QJCUB5DZHKVMC5AKQQBIIGDRXNECNZFULEDDA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.415736332 (375h38m1.455309462s) [noise=true , address=145.79.227.25:4637 , piece_id=3SMC4AHCMKJEDXJOJVZ3FJEKW23VZDSWW6MDCENAYJ7HKOPWNHBQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.437405044 (375h38m1.476978184s) [noise=true , address=145.79.227.31:4818 , piece_id=KAJZKTKOHNN5U4CTYJ7CLO32ONU6VSEJ2QBLPBI5WWZZ37YOI5NA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.465513107 (375h38m1.505086237s) [noise=true , address=77.90.31.19:4847 , piece_id=6GLTB4OVJTM67VRKRWATCIWE7WVK35IQMQIVOEZMZM6I5DFNRIBA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.501349827 (375h38m1.540922957s) [noise=true , address=45.91.116.9:4292 , piece_id=3QGXKEK5C77XDD34Z77C3K2LYEMGJT2SADTL6RGW2YCYQFV5CLUA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.579007182 (375h38m1.618580302s) [noise=true , address=77.90.31.16:4764 , piece_id=PSAP3HPVNEURWWNC7K2K2G4NEUENKWDUR7OJY6UBGRTCSBYNHRWQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:06:25.586807715 (375h38m1.626380845s) [noise=true , address=45.91.116.14:4214 , piece_id=DOY7FC4SJKGAFP3LHSJXKWCRSYI6I6IR2R2FZ47QJNDM4SCAUXQQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: <nil> Sep 22 05:07:22.245417769 (375h38m58.284990899s) [noise=true , address=145.79.227.16:4933 , piece_id=WOBRP7MRD6VVMMXEHKNL4XDSPKQBHVINQD3I4N2SQU2MOSC2ZUCQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:19m4inaXvTdwf2rKip3eFELv6JAgFvXJ8KqjPZYbb48i8Ktnwe, address:145.79.227.16:4933): protocol: rpc: dial tcp 145.79.227.16:4933: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245544310 (375h38m58.28511744s) [noise=true , address=145.79.227.12:4770 , piece_id=X5KVN5TA67L4TTD7LVG6N4I54BKNESFAHXRPVIWGDFCEPSHRKXPA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1Ug6GpYToYDXiHmiJP4GoE7eJV3kqRveix2S5SLDcKQJtEbNPz, address:145.79.227.12:4770): protocol: rpc: dial tcp 145.79.227.12:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245567825 (375h38m58.285140955s) [noise=true , address=145.79.227.16:4939 , piece_id=DLS7J35PB7NXNMAPTVHBZ3VN3U725CSVZZKEYY37HZWQPWCCJO7A, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1mRmp3HhD95PruS4AF8soRsSXNYUeb4mJE7y52FvoeN9D3Gu4F, address:145.79.227.16:4939): protocol: rpc: dial tcp 145.79.227.16:4939: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245634181 (375h38m58.285207312s) [noise=true , address=145.79.227.7:4308 , piece_id=QEUKJMQDWYMBVKSVNNVDK53P6CZ4X47IINRSWCHAFG7APSFOOAFQ, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:1PgA5SZkBvK673LsV7p2sXpbi1r79esNNZspPZTzaitqfbjtyY, address:145.79.227.7:4308): protocol: rpc: dial tcp 145.79.227.7:4308: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245657335 (375h38m58.285230455s) [noise=true , address=145.79.227.10:4420 , piece_id=ANGUQNIL2D4K2YR7KVJMY37WW7ILRVXSZOB7FRRAKJPD3AGGPGOA, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12hJrjeMUMYmFw3yxRtQQKRh9ciPfdu5FXo8NGrsc6HBqgfJzzp, address:145.79.227.10:4420): protocol: rpc: dial tcp 145.79.227.10:4420: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245675269 (375h38m58.28524839s) [noise=true , address=145.79.227.6:4292 , piece_id=BYUJPZSYFTYD4C4F5EBAW6BSFXTILW6EASOIRHZQI7ARCSG63F2Q, seg_pos={0 0}, upload=330171]: Done uploading piece. err: ecclient: upload failed (node:12UpWmMhEmk5veKZvzsQ3MCxCMnfEKtFgCugBTsYfy9Ejm7G4xD, address:145.79.227.6:4292): protocol: rpc: dial tcp 145.79.227.6:4292: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Sep 22 05:07:22.245712370 (375h38m58.2852855s) [seg_pos={0 0}, upload=330171]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Sep 22 05:07:22.500217377 (375h38m58.539790507s) [upload=330171]: Done uploading segments... Sep 22 05:07:22.500253296 (375h38m58.539826426s) [upload=330171]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.854632847s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)